numpy 1.22.4 Pypi GitHub Homepage
Other Docs
AttributesMethodsNotesParametersBackRef

Attributes

filename : str or pathlib.Path instance

Path to the mapped file.

offset : int

Offset position in the file.

mode : str

File mode.

Memory-mapped files are used for accessing small segments of large files on disk, without reading the entire file into memory. NumPy's memmap's are array-like objects. This differs from Python's mmap module, which uses file-like objects.

This subclass of ndarray has some unpleasant interactions with some operations, because it doesn't quite fit properly as a subclass. An alternative to using this subclass is to create the mmap object yourself, then create an ndarray with ndarray.__new__ directly, passing the object created in its 'buffer=' parameter.

This class may at some point be turned into a factory function which returns a view into an mmap buffer.

Flush the memmap instance to write the changes to the file. Currently there is no API to close the underlying mmap . It is tricky to ensure the resource is actually closed, since it may be shared between different memmap instances.

Methods

Notes

The memmap object can be used anywhere an ndarray is accepted. Given a memmap fp , isinstance(fp, numpy.ndarray) returns True .

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.

Parameters

filename : str, file-like object, or pathlib.Path instance

The file name or file object to be used as the array data buffer.

dtype : data-type, optional

The data-type used to interpret the file contents. Default is uint8 .

mode : {'r+', 'r', 'w+', 'c'}, optional

The file is opened in this mode:

+------+-------------------------------------------------------------+ | 'r' | Open existing file for reading only. | +------+-------------------------------------------------------------+ | 'r+' | Open existing file for reading and writing. | +------+-------------------------------------------------------------+ | 'w+' | Create or overwrite existing file for reading and writing. | +------+-------------------------------------------------------------+ | 'c' | Copy-on-write: assignments affect data in memory, but | | | changes are not saved to disk. The file on disk is | | | read-only. | +------+-------------------------------------------------------------+

Default is 'r+'.

offset : int, optional

In the file, array data starts at this offset. Since :None:None:`offset` is measured in bytes, it should normally be a multiple of the byte-size of dtype . When mode != 'r' , even positive offsets beyond end of file are valid; The file will be extended to accommodate the additional data. By default, memmap will start at the beginning of the file, even if filename is a file pointer fp and fp.tell() != 0 .

shape : tuple, optional

The desired shape of the array. If mode == 'r' and the number of remaining bytes after :None:None:`offset` is not a multiple of the byte-size of dtype , you must specify shape . By default, the returned array will be 1-D with the number of elements determined by file size and data-type.

order : {'C', 'F'}, optional

Specify the order of the ndarray memory layout: row-major , C-style or column-major , Fortran-style. This only has an effect if the shape is greater than 1-D. The default order is 'C'.

Create a memory-map to an array stored in a binary file on disk.

See Also

lib.format.open_memmap

Create or load a memory-mapped .npy file.

Examples

>>> data = np.arange(12, dtype='float32')
... data.resize((3,4))

This example uses a temporary file so that doctest doesn't write files to your directory. You would use a 'normal' filename.

>>> from tempfile import mkdtemp
... import os.path as path
... filename = path.join(mkdtemp(), 'newfile.dat')

Create a memmap with dtype and shape that matches our data:

>>> fp = np.memmap(filename, dtype='float32', mode='w+', shape=(3,4))
... fp memmap([[0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]], dtype=float32)

Write data to memmap array:

>>> fp[:] = data[:]
... fp memmap([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.]], dtype=float32)
>>> fp.filename == path.abspath(filename)
True

Flushes memory changes to disk in order to read them back

>>> fp.flush()

Load the memmap and verify data was stored:

>>> newfp = np.memmap(filename, dtype='float32', mode='r', shape=(3,4))
... newfp memmap([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.]], dtype=float32)

Read-only memmap:

>>> fpr = np.memmap(filename, dtype='float32', mode='r', shape=(3,4))
... fpr.flags.writeable False

Copy-on-write memmap:

>>> fpc = np.memmap(filename, dtype='float32', mode='c', shape=(3,4))
... fpc.flags.writeable True

It's possible to assign to copy-on-write array, but values are only written into the memory copy of the array, and not written to disk:

>>> fpc
memmap([[  0.,   1.,   2.,   3.],
        [  4.,   5.,   6.,   7.],
        [  8.,   9.,  10.,  11.]], dtype=float32)
>>> fpc[0,:] = 0
... fpc memmap([[ 0., 0., 0., 0.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.]], dtype=float32)

File on disk is unchanged:

>>> fpr
memmap([[  0.,   1.,   2.,   3.],
        [  4.,   5.,   6.,   7.],
        [  8.,   9.,  10.,  11.]], dtype=float32)

Offset into a memmap:

>>> fpo = np.memmap(filename, dtype='float32', mode='r', offset=16)
... fpo memmap([ 4., 5., 6., 7., 8., 9., 10., 11.], dtype=float32)
See :

Back References

The following pages refer to to this document either explicitly or contain code examples using this.

numpy.lib.format.open_memmap numpy.load numpy.lib.arrayterator.Arrayterator numpy.core.memmap.memmap.flush

Local connectivity graph

Hover to see nodes names; edges to Self not shown, Caped at 50 nodes.

Using a canvas is more power efficient and can get hundred of nodes ; but does not allow hyperlinks; , arrows or text (beyond on hover)

SVG is more flexible but power hungry; and does not scale well to 50 + nodes.

All aboves nodes referred to, (or are referred from) current nodes; Edges from Self to other have been omitted (or all nodes would be connected to the central node "self" which is not useful). Nodes are colored by the library they belong to, and scaled with the number of references pointing them


GitHub : /numpy/__init__.py#None
type: <class 'type'>
Commit: