Reading BinPickle Files#

Reading an object from a BinPickle file is done with load():

from binpickle import load
obj = load('file.bpk')

The load Function#

binpickle.load(file)#

Load an object from a BinPickle file.

Parameters:

file (str | PathLike[str]) – The file to load.

Return type:

object

The BinPickleFile Class#

For full control of the deserialization process, and in particular to support memory-mapped object contents (e.g. for shared memory use), use binpickle.BinPickleFile directly.

If you open the BinPickle file in direct mode (the direct=True argument to binpickle.BinPickleFile.__init__()), then the contents of buffers in the object (e.g. NumPy arrays or Pandas columns) will be directly backed by a read-only memory-mapped region from the BinPickle file. This has two consequences:

  1. Multiple processes with the same file open in direct mode will (usually) share buffer memory. This is one of BinPickle’s particular benefits: an easy way to load large array data, such as the learned parameters of many SciKit-style machine learning models, in multiple processes with minimal duplication.

  2. The binpickle.BinPickleFile object cannot be closed until all objects referencing its memory have been destroyed. binpickle.BinPickleFile.close() will throw an exception if it is closed prematurely.

class binpickle.BinPickleFile(filename, *, direct=False, verify=True)#

Class representing a binpickle file in memory.

Parameters:

  • filename (str | PathLike[str]) – The name of the file to load.

  • direct (bool | str) – If True, returned objects zero-copy when possible, but cannot outlast the BinPickleFile instance. If False, they are copied from the file and do not need to be freed before close() is called. If the string 'nowarn', open in direct mode but do not warn if the file is unmappable.

  • verify (bool) – If True (the default), verify file checksums while reading.

load()#

Load the object from the binpickle file.

Return type:

object

property is_mappable: bool#

Query whether this file can be memory-mapped.

find_errors()#

Verify binpickle data structure validity. If the file is invalid, returns a list of errors.

Fatal index errors will result in a failure to open the file, so things such as invalid msgpack formats in the index won’t be detected here. This method checks buffer hashes, offset overlaps, and such.

Return type:

list[str]

close()#

Close the BinPickle file. If the file is in direct mode, all retrieved objects and associated views must first be deleted.

Return type:

None