Reading BinPickle Files

Reading an object from a BinPickle file is done with load():

from binpickle import load
obj = load('file.bpk')

The load Function

binpickle.load(file)

Load an object from a BinPickle file.

Parameters

file (str or pathlib.Path) – The file to load.

The BinPickleFile Class

For full control of the deserialization process, and in particular to support memory-mapped object contents (e.g. for shared memory use), use binpickle.BinPickleFile directly.

If you open the BinPickle file in direct mode (the direct=True argument to binpickle.BinPickleFile.__init__()), then the contents of buffers in the object (e.g. NumPy arrays or Pandas columns) will be directly backed by a read-only memory-mapped region from the BinPickle file. This has two consequences:

  1. Multiple processes with the same file open in direct mode will (usually) share buffer memory. This is one of BinPickle’s particular benefits: an easy way to load large array data, such as the learned parameters of many SciKit-style machine learning models, in multiple processes with minimal duplication.

  2. The binpickle.BinPickleFile object cannot be closed until all objects referencing its memory have been destroyed. binpickle.BinPickleFile.close() will throw an exception if it is closed prematurely.

class binpickle.BinPickleFile(filename, *, direct=False)

Class representing a binpickle file in memory.

Parameters

  • filename (str or pathlib.Path) – The name of the file to load.

  • direct (bool) – If True, returned objects zero-copy when possible, but cannot outlast the BinPickleFile instance. If False, they are copied from the file and do not need to be freed before close() is called.

load()

Load the object from the binpickle file.

find_errors()

Verify binpickle data structure validity. If the file is invalid, returns a list of errors.

Fatal index errors will result in a failure to open the file, so things such as invalid msgpack formats in the index won’t be detected here. This method checks buffer checksums, offset overlaps, and such.

close()

Close the BinPickle file. If the file is in direct mode, all retrieved objects and associated views must first be deleted.