Awesome Open Source
Awesome Open Source

.. image:: :target:

======== ml-pyxis

Tool for reading and writing datasets of tensors (numpy.ndarray) with MessagePack and Lightning Memory-Mapped Database (LMDB).


.. code-block:: python

import numpy as np import pyxis as px

Create data

nb_samples = 10 X = np.ones((nb_samples, 2, 2), dtype=np.float32) y = np.arange(nb_samples, dtype=np.uint8)


db = px.Writer(dirpath='data', map_size_limit=1) db.put_samples('input', X, 'target', y) db.close()


db = px.Reader(dirpath='data') sample = db[0] db.close()


.. code-block:: python

{'input': array([[ 1., 1.], [ 1., 1.]], dtype=float32), 'target': array(0, dtype=uint8)}

More examples can be found in the examples/ directory.


The installation instructions are generic and should work on most operating systems that support the prerequisites.

ml-pyxis requires Python version 2.7, 3.4, 3.5, or 3.6. We recommend installing ml-pyxis, as well as all prerequisites, in a virtual environment via virtualenv_.


The following Python packages are required to use ml-pyxis:

  • lmdb_ - Universal Python binding for the LMDB 'Lightning' Database_
  • msgpack_ - MessagePack_ implementation for Python (binary serialisation)
  • NumPy_ - N-dimensional array object and tools for operating on them
  • six_ - A Python 2 and 3 compatibility library

Please refer to the individual packages for more information about additional dependencies and how to install them for your operating system.

Bleeding-edge installation

To install the latest version of ml-pyxis, use the following command:

.. code-block:: bash

pip install --upgrade

Add the --user tag if you want to install the package in your home directory.


The previous LMDB-only API has been deprecated in favour of a combination between LMDB and msgpack. The old version can be installed by using the following commit hash with pip:

.. code-block:: bash

pip install --upgrade git+git://[email protected]

Development installation

ml-pyxis can be installed from source in such a way that any changes to your local copy will take effect without having to reinstall the package. Start by making a copy of the repository:

.. code-block:: bash

git clone

Next, enter the directory and install ml-pyxis in development mode by issuing the following command:

.. code-block:: bash

cd ml-pyxis python develop

.. Links

.. _virtualenv: .. _lmdb: .. _LMDB 'Lightning' Database: .. _msgpack: .. _MessagePack: .. _NumPy: .. _six:

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (48,505
deep-learning (3,536
machine-learning (3,237
data-science (779
data (355
dataset (332
msgpack (34
lmdb (19

Find Open Source By Browsing 7,000 Topics Across 59 Categories