🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Alternatives To Optimus
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Synapseml4,56634 days ago9November 22, 2022321mitScala
Simple and Distributed Machine Learning
Machine Learning2,570
12 hours ago5mitHTML
:earth_americas: machine learning tutorials (mainly in Python3)
Spark Py Notebooks1,515
6 months ago9otherJupyter Notebook
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
14 days ago32June 19, 202227apache-2.0Python
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Pyspark Example Project1,034
a year ago11Python
Example project implementing best practices for PySpark ETL jobs and applications.
4 hours ago1September 11, 20199agpl-3.0Java
Hopsworks - Data-Intensive AI platform with a Feature Store
a year ago22apache-2.0JavaScript
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
2 years ago22January 25, 20202Python
PandaPy has the speed of NumPy and the usability of Pandas 10x to 50x faster (by @firmai)
Datacompy31617 days ago10April 19, 202213apache-2.0Python
Pandas and Spark DataFrame comparison for humans and more!
Sk Dist283
28 months ago12May 14, 20208apache-2.0Python
Distributed scikit-learn meta-estimators in PySpark
Alternatives To Optimus
Select To Compare

Alternative Project Comparisons


Logo Optimus

Tests Docker image updated PyPI Latest Release GitHub release CalVer

Downloads Downloads Downloads Mentioned in Awesome Data Science Slack


Optimus is an opinionated python library to easily load, process, plot and create ML models that run over pandas, Dask, cuDF, dask-cuDF, Vaex or Spark.

Some amazing things Optimus can do for you:

  • Process using a simple API, making it easy to use for newcomers.
  • More than 100 functions to handle strings, process dates, urls and emails.
  • Easily plot data from any size.
  • Out of box functions to explore and fix data quality.
  • Use the same code to process your data in your laptop or in a remote cluster of GPUs.

See Documentation

Try Optimus

To launch a live notebook server to test optimus using binder or Colab, click on one of the following badges:

Binder Colab

Installation (pip):

In your terminal just type:

pip install pyoptimus

By default Optimus install Pandas as the default engine, to install other engines you can use the following commands:

Engine Command
Dask pip install pyoptimus[dask]
cuDF pip install pyoptimus[cudf]
Dask-cuDF pip install pyoptimus[dask-cudf]
Vaex pip install pyoptimus[vaex]
Spark pip install pyoptimus[spark]

To install from the repo:

pip install git+https://github.com/hi-primus/[email protected]

To install other engines:

pip install git+https://github.com/hi-primus/[email protected]#egg=pyoptimus[dask]


  • Python 3.7 or 3.8


You can go to 10 minutes to Optimus where you can find the basics to start working in a notebook.

Also you can go to the Examples section and find specific notebooks about data cleaning, data munging, profiling, data enrichment and how to create ML and DL models.

Here's a handy Cheat Sheet with the most common Optimus' operations.

Start Optimus

Start Optimus using "pandas", "dask", "cudf","dask_cudf","vaex" or "spark".

from optimus import Optimus
op = Optimus("pandas")

Loading data

Now Optimus can load data in csv, json, parquet, avro and excel formats from a local file or from a URL.

df = op.load.csv("../examples/data/foo.csv")

df = op.load.json("../examples/data/foo.json")

# using a url
df = op.load.json("https://raw.githubusercontent.com/hi-primus/optimus/develop-23.5/examples/data/foo.json")

# parquet
df = op.load.parquet("../examples/data/foo.parquet")

# ...or anything else
df = op.load.file("../examples/data/titanic3.xls")

Also, you can load data from Oracle, Redshift, MySQL and Postgres databases.

Saving Data


# json

# parquet

You can also save data to oracle, redshift, mysql and postgres.

Create dataframes

Also, you can create a dataframe from scratch

df = op.create.dataframe({
    'A': ['a', 'b', 'c', 'd'],
    'B': [1, 3, 5, 7],
    'C': [2, 4, 6, None],
    'D': ['1980/04/10', '1980/04/10', '1980/04/10', '1980/04/10']

Using display you have a beautiful way to show your data with extra information like column number, column data type and marked white spaces.


Cleaning and Processing

Optimus was created to make data cleaning a breeze. The API was designed to be super easy to newcomers and very familiar for people that comes from Pandas. Optimus expands the standard DataFrame functionality adding .rows and .cols accessors.

For example you can load data from a url, transform and apply some predefined cleaning functions:

new_df = df\
    .rows.sort("rank", "desc")\
    .cols.lower(["names", "function"])\
    .cols.date_format("date arrival", "yyyy/MM/dd", "dd-MM-YYYY")\
    .cols.years_between("date arrival", "dd-MM-YYYY", output_cols="from arrival")\
    .cols.rename("*", str.lower)\
    .cols.unnest("japanese name", output_cols="other names")\
    .cols.unnest("last position seen", separator=",", output_cols="pos")\
    .cols.drop(["last position seen", "japanese name", "date arrival", "cybertronian", "nulltype"])

Need help?


Feedback is what drive Optimus future, so please take a couple of minutes to help shape the Optimus' Roadmap: http://bit.ly/optimus_survey

Also if you want to a suggestion or feature request use https://github.com/hi-primus/optimus/issues


If you have issues, see our Troubleshooting Guide

Contributing to Optimus

Contributions go far beyond pull requests and commits. We are very happy to receive any kind of contributions

  • Documentation updates, enhancements, designs, or bugfixes.
  • Spelling or grammar fixes.
  • README.md corrections or redesigns.
  • Adding unit, or functional tests
  • Triaging GitHub issues -- especially determining whether an issue still persists or is reproducible.
  • Blogging, speaking about, or creating tutorials about Optimus and its many features.
  • Helping others on our official chats

Backers and Sponsors

Become a backer or a sponsor and get your image on our README on Github with a link to your site.

OpenCollective OpenCollective

Popular Pyspark Projects
Popular Data Science Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Machine Learning
Data Science
Data Analysis
Big Data
Data Cleaning
Data Transformation