Koalas: pandas API on Apache Spark
Alternatives To Koalas
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Spark35,9452,39488211 hours ago46May 09, 2021266apache-2.0Scala
Apache Spark - A unified analytics engine for large-scale data processing
2 months ago110apache-2.0
The Data Engineering Cookbook
God Of Bigdata7,992
2 months ago2
Zeppelin6,06332232 days ago2June 21, 2017143apache-2.0Java
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
2 years ago27
Notes talking about the design and implementation of Apache Spark
11 hours ago4May 23, 20221,361apache-2.0Java
Apache Iceberg
Bigdl4,22310a day ago16April 19, 2021747apache-2.0Jupyter Notebook
Fast, distributed, secure AI for Big Data
5a month ago32April 21, 202213apache-2.0Python
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Spark Nlp3,278222 days ago90March 05, 202137apache-2.0Scala
State of the Art Natural Language Processing
Koalas3,2281126 months ago47October 19, 2021109apache-2.0Python
Koalas: pandas API on Apache Spark
Alternatives To Koalas
Select To Compare

Alternative Project Comparisons

DEPRECATED: Koalas supports Apache Spark 3.1 and below as it is officially included to PySpark in Apache Spark 3.2. This repository is now in maintenance mode. For Apache Spark 3.2 and above, please use PySpark directly.

pandas API on Apache Spark
Explore Koalas docs »

Live notebook · Issues · Mailing list
Help Thirsty Koalas Devastated by Recent Fires

The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:

  • Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
  • Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).

We would love to have you try it and give us feedback, through our mailing lists or GitHub issues.

Try the Koalas 10 minutes tutorial on a live Jupyter notebook here. The initial launch can take up to several minutes.

Github Actions codecov Documentation Status Latest Release Conda Version Binder Downloads

Getting Started

Koalas can be installed in many ways such as Conda and pip.

# Conda
conda install koalas -c conda-forge
# pip
pip install koalas

See Installation for more details.

For Databricks Runtime, Koalas is pre-installed in Databricks Runtime 7.1 and above. Try Databricks Community Edition for free. You can also follow these steps to manually install a library on Databricks.

Lastly, if your PyArrow version is 0.15+ and your PySpark version is lower than 3.0, it is best for you to set ARROW_PRE_0_15_IPC_FORMAT environment variable to 1 manually. Koalas will try its best to set it for you but it is impossible to set it if there is a Spark context already launched.

Now you can turn a pandas DataFrame into a Koalas DataFrame that is API-compliant with the former:

import databricks.koalas as ks
import pandas as pd

pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})

# Create a Koalas DataFrame from pandas DataFrame
df = ks.from_pandas(pdf)

# Rename the columns
df.columns = ['x', 'y', 'z1']

# Do some operations in place:
df['x2'] = df.x * df.x

For more details, see Getting Started and Dependencies in the official documentation.

Contributing Guide

See Contributing Guide and Design Principles in the official documentation.


See FAQ in the official documentation.

Best Practices

See Best Practices in the official documentation.

Koalas Talks and Blogs

See Koalas Talks and Blogs in the official documentation.

Popular Spark Projects
Popular Apache Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Data Science
Big Data
Apache Spark