DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.
DataFusion also supports distributed query execution via the Ballista crate.
DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.
Projects that adapt to or serve as plugins to DataFusion:
Here are some of the projects known to use DataFusion:
(if you know of another project, please submit a PR to add a link!)
Please see example usage to find how to use DataFusion.
Please see Roadmap for information of where the project is headed.
There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.
Please see User Guide for more information about DataFusion.
Please see Contribution Guide for information about contributing to DataFusion.