LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.
Alternatives To Lakesoul
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
35 minutes ago1June 08, 20222,781otherClojure
The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum:
an hour ago1,745apache-2.0Java
Free universal database tool and SQL client
2 hours ago837mitJavaScript
Feature-rich ORM for modern Node.js and TypeScript, it supports PostgreSQL (with JSON and JSONB support), MySQL, MariaDB, SQLite, MS SQL Server, Snowflake, Oracle DB (v6), DB2 and DB2 for IBM i.
14 hours ago2May 05, 2020776bsd-2-clausePython
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Directus20,7155040 minutes ago55September 22, 2022348gpl-3.0TypeScript
The Modern Data Stack 🐰 — Directus is an instant REST+GraphQL API and intuitive no-code data collaboration app for any SQL database.
4a day ago37July 12, 2022201mitHaskell
REST API for any Postgres database
Shardingsphere18,1358an hour ago7June 04, 2020567apache-2.0Java
Ecosystem to transform any database into a distributed database system, and enhance it with sharding, elastic scaling, encryption features & more
Knex17,29118,0962,7888 hours ago248August 31, 2022822mitJavaScript
A query builder for PostgreSQL, MySQL, CockroachDB, SQL Server, SQLite3 and Oracle, designed to be flexible, portable, and fun to use.
an hour ago750otherRust
📊 Cube — The Semantic Layer for Building Data Applications
2 hours ago520otherC
An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
Alternatives To Lakesoul
Select To Compare

Alternative Project Comparisons


LakeSoul is a cloud-native Lakehouse framework developed by DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing. LakeSoul Arch

LakeSoul implements incremental upserts for both row and column and allows concurrent updates. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales metadata management and achieves ACID control by using PostgreSQL. LakeSoul provides tools to ingest CDC and log streams automatically in a zero-ETL style.

More detailed features please refer to our doc page: Documentations

Maven Test Flink CDC Test

Quick Start

Follow the Quick Start to quickly set up a test env.


Please find tutorials in doc site: Tutorials

Usage Documentations

Please find usage documentations in doc site: Usage Doc

Feature Roadmap

  • Meta Management (#23)
    • [x] Multiple Level Partitioning: Multiple range partition and at most one hash partition
    • [x] Concurrent write with auto conflict resolution
    • [x] MVCC with read isolation
    • [x] Write transaction (two-stage commit) through Postgres Transaction
    • [x] Schema Evolution: Column add/delete supported
  • Table operations
    • [x] LSM-Tree style upsert for hash partitioned table
    • [x] Merge on read for hash partition with upsert delta file
    • [x] Copy on write update for non hash partitioned table
    • [x] Compaction
  • Data Warehousing
    • [x] CDC stream ingestion with auto ddl sync
    • [x] Incremental and Snapshot Query
      • [x] Snapshot Query (#103)
      • [x] Incremental Query (#103)
      • [x] Incremental Streaming Source (#130)
    • [ ] Materialized View
      • [ ] Incremental MV Build
      • [ ] Auto query rewrite
  • Spark Integration
    • [x] Table/Dataframe API
    • [x] SQL support with catalog except upsert
    • [x] Query optimization
      • [x] Shuffle/Join elimination for operations on primary key
    • [x] Merge UDF (Merge operator)
    • [ ] Merge Into SQL support
      • [x] Merge Into SQL with match on Primary Key (Merge on read)
      • [ ] Merge Into SQL with match on non-pk
      • [ ] Merge Into SQL with match condition and complex expression (Merge on read when match on PK) (depends on #66)
    • [x] Multiple Spark Versions Support
      • [x] Support Spark 3.3, 3.2 and 3.1
  • Flink Integration and CDC Ingestion (#57)
    • [x] Table API
    • [x] Flink CDC
      • [x] Exactly Once Sink
      • [x] Auto Schema Change (DDL) Sync
      • [x] Auto Table Creation (depends on #78)
      • [x] Support multiple source tables with different schemas (#84)
  • Hive Integration
    • [x] Export to Hive partition after compaction
    • [x] Apache Kyuubi (Hive JDBC) Integration
  • Realtime Data Warehousing
    • [x] CDC ingestion
    • [x] Time Travel (Snapshot read)
    • [x] Snapshot rollback
    • [ ] MPP Engine Integration (depends on #66)
      • [ ] Presto
      • [ ] Apache Doris
  • Cloud and Native IO (#66)
    • [x] Object storage IO optimization
    • [ ] Native merge on read
    • [ ] Multi-layer storage classes support with data tiering

Community guidelines

Community guidelines

Feedback and Contribution

Please feel free to open an issue or dicussion if you have any questions.

Join our slack user group

Contact Us

Email us at [email protected].

Opensource License

LakeSoul is opensourced under Apache License v2.0.

Popular Postgresql Projects
Popular Sql Projects
Popular Data Storage Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Big Data