|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Pipeline||4,160||7 months ago||85||July 18, 2017||1||apache-2.0||Jsonnet|
|PipelineAI Kubeflow Distribution|
|Zio Quill||2,122||1||20||43 minutes ago||96||September 09, 2022||399||apache-2.0||Scala|
|Compile-time Language Integrated Queries for Scala|
|Spark Cassandra Connector||1,903||109||22||14 days ago||81||April 08, 2021||22||apache-2.0||Scala|
|DataStax Spark Cassandra Connector|
|Elassandra||1,633||1||41||a year ago||17||September 01, 2020||41||apache-2.0||Java|
|Elassandra = Elasticsearch + Apache Cassandra|
|Killrweather||1,174||6 years ago||23||apache-2.0||Scala|
|KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.|
|Dockerfiles||1,132||19 days ago||14||mit||Shell|
|50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak|
|Freestyle||615||34||4 years ago||19||June 07, 2018||38||apache-2.0||Scala|
|A cohesive & pragmatic framework of FP centric Scala libraries|
|Reference Apps||615||6 years ago||32||other||Scala|
|Spark reference applications|
|Cassandra Lucene Index||574||1||3 years ago||126||October 02, 2018||64||apache-2.0||Java|
|Lucene based secondary indexes for Cassandra|
|Data Engineering Projects||322||4 months ago||5||Jupyter Notebook|
|Personal Data Engineering Projects|
|Community||Chat with us at Datastax and Cassandra Q&A|
|Scala Docs||Most Recent Release (3.3.0): Spark-Cassandra-Connector, Spark-Cassandra-Connector-Driver|
|Latest Production Release||3.3.0|
Lightning-fast cluster computing with Apache Spark™ and Apache Cassandra®.
This library lets you expose Cassandra tables as Spark RDDs and Datasets/DataFrames, write Spark RDDs and Datasets/DataFrames to Cassandra tables, and execute arbitrary CQL queries in your Spark applications.
joinWithCassandraTablecall for RDDs, and optimizes join with data in Cassandra when using Datasets/DataFrames
The connector project has several branches, each of which map into different supported versions of Spark and Cassandra. For previous releases the branch is named "bX.Y" where X.Y is the major+minor version; for example the "b1.6" branch corresponds to the 1.6 release. The "master" branch will normally contain development for the next connector release in progress.
Currently, the following branches are actively supported: 3.3.x (master), 3.2.x (b3.2), 3.1.x (b3.1), 3.0.x (b3.0) and 2.5.x (b2.5).
|Connector||Spark||Cassandra||Cassandra Java Driver||Minimum Java Version||Supported Scala Versions|
|3.3||3.3||2.1.5*, 2.2, 3.x, 4.x||4.13||8||2.12|
|3.2||3.2||2.1.5*, 2.2, 3.x, 4.0||4.13||8||2.12|
|3.1||3.1||2.1.5*, 2.2, 3.x, 4.0||4.12||8||2.12|
|3.0||3.0||2.1.5*, 2.2, 3.x, 4.0||4.12||8||2.12|
|2.5||2.4||2.1.5*, 2.2, 3.x, 4.0||4.12||8||2.11, 2.12|
|2.4.2||2.4||2.1.5*, 2.2, 3.x||3.0||8||2.11, 2.12|
|2.4||2.4||2.1.5*, 2.2, 3.x||3.0||8||2.11|
|2.3||2.3||2.1.5*, 2.2, 3.x||3.0||8||2.11|
|2.0||2.0, 2.1, 2.2||2.1.5*, 2.2, 3.x||3.0||8||2.10, 2.11|
|1.6||1.6||2.1.5*, 2.2, 3.0||3.0||7||2.10, 2.11|
|1.5||1.5, 1.6||2.1.5*, 2.2, 3.0||3.0||7||2.10, 2.11|
|1.2||1.2||2.1, 2.0||2.1||7||2.10, 2.11|
|1.1||1.1, 1.0||2.1, 2.0||2.1||7||2.10, 2.11|
|1.0||1.0, 0.9||2.0||2.0||7||2.10, 2.11|
*Compatible with 2.1.X where X >= 5
API documentation for the Scala and Java interfaces are available online:
This project is available on the Maven Central Repository. For SBT to download the connector binaries, sources and javadoc, put this in your project SBT config:
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "3.3.0"
DataStax Academy provides free online training for Apache Cassandra and DataStax Enterprise. In DS320: Analytics with Spark, you will learn how to effectively and efficiently solve analytical problems with Apache Spark, Apache Cassandra, and DataStax Enterprise. You will learn about Spark API, Spark-Cassandra Connector, Spark SQL, Spark Streaming, and crucial performance optimization techniques.
New issues may be reported using JIRA. Please include all relevant details including versions of Spark, Spark Cassandra Connector, Cassandra and/or DSE. A minimal reproducible case with sample code is ideal.
Questions and requests for help may be submitted to the user mailing list.
The DataStax Community provides a free question and answer website for any and all questions relating to any DataStax Related technology. Including the Spark Cassandra Connector. Both DataStax engineers and community members frequent this board and answer questions.
To protect the community, all contributors are required to sign the DataStax Spark Cassandra Connector Contribution License Agreement. The process is completely electronic and should only take a few minutes.
To develop this project, we recommend using IntelliJ IDEA. Make sure you have installed and enabled the Scala Plugin. Open the project with IntelliJ IDEA and it will automatically create the project structure from the provided SBT configuration.
Tips for Developing the Spark Cassandra Connector
Checklist for contributing changes to the project:
To run unit and integration tests:
./sbt/sbt test ./sbt/sbt it:test
Note that the integration tests require CCM to be installed on your machine. See Tips for Developing the Spark Cassandra Connector for details.
By default, integration tests start up a separate, single Cassandra instance and run Spark in local mode. It is possible to run integration tests with your own Cassandra and/or Spark cluster. First, prepare a jar with testing code:
Then copy the generated test jar to your Spark nodes and run:
export IT_TEST_CASSANDRA_HOST=<IP of one of the Cassandra nodes> export IT_TEST_SPARK_MASTER=<Spark Master URL> ./sbt/sbt it:test
To generate the Reference Document use
./sbt/sbt spark-cassandra-connector-unshaded/run (outputLocation)
outputLocation defaults to doc/reference.md
Copyright 2014-2022, DataStax, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.