Spark Redis
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Flink Learning | 13,198 | a month ago | apache-2.0 | Java | ||||||
| flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》 | ||||||||||
| Dev Setup | 5,802 | 7 months ago | 34 | other | Python | |||||
| macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults. | ||||||||||
| Technical Books | 4,578 | 8 months ago | cc-by-sa-4.0 | HTML | ||||||
| 😆 国内外互联网技术大牛们都写了哪些书籍：计算机基础、网络、前端、后端、数据库、架构、大数据、深度学习... | ||||||||||
| Pipeline | 4,140 | 5 months ago | 85 | July 18, 2017 | 1 | apache-2.0 | Jsonnet | |||
| PipelineAI Kubeflow Distribution | ||||||||||
| Szt Bigdata | 1,702 | 4 months ago | 15 | other | Scala | |||||
| 深圳地铁大数据客流分析系统🚇🚄🌟 | ||||||||||
| Tutorial | 1,120 | 7 days ago | 40 | Shell | ||||||
| 后端 （Java Golang）全栈知识架构体系总结 | ||||||||||
| Spark Redis | 885 | 1 | 6 months ago | 10 | March 09, 2020 | 121 | bsd-3-clause | Scala | ||
| A connector for Spark that allows reading and writing to/from Redis cluster | ||||||||||
| Hadoop_study | 817 | a year ago | 21 | Java | ||||||
| 定期更新Hadoop生态圈中常用大数据组件文档 重心依次为: Flink Solr Sparksql ES Scala Kafka Hbase/phoenix Redis Kerberos (项目包含hadoop思维导图 印象笔记 Scala版本简单demo 常用工具类 去敏后的train code 持续更新!!!) | ||||||||||
| Pkpmspark | 697 | 3 years ago | Scala | |||||||
| awesome 三维数据挖掘 数据分析 & 推荐 | ||||||||||
| Freestyle | 615 | 34 | 4 years ago | 19 | June 07, 2018 | 38 | apache-2.0 | Scala | ||
| A cohesive & pragmatic framework of FP centric Scala libraries | ||||||||||
Spark-Redis
A library for reading and writing data in Redis using Apache Spark.
Spark-Redis provides access to all of Redis' data structures - String, Hash, List, Set and Sorted Set - from Spark as RDDs. It also supports reading and writing with DataFrames and Spark SQL syntax.
The library can be used both with Redis stand-alone as well as clustered databases. When used with Redis cluster, Spark-Redis is aware of its partitioning scheme and adjusts in response to resharding and node failure events.
Spark-Redis also supports Spark Streaming (DStreams) and Structured Streaming.
Version compatibility and branching
The library has several branches, each corresponds to a different supported Spark version. For example, 'branch-2.3' works with any Spark 2.3.x version. The master branch contains the recent development for the next release.
| Spark-Redis | Spark | Redis | Supported Scala Versions |
|---|---|---|---|
| master | 3.2.x | >=2.9.0 | 2.12 |
| 3.0 | 3.0.x | >=2.9.0 | 2.12 |
| 2.4, 2.5, 2.6 | 2.4.x | >=2.9.0 | 2.11, 2.12 |
| 2.3 | 2.3.x | >=2.9.0 | 2.11 |
| 1.4 | 1.4.x | 2.10 |
Known limitations
- Java, Python and R API bindings are not provided at this time
Additional considerations
This library is a work in progress so the API may change before the official release.
Documentation
Please make sure you use documentation from the correct branch (2.4, 2.3, etc).
- Getting Started
- RDD
- Dataframe
- Streaming
- Structured Streaming
- Cluster
- Java
- Python
- Configuration
- Dev environment
Contributing
You're encouraged to contribute to the Spark-Redis project.
There are two ways you can do so:
Submit Issues
If you encounter an issue while using the library, please report it via the project's issues tracker.
Author Pull Requests
Code contributions to the Spark-Redis project can be made using pull requests. To submit a pull request:
- Fork this project.
- Make and commit your changes.
- Submit your changes as a pull request.