Spark Sql On Hbase

Native, optimized access to HBase Data through Spark SQL/Dataframe Interfaces
Alternatives To Spark Sql On Hbase
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Bigdata Notes13,291
4 months ago33Java
大数据入门指南 :star:
Flink Learning13,198
3 months agoapache-2.0Java
flink learning blog. 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例,还有 Flink 落地应用的大型项目案例(PVUV、日志存储、百亿数据实时去重、监控告警)分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Technology Talk13,004
3 months ago10
God Of Bigdata7,992
2 months ago2
Spring Boot Quick2,152
2 months ago12Java
:herb: 基于springboot的快速学习示例,整合自己遇到的开源框架,如:rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、k3s、k3d、k8s、mybatis加解密插件、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等:pushpin:
2 days agoJava
Szt Bigdata1,702
6 months ago15otherScala
Gaffer1,701421a day ago94July 11, 2022115apache-2.0Java
A large-scale entity and relation database supporting aggregation of properties
Bigdata Interview1,397
2 years agon,ull
:dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
16 days ago14mitShell
50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak
Alternatives To Spark Sql On Hbase
Select To Compare

Alternative Project Comparisons

Astro: Fast SQL on HBase using SparkSQL

Apache HBase is a distributed Key-Value store of data on HDFS. It is modeled after Google’s Big Table, and provides APIs to query the data. The data is organized, partitioned and distributed by its “row keys”. Per partition, the data is further physically partitioned by “column families” that specify collections of “columns” of data. The data model is for wide and sparse tables where columns are dynamic and may well be sparse.

Although HBase is a very useful big data store, its access mechanism is very primitive and only through client-side APIs, Map/Reduce interfaces and interactive shells. SQL accesses to HBase data are available through Map/Reduce or interfaces mechanisms such as Apache Hive and Impala, or some “native” SQL technologies like Apache Phoenix. While the former is usually cheaper to implement and use, their latencies and efficiencies often cannot compare favorably with the latter and are often suitable only for offline analysis. The latter category, in contrast, often performs better and qualifies more as online engines; they are often on top of purpose-built execution engines.

Currently Spark supports queries against HBase data through HBase’s Map/Reduce interface (i.e., TableInputFormat). Spark SQL supports use of Hive data, which theoretically should be able to support HBase data access, out-of-box, through HBase’s Map/Reduce interface and therefore falls into the first category of the “SQL on HBase” technologies.

We believe, as an unified big data processing engine, Spark is in good position to provide better HBase support.

Online Documentation

Online documentation


This version of 1.0.0 requires Spark 1.4.0.

Building Spark HBase

Spark HBase is built using Apache Maven.

I. Clone and build Huawei-Spark/Spark-SQL-on-HBase

$ git clone spark-hbase

II. Go to the root of the source tree

$ cd spark-hbase

III. Build the project Build without testing

$ mvn -DskipTests clean install 

Or, build with testing. It will run test suites against a HBase minicluster.

$ mvn clean install

Activate Coprocessor and Custom Filter in HBase

First, add the path of spark-hbase jar to the in $HBASE_HOME/conf directory, as follows:


Then, register the coprocessor service 'CheckDirEndPoint' to hbase-site.xml in the same directory, as follows:


(Warning: Don't register another coprocessor service 'SparkSqlRegionObserver' here !)

Interactive Scala Shell

The easiest way to start using Spark HBase is through the Scala shell:


Python Shell

First, add the spark-hbase jar to the SPARK_CLASSPATH in the $SPARK_HOME/conf directory, as follows:


Then go to the spark-hbase installation directory and issue


A successfull message is as follows:

You are using Spark SQL on HBase!!! HBaseSQLContext available as hsqlContext.

To run a python script, the PYTHONPATH environment should be set to the "python" directory of the Spark-HBase installation. For example,

export PYTHONPATH=/root-of-Spark-HBase/python

Note that the shell commands are not included in the Zip file of the Spark release. They are for developers' use only for this version of 1.0.0. Instead, users can use "$SPARK_HOME/bin/spark-shell --packages Huawei-Spark/Spark-SQL-on-HBase:1.0.0" for SQL shell or "$SPARK_HOME/bin/pyspark --packages Huawei-Spark/Spark-SQL-on-HBase:1.0.0" for Pythin shell.

Running Tests

Testing first requires building Spark HBase. Once Spark HBase is built ...

Run all test suites from Maven:

mvn -Phbase,hadoop-2.4 test

Run a single test suite from Maven, for example:

mvn -Phbase,hadoop-2.4 test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite

IDE Setup

We use IntelliJ IDEA for Spark HBase development. You can get the community edition for free and install the JetBrains Scala plugin from Preferences > Plugins.

To import the current Spark HBase project for IntelliJ:

  1. Download IntelliJ and install the Scala plug-in for IntelliJ. You may also need to install Maven plug-in for IntelliJ.
  2. Go to "File -> Import Project", locate the Spark HBase source directory, and select "Maven Project".
  3. In the Import Wizard, select "Import Maven projects automatically" and leave other settings at their default.
  4. Make sure some specific profiles are enabled. Select corresponding Hadoop version, "maven3" and also"hbase" in order to get dependencies.
  5. Leave other settings at their default and you should be able to start your development.
  6. When you run the scala test, sometimes you will get out of memory exception. You can increase your VM memory usage by the following setting, for example:
-XX:MaxPermSize=512m -Xmx3072m

You can also make those setting to be the default by setting to the "Defaults -> ScalaTest".


Please refer to the Configuration guide in the online documentation for an overview on how to configure Spark.

Popular Spark Projects
Popular Hbase Projects
Popular Data Processing Categories

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.