Hadoop Connectors
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Rclone | 38,652 | 21 | 2 days ago | 254 | September 15, 2022 | 897 | mit | Go | ||
| "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files | ||||||||||
| Analytics | 14,972 | a day ago | 40 | agpl-3.0 | Elixir | |||||
| Simple, open-source, lightweight (< 1 KB) and privacy-friendly web analytics alternative to Google Analytics. | ||||||||||
| Awesome Kubernetes | 13,893 | 14 days ago | 9 | other | Shell | |||||
| A curated list for awesome kubernetes sources :ship::tada: | ||||||||||
| Xg2xg | 12,414 | 8 days ago | 36 | |||||||
| by ex-googlers, for ex-googlers - a lookup table of similar tech & services | ||||||||||
| Functions Samples | 11,675 | a day ago | 143 | apache-2.0 | JavaScript | |||||
| Collection of sample apps showcasing popular use cases using Cloud Functions for Firebase | ||||||||||
| Infracost | 9,117 | 5 days ago | 129 | August 30, 2022 | 138 | apache-2.0 | Go | |||
| Cloud cost estimates for Terraform in pull requests💰📉 Love your cloud bill! | ||||||||||
| Training Data Analyst | 6,795 | 6 days ago | 325 | apache-2.0 | Jupyter Notebook | |||||
| Labs and demos for courses for GCP Training (http://cloud.google.com/training). | ||||||||||
| Python Docs Samples | 6,320 | a day ago | 2 | May 24, 2021 | 106 | apache-2.0 | Jupyter Notebook | |||
| Code samples used on cloud.google.com | ||||||||||
| Google Cloud Python | 4,212 |  | 438 | 77 | a day ago | 20 | July 30, 2018 | 234 | apache-2.0 | Python |
| Google Cloud Client Library for Python | ||||||||||
| Apps Script Samples | 3,893 | a month ago | 36 | apache-2.0 | JavaScript | |||||
| Apps Script samples for Google Workspace products. | ||||||||||
Apache Hadoop Connectors
Libraries and tools for interoperability between Apache Hadoop related open-source software and Google Cloud Platform.
Google Cloud Storage connector for Apache Hadoop (HCFS)
The Google Cloud Storage connector for Hadoop enables running MapReduce jobs directly on data in Google Cloud Storage by implementing the Hadoop FileSystem interface. For details, see the README.
Building the Cloud Storage connector
Note that build requires Java 11+ and fails with older Java versions.
To build the connector for specific Hadoop version, run the following commands from the main directory:
./mvnw clean package
In order to verify test coverage for specific Hadoop version, run the following commands from the main directory:
./mvnw -P coverage clean verify
The Cloud Storage connector JAR can be found in gcs/target/ directory.
Adding the Cloud Storage connector to your build
Maven group ID is com.google.cloud.bigdataoss and artifact ID for Cloud
Storage connector is gcs-connector.
To add a dependency on Cloud Storage connector using Maven, use the following:
<dependency>
<groupId>com.google.cloud.bigdataoss</groupId>
<artifactId>gcs-connector</artifactId>
<version>hadoop3-2.2.10</version>
</dependency>
Resources
On Stack Overflow, use the tag
google-cloud-dataproc
for questions about the connectors in this repository. This tag receives
responses from the Stack Overflow community and Google engineers, who monitor
the tag and offer unofficial support.
