|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Hadoop Connectors||267||22||46||2 days ago||578||December 12, 2022||50||apache-2.0||Java|
|Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.|
|Spydra||132||a year ago||20||December 08, 2020||12||apache-2.0||Java|
|Ephemeral Hadoop clusters using Google Compute Platform|
|Bdutil||114||4 years ago||32||apache-2.0||Shell|
|[DEPRECATED] Script used to manage Hadoop and Spark instances on Google Compute Engine|
|Solutions Google Compute Engine Cluster For Hadoop||81||5 years ago||8||apache-2.0||Python|
|This sample app will get up and running quickly with a Hadoop cluster on Google Compute Engine. For more information on running Hadoop on GCE, read the papers at https://cloud.google.com/resources/.|
|Data Pipeline||79||9 years ago||2||apache-2.0||Python|
|Data pipeline is a tool to run Data loading pipelines. It is an open sourced app engine app that users can extend to suit their own needs. Out of the box it will load files from a source, transform them and then output them (output might be writing to a file or loading them into a data analysis tool). It is designed to be modular and support various sources, transformation technologies and output types. The transformations can be chained together to form complex pipelines.|
|Compute Hadoop Java Python||28||8 years ago||1||apache-2.0||Python|
|This software demonstrates one way to create and manage a cluster of Hadoop nodes running on Google Compute Engine.|
|Solutions Apache Hive And Pig On Google Compute Engine||19||5 years ago||apache-2.0||Shell|
|This sample app will get up and running quickly with Hive and/or Pig on a Hadoop cluster on Google Compute Engine. For more information on running Hadoop on GCE, read the papers at https://cloud.google.com/resources/.|
|Hive Bigquery Storage Handler||16||a year ago||8||apache-2.0||Java|
|Hive Storage Handler for interoperability between BigQuery and Apache Hive|
|Google Cloud Dataproc is a managed Apache Spark and Apache Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning.|
|Java Dataproc||13||7||9||2 days ago||137||May 25, 2022||2||apache-2.0||Java|
Java idiomatic client for Dataproc.
🚌 In October 2022, this library has moved to
This repository will be archived in the future.
Future releases will appear in the new repository (https://github.com/googleapis/google-cloud-java/releases).
The Maven artifact coordinates (
com.google.cloud:google-cloud-dataproc) remain the same.
If you are using Maven with BOM, add this to your pom.xml file:
<dependencyManagement> <dependencies> <dependency> <groupId>com.google.cloud</groupId> <artifactId>libraries-bom</artifactId> <version>26.1.3</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>com.google.cloud</groupId> <artifactId>google-cloud-dataproc</artifactId> </dependency> </dependencies>
If you are using Maven without BOM, add this to your dependencies:
<dependency> <groupId>com.google.cloud</groupId> <artifactId>google-cloud-dataproc</artifactId> <version>4.0.8</version> </dependency>
If you are using Gradle 5.x or later, add this to your dependencies:
implementation platform('com.google.cloud:libraries-bom:26.1.4') implementation 'com.google.cloud:google-cloud-dataproc'
If you are using Gradle without BOM, add this to your dependencies:
If you are using SBT, add this to your dependencies:
libraryDependencies += "com.google.cloud" % "google-cloud-dataproc" % "4.2.0"
See the Authentication section in the base directory's README.
The client application making API calls must be granted authorization scopes required for the desired Dataproc APIs, and the authenticated principal must have the IAM role(s) required to access GCP resources using the Dataproc API calls.
You will need a Google Cloud Platform Console project with the Dataproc API enabled.
You will need to enable billing to use Google Dataproc.
Follow these instructions to get your project set up. You will also need to set up the local development environment by
installing the Google Cloud SDK and running the following commands in command line:
gcloud auth login and
gcloud config set project [YOUR PROJECT ID].
You'll need to obtain the
google-cloud-dataproc library. See the Quickstart section
google-cloud-dataproc as a dependency in your code.
Dataproc is a faster, easier, more cost-effective way to run Apache Spark and Apache Hadoop.
See the Dataproc client library docs to learn how to use this Dataproc Client Library.
Samples are in the
|Sample||Source Code||Try it|
|Create Cluster||source code|
|Create Cluster With Autoscaling||source code|
|Instantiate Inline Workflow Template||source code|
|Submit Hadoop Fs Job||source code|
|Submit Job||source code|
To get help, follow the instructions in the shared Troubleshooting document.
Dataproc uses gRPC for the transport layer.
Java 8 or above is required for using this client.
Google's Java client libraries, Google Cloud Client Libraries and Google Cloud API Libraries, follow the Oracle Java SE support roadmap (see the Oracle Java SE Product Releases section).
In general, new feature development occurs with support for the lowest Java LTS version covered by Oracle's Premier Support (which typically lasts 5 years from initial General Availability). If the minimum required JVM for a given library is changed, it is accompanied by a semver major release.
Java 11 and (in September 2021) Java 17 are the best choices for new development.
Google tests its client libraries with all current LTS versions covered by Oracle's Extended Support (which typically lasts 8 years from initial General Availability).
Google's client libraries support legacy versions of Java runtimes with long term stable libraries that don't receive feature updates on a best efforts basis as it may not be possible to backport all patches.
Google provides updates on a best efforts basis to apps that continue to use Java 7, though apps might need to upgrade to current versions of the library that supports their JVM.
The latest versions and the supported Java versions are identified on
the individual GitHub repository
and on google-cloud-java.
This library follows Semantic Versioning.
Contributions to this library are always welcome and highly encouraged.
See CONTRIBUTING for more information how to get started.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.
Apache 2.0 - See LICENSE for more information.
|Java 8 OSX|
|Java 8 Windows|
Java is a registered trademark of Oracle and/or its affiliates.