Spark_gce
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Ingress Gce | 1,232 | 2 | a day ago | 175 | August 30, 2023 | 53 | apache-2.0 | Go | ||
| Ingress controller for Google Cloud | ||||||||||
| Terraform Google Examples | 382 | 4 years ago | 6 | apache-2.0 | HCL | |||||
| Collection of examples for using Terraform with Google Cloud Platform. | ||||||||||
| Compute Image Packages | 365 |  | 108 | 16 | 2 years ago | 38 | January 24, 2019 | 20 | apache-2.0 | Python |
| Packages for Google Compute Engine Linux images. | ||||||||||
| Vagrant Google | 328 | 3 months ago | 32 | apache-2.0 | Ruby | |||||
| Vagrant provider for GCE. | ||||||||||
| Compute Archlinux Image Builder | 257 | 10 months ago | 2 | apache-2.0 | Shell | |||||
| A tool to build a Arch Linux Image for GCE | ||||||||||
| Ccat | 238 | 4 years ago | 2 | bsd-3-clause | Python | |||||
| Cloud Container Attack Tool (CCAT) is a tool for testing security of container environments. | ||||||||||
| Cloud Bigtable Examples | 226 | 3 months ago | 3 | February 01, 2022 | 6 | apache-2.0 | Java | |||
| Examples of how to use Cloud Bigtable both with GCE map/reduce as well as stand alone applications. | ||||||||||
| Compute Image Tools | 187 | 5 | 14 days ago | 85 | April 24, 2021 | 49 | apache-2.0 | Go | ||
| Tools and scripts for Google Compute Engine images. | ||||||||||
| Nodejs Compute | 115 |  | 2 | 2 months ago | 1 | February 11, 2019 | apache-2.0 | |||
| This repository is deprecated. All of its content and history has been moved to googleapis/google-cloud-node. | ||||||||||
| Compute Image Windows | 98 | a month ago | 1 | February 04, 2021 | 14 | apache-2.0 | PowerShell | |||
| Windows agents and scripts for Google Compute Engine images. | ||||||||||
Spark GCE
Spark GCE is like Spark Ec2 but for those who run their cluster on Google Cloud.
- Make Sure you have installed and authenticated gcutils where you are running this script.
- Helps you launch a spark cluster in the Google Cloud
- Attaches 500GB empty disk to all nodes in the cluster
- Installs and Configures everything Automatically
- Starts the Shark server Automatically
Spark GCE is a python script which will help you launch a spark cluster in the google cloud like the way spark_ec2 script does for AWS.
Usage
spark_gce.py project-name number-of-slaves slave-type master-type identity-file zone cluster-name
project-id: Project ID of the project where you are going to launch your spark cluster.
number-of-slave: Number of slaves that you want to launch.
slave-type: Instance type for the slave machines.
master-type: Instance type for the master node.
identity-file: Identity file to authenticate with your GCE instances, Usually resides at ~/.ssh/google_compute_engine once you authenticate using gcutils.
zone: Specify the zone where you are going to launch the cluster.
cluster-name: Name the cluster that you are going to launch.
spark_gce.py project-name cluster-name destroy
- project-id: Project id of the project where the spark cluster is at.
- cluster-name: Name of the cluster that you are going to destroy.
Installation
git clone https://github.com/sigmoidanalytics/spark_gce.git
cd spark_gce
python spark_gce.py
Need Help?
- Drop us an email: [email protected]
- Read Our Blog: http://www.sigmoidanalytics.com/spark-gce/
- Read Our Wiki: http://docs.sigmoidanalytics.com/index.php/SparkGCE
