machine learning for genomic variants
Alternatives To Variantspark
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Data Science Ipython Notebooks25,025
22 days ago33otherPython
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Dev Setup5,802
9 months ago34otherPython
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
Seldon Server1,420
3 years ago44June 28, 201726apache-2.0Java
Machine Learning Platform and Recommendation Engine built on Kubernetes
Aws Serverless Java Container1,3508915a day ago30June 10, 202247apache-2.0Java
A Java wrapper to run Spring, Jersey, Spark, and other apps inside AWS Lambda.
Aws Glue Samples1,282
4 days ago36mit-0Python
AWS Glue code samples
Devops Python Tools651
18 days ago30mitPython
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
49 months ago13June 13, 202136otherPython
A command-line tool for launching Apache Spark clusters.
Aws Glue Libs514
5 months ago80otherPython
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Spark Redshift514414 years ago10November 01, 2016134apache-2.0Scala
Redshift data source for Apache Spark
2 months ago7mitJupyter Notebook
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Alternatives To Variantspark
Select To Compare

Alternative Project Comparisons

Variant Spark

Build Documentation Status

variant-spark is a scalable toolkit for genome-wide association studies optimized for GWAS like datasets.

Machine learning methods and, in particular, random forests (RFs) are a promising alternative to standard single SNP analyses in genome-wide association studies (GWAS). RFs provide variable importance measures to rank SNPs according to their predictive power. Although there are number of existing random forest implementations available, some even parallel or distributed such as: Random Jungle, ranger or SparkML, most of them are not optimized to deal with GWAS datasets, which usually come with thousands of samples and millions of variables.

variant-spark currently provides the basic functionality of building random forest model and estimating variable importance with mean decrease gini method and can operate on VCF and CSV files. Future extensions will include support of other importance measures, variable selection methods and data formats.

variant-spark utilizes a novel approach of building random forest from data in transposed representation, which allows it to efficiently deal with even extremely wide GWAS datasets. Moreover, since the most common genomics variant calls VCF and uses the transposed representation, variant-spark can work directly with the VCF data, without the costly pre-processing required by other tools.

variant-spark is built on top of Apache Spark – a modern distributed framework for big data processing, which gives variant-spark the ability to to scale horizontally on both bespoke cluster and public clouds.

The potential users include:

  • Medical researchers seeking to perform GWAS-like analysis on large cohort data of genome wide sequencing data or imputed SNP array data.
  • Medical researchers or clinicians seeking to perform clustering on genomic profiles to stratify large-cohort genomic data
  • General researchers with classification or clustering needs of datasets with millions of features.


Please feel free to add issues and/or upvote issues you care about. Also join the Gitter chat. We also started ReadTheDocs and there is always the this repo's issues page for you to add requests. Thanks for your support.

Learn More

To learn more watch this video from YOW! Brisbane 2017.

variant-spark YOW! Brisbane 2017


variant-spark requires java jdk 1.8+ and maven 3+

In order to build the binaries use:

mvn clean install

For python variant-spark requires python 3.6+ with pip. The other packages required for development are listed in dev/dev-requirements.txt and can be installed with:

pip install -r dev/dev-requirements.txt

or with:


The complete built including all check can be run with:



variant-spark requires an existing spark 3.1+ installation (either a local one or a cluster one).

To run variant-spark use:

./variant-spark [(--spark|--local) <spark-options>* --] [<command>] <command-options>*

In order to obtain the list of the available commands use:

./variant-spark -h

In order to obtain help for a specific command (for example importance) use:

./variant-spark importance -h

You can use --spark marker before the command to pass spark-submit options to variant-spark. The list of spark options needs to be terminated with --, e.g:

./variant-spark --spark --master yarn-client --num-executors 32 -- importance ....

Please, note that --spark needs to be the first argument of variant-spark

You can also run variant-spark in the --local mode. In this mode variant-spark will ignore any Hadoop or Spark configuration files and run in the local mode for both Hadoop and Spark. In particular in this mode all file paths are interpreted as local file system paths. Also any parameters passed after --local and before -- are ignored. For example:

./variant-spark --local -- importance  -if data/chr22_1000.vcf -ff data/chr22-labels.csv -fc 22_16051249 -v -rn 500 -rbs 20 -ro


The difference between running in --local mode and in --spark with local master is that in the latter case Spark uses the hadoop filesystem configuration and the input files need to be copied to this filesystem (e.g. HDFS) Also the output will be written to the location determined by the hadoop filesystem settings. In particular paths without schema e.g. 'output.csv' will be resolved with the hadoop default filesystem (usually HDFS) To change this behavior you can set the default filesystem in the command line using option. For example to use local filesystem as the default use:

./variant-spark --spark ... --conf "" ... -- importance  ... -of output.csv

You can also use the full URI with the schema to address any filesystem for both input and output files e.g.:

./variant-spark --spark ... --conf "" ... -- importance  -if hdfs:///user/data/input.csv ... -of output.csv

Running examples

There are multiple methods for running variant-spark examples

Manual Examples

variant-spark comes with a few example scripts in the scripts directory that demonstrate how to run its commands on sample data .

There is a few small data sets in the data directory suitable for running on a single machine. For example


runs variable importance command on a small sample of the chromosome 22 vcf file (from 1000 Genomes Project)

The full size examples require a cluster environment (the scripts are configured to work with Spark on YARN).

The data required for the examples can be obtained from:

This repository uses the git Large File Support extension, which needs to be installed first (see:

Clone the variant-spark-data repository and then to install the test data into your hadoop filesystem use:


By default the sample data will installed into the variant-spark-data\input sub directory of your HDFS home directory.

You can choose a different location by setting the VS_DATA_DIR environment variable.

After the test data has been successfully copied to HDFS you can run examples scripts, e.g.:


Note: if you installed the data to a non default location the VS_DATA_DIR needs to be set accordingly when running the examples

VariantSpark on the cloud

VariantSpark can easily be used in AWS and Azure. For more examples and information, check the cloud folder. For a quick start, check the few pointers below.

AWS Marketplace

VariantSpark is now available on AWS Marketplace. Please read the Guidlines for specification and step-by-step instructions.

Azure Databricks

VariantSpark can be easily deployed in Azure Databricks through the button below. Please read the VariantSpark azure manual for specification and step-by-step instructions.

Deploy to Azure



JsonRfAnalyser is a python program that looks into the JSON RandomForest model and list variables on each tree and branch. Please read README to see the complete list of functionalities.


rfview.html is a web program (run locally on your machine) where you can upload the json model produced by variantspark and it visualises trees in the model. You can identify which tree to be visualised. Node color and node labels could be set to different parameters such as number of samples in the node or the node impurity. It uses vis.js for tree Visualisation.

Popular Amazon Web Services Projects
Popular Spark Projects
Popular Cloud Computing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Random Forest