Awesome Open Source
Awesome Open Source

Dockerfiles for DevOps, CI/CD, Big Data & NoSQL

GitHub stars GitHub forks Contributors Lines of Code License GitHub Last Commit

Codacy CodeFactor Quality Gate Status Maintainability Rating Reliability Rating Security Rating

Linux Mac Docker StarTrack StarCharts

Alpine CentOS Debian Fedora Redhat Ubuntu

CI Builds Overview Jenkins Concourse GoCD TeamCity

CircleCI BuildKite AppVeyor Drone Codeship Status for HariSekhon/Dockerfiles Codefresh Cirrus CI Semaphore Wercker Buddy Shippable Travis CI

Azure DevOps Pipeline GitLab Pipeline BitBucket Pipeline AWS CodeBuild GCP Cloud Build

Repo on Azure DevOps Repo on GitHub Repo on GitLab Repo on BitBucket Dockerfiles ShellCheck JSON YAML XML Validation Semgrep Semgrep Cloud Kics

GitHub Actions Ubuntu Mac Mac 11 Mac 12 Ubuntu Ubuntu 14.04 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04 Debian Debian 8 Debian 9 Debian 10 CentOS CentOS 7 CentOS 8 Fedora Alpine Alpine 3

git.io/dockerhub

Contains 50+ DockerHub repos with 340+ tags, many different versions of standard official open source software, see Full Inventory futher down.

These docker images are tested by hundreds of tools and also used in the full functional test suites of various other GitHub repos.

See also the Kubernetes configs repo.

Overview - this repo contains:

  • Hadoop & Big Data ecosystem technologies (Spark, Kafka, Presto, Drill, Nifi, ZooKeeper)
  • NoSQL datastores (HBase, Cassandra, Riak, SolrCloud)
  • OS & development images (Alpine, CentOS, Debian, Fedora, Ubuntu)
  • DevOps, CI/CD (CircleCI, GitHub Actions, Jenkins, TeamCity etc), open source (RabbitMQ Cluster, Mesos, Consul)
  • My GitHub repos containing hundreds of tools related to these technologies with all dependencies pre-built in the docker images

These images are all available pre-built on My DockerHub - https://hub.docker.com/u/harisekhon/.

  • Quality and Testing - this repo has entire test suites run against it from various GitHub repositories to validate the docker images' functionality, branches vs tagged versions align, latest contains correct version from master branch, syntax checks covering all common build and file formats (Make/JSON/CSV/INI/XML/YAML configurations) etc.

These are reusable tests that can anybody can implement and can be found in my DevOps Python Tools and DevOps Bash Tools repos as well as the Advanced Nagios Plugins Collection which contains hundreds of technology specific API-level test programs to ensure the docker images are functioning as intended.

Continuous Integration in run on this and adjacent repos that form a bi-directional validation between these docker images and several other repositories full of hundreds of programs. All of this is intended to keep the quality of this repo as high as possible.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

My LinkedIn

(you're welcome to connect with me on LinkedIn)

Ready to run Docker images

docker search harisekhon
docker run harisekhon/nagios-plugins

To see more than the 25 DockerHub repos limited by docker search (docker issue 23055) I wrote dockerhub_search.py using the DockerHub API, available in my DevOps Python Tools github repo and as a pre-built docker image:

docker run harisekhon/pytools dockerhub_search.py harisekhon

There are lots of tagged versions of official software in my repos to allow development testing across multiple versions, usually more versions than available from the official repos (and new version updates available on request, just raise a GitHub issue).

DockerHub tags are not shown by docker search (docker issue 17238) so I wrote dockerhub_show_tags.py available in my DevOps Python Tools github repo and as a pre-built docker image - eg. to see an organized list of all CentOS tags:

docker run harisekhon/pytools dockerhub_show_tags.py centos

For service technologies like Hadoop, HBase, ZooKeeper etc for which you'll also want port mappings, each directory in the GitHub project contains both a standard docker-compose configuration as well as a make run shortcut (which doesn't require docker-compose to be installed) - either way you don't have to remember all the command line switches and port number specifics:

cd zookeeper
docker-compose up

or for technologies with interactive shells like Spark, ZooKeeper, HBase, Drill, Cassandra where you want to be dropped in to an interactive shell, use the make run shortcut instead:

cd zookeeper
make run

which is much easier to type and remember than the equivalent bigger commands like:

docker run -ti -p 2181:2181 harisekhon/zookeeper

and avoid this for more complex services like Hadoop / HBase:

docker run -ti -p 2181:2181 -p 8080:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p 16000:16000 -p 16010:16010 -p 16201:16201 -p 16301:16301 harisekhon/hbase
docker run -ti -p 8020:8020 -p 8032:8032 -p 8088:8088 -p 9000:9000 -p 10020:10020 -p 19888:19888 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075 -p 50090:50090 harisekhon/hadoop

Full Inventory

Official Standard Open Source Technologies

More specific information can be found in the readme page under each respective directory in the Dockerfiles git repo.

Repos suffixed with -dev are the official technologies + development & debugging tools + my github repos with all dependencies pre-built.

My GitHub Repos (with all libs + deps pre-built)
  • Advanced Nagios Plugins Collection - 450+ nagios plugins for every Hadoop distribution and every major NoSQL technology - Hadoop, Redis, Elasticsearch, Solr, HBase, Cassandra & DataStax OpsCenter, MongoDB, MySQL, Kafka, Riak, Memcached, Couchbase, CouchDB, Mesos, Spark, Neo4j, Datameer, H2O, WanDisco, Yarn, HDFS, Impala, Apache Drill, Presto, ZooKeeper, Cloudera, Hortonworks, MapR, IBM BigInsights, Infrastructure - Linux, DNS, Whois, SSL Certs etc
    • DockerHub Nagios Plugins Alpine
    • DockerHub Nagios Plugins Centos DockerHub Nagios Plugins Latest
    • DockerHub Nagios Plugins Debian
    • DockerHub Nagios Plugins Fedora
    • DockerHub Nagios Plugins Ubuntu
    • DockerHub Nagios Plugins Perl
    • DockerHub Nagios Plugins Python
  • harisekhon/tools - DevOps Tools superset of the below images, containing hundreds of programs:
    • DockerHub PyTools - DevOps Python Tools - 80+ DevOps CLI tools tools for AWS, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, Ambari, Blueprints, CloudFormation, Elasticsearch, Solr, Pig etc.
    • DockerHub Bash Tools - DevOps Bash Tools - 750+ DevOps CLI tools for AWS, GCP, Kubernetes, Hadoop, Hive, Impala, Kafka, Docker, LDAP, Git, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies
    • DockerHub Perl Tools - DevOps Perl Tools - 25+ DevOps CLI Tools - Log Anonymizer, Hadoop HDFS & Hive tools, Solr/SolrCloud CLI, SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Linux, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Ambari FreeIPA Kerberos, Datameer etc.
    • all of the above repos come with tags for alpine, centos, debian, fedora and ubuntu builds
  • Spotify Tools - Spotify API tools - eg. convert Spotify URIs to Artist - Track form by querying the Spotify API - readme

Github repos

DockerHub GitHub Ubuntu

My GitHub repo pre-built on major Linux distros with CLI programs located at /github/<project>

Available as both harisekhon/github:<distro> and harisekhon/<distro>-github for convenience, and to allow shorter use of :latest by using just harisekhon/github

harisekhon/github:latest is the same as harisekhon/github:ubuntu

Docker Build GitHub Alpine Docker Build GitHub CentOS Docker Build GitHub Debian Docker Build GitHub Fedora Docker Build GitHub Ubuntu

  • DockerHub GitHub Alpine DockerHub GitHub Alpine
  • DockerHub GitHub CentOS DockerHub GitHub CentOS
  • DockerHub GitHub Debian DockerHub GitHub Debian
  • DockerHub GitHub Fedora DockerHub GitHub Fedora
  • DockerHub GitHub Ubuntu DockerHub GitHub Ubuntu DockerHub GitHub Ubuntu

Base Images

Linux Distros + Development Tools

Available as both harisekhon/<distro>-dev and harisekhon/dev:<distro>

harisekhon/dev:latest is the same as harisekhon/dev:ubuntu

Docker Build Alpine Dev Docker Build CentOS Dev Docker Build Debian Dev Docker Build Fedora Dev Docker Build Ubuntu Dev

  • DockerHub Alpine Dev - Alpine latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
  • DockerHub CentOS Dev - CentOS latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect, EPEL etc.
  • DockerHub Debian Dev - Debian latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
  • DockerHub Fedora Dev - Fedora latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
  • DockerHub Ubuntu Dev - Ubuntu latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
Base Images of Java / Scala

All builds use OpenJDK with jre and jdk numbered tags. See this article below for why it might be illegal to bundle Oracle Java (and why no Linux distributions do this either):

https://www.javacodegeeks.com/2016/03/running-java-docker-youre-breaking-law.html

Docker Build Alpine Java Docker Build CentOS Java Docker Build Debian Java Docker Build Fedora Java Docker Build Ubuntu Java

  • Docker Build Alpine Java - Alpine latest with Java 8
  • Docker Build CentOS Java - CentOS latest combinations of Java 7 / 8 and Scala 2.10 / 2.11
  • Docker Build Debian Java - Debian latest with Java 7, 8
  • Docker Build Fedora Java - Fedora latest combinations of Java 7/8 and Scala 2.10/2.11
  • Docker Build Ubuntu Java
    • Ubuntu 14.04 with Java 7
    • Ubuntu latest with Java 8, 9

Build from Source

All images come pre-built on DockerHub but if you want to compile from source for any reason such as developing improvements, I've made this easy to do:

git clone https://github.com/HariSekhon/Dockerfiles

cd Dockerfiles

To build all Docker images, just run the make command at the top level:

make

To build a specific Docker image, enter its directory and run make:

cd nagios-plugins

make

You can also build a specific version by checking out the git branch for the version and running the build:

cd consul
git checkout consul-0.9
make

or build all versions of a given software project like so:

cd hadoop
make build-versions

See the top level Makefile as well as the Makefile.in which is sourced per project with any project specific overrides in the <project_directory>/Makefile.

Support

Please raise tickets for issues and improvements at https://github.com/HariSekhon/Dockerfiles/issues

Related Repositories

  • DevOps Bash Tools - 800+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

  • SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

  • Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

  • Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

  • Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

  • DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

  • DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

  • The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

  • HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

Stargazers over time

Stargazers over time

git.io/dockerhub



Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Shell (182,340
Docker (99,630
Kubernetes (25,941
Spark (10,794
Kafka (9,776
Rabbitmq (6,385
Hadoop (5,353
Repos (4,008
Zookeeper (3,630
Cassandra (3,299
Solr (2,993
Consul (2,956
Hbase (1,595
Presto (459
Nagios Plugin (342
Nifi (129
Solrcloud (38
Apache Drill (29
Rabbitmq Cluster (17