DevOps, Cloud, Big Data, NoSQL, Python & Linux tools. All programs have
repos which contains hundreds more scripts and programs for Cloud, Big Data, SQL, NoSQL, Web and Linux.
Cloud & Big Data Contractor, United Kingdom
make updateif updating and not just
git pullas you will often need the latest library submodule and possibly new upstream libraries.
All programs and their pre-compiled dependencies can be found ready to run on DockerHub.
List all programs:
docker run harisekhon/pytools
Run any given program:
docker run harisekhon/pytools <program> <args>
installs git, make, pulls the repo and build the dependencies:
curl -L https://git.io/python-bootstrap | sh
git clone https://github.com/harisekhon/devops-python-tools pytools cd pytools make
To only install pip dependencies for a single script, you can just type make and the filename with a
.pyc extension instead of
Make sure to read Detailed Build Instructions further down for more information.
Some Hadoop tools with require Jython, see Jython for Hadoop Utils for details.
All programs come with a
--help switch which includes a program description and the list of command line options.
Environment variables are supported for convenience and also to hide credentials from being exposed in the process list eg.
$TRAVIS_TOKEN. These are indicated in the
--help descriptions in brackets next to each option and often have more specific overrides with higher precedence eg.
$HBASE_HOST take priority over
anonymize.py- anonymizes your configs / logs from files or stdin (for pasting to Apache Jira tickets or mailing lists)
anonymize_custom.conf- put regex of your Name/Company/Project/Database/Tables to anonymize to
--ip-prefixleaves the last IP octect to aid in cluster debugging to still see differentiated nodes communicating with each other to compare configs and log communications
--hash-hostnames- hashes hostnames to look like Docker temporary container ID hostnames so that vendors support teams can differentiate hosts in clusters
anonymize_parallel.sh- splits files in to multiple parts and runs
anonymize.pyon each part in parallel before re-joining back in to a file of the same name with a
.anonymizedsuffix. Preserves order of evaluation important for anonymization rules, as well as maintaining file content order. On servers this parallelization can result in a 30x speed up for large log files
find_duplicate_files.py- finds duplicate files in one or more directory trees via multiple methods including file basename, size, MD5 comparison of same sized files, or bespoke regex capture of partial file basename
find_active_server.py- finds fastest responding healthy server or active master in high availability deployments, useful for scripting against clustered technologies (eg. Elasticsearch, Hadoop, HBase, Cassandra etc). Multi-threaded for speed and highly configurable - socket, http, https, ping, url and/or regex content match. See further down for more details and sub-programs that simplify usage for many of the most common cluster technologies
welcome.py- cool spinning welcome message greeting your username and showing last login time and user to put in your shell's
.profile(there is also a perl version in my DevOps Perl Tools repo)
aws_users_access_key_age.py- lists all users access keys, status, date of creation and age in days. Optionally filters for active keys and older than N days (for key rotation governance)
aws_users_unused_access_keys.py- lists users access keys that haven't been used in the last N days or that have never been used (these should generally be removed/disabled). Optionally filters for only active keys
aws_users_last_used.py- lists all users and their days since last use across both passwords and access keys. Optionally filters for users not used in the last N days to find old accounts to remove
aws_users_pw_last_used.py- lists all users and dates since their passwords were last used. Optionally filters for users with passwords not used in the last N days
gcp_service_account_credential_keys.py- lists all GCP service account credential keys for a given project with their age and expiry details, optionally filtering by non-expiring, already expired, or will expire within N days
quay_show_tags.py- shows tags for docker repos in a docker registry or on DockerHub or Quay.io - Docker CLI doesn't support this yet but it's a very useful thing to be able to see live on the command line or use in shell scripts (use
--quietto return only the tags for easy shell scripting). You can use this to pre-download all tags of a docker image before running tests across versions in a simple bash for loop, eg.
dockerhub_search.py- search DockerHub with a configurable number of returned results (older official
docker searchwas limited to only 25 results), using
--verbosewill also show you how many results were returned to the termainal and how many DockerHub has in total (use
-q / --quietto return only the image names for easy shell scripting). This can be used to download all of my DockerHub images in a simple bash for loop eg.
docker_pull_all_images.shand can be chained with
dockerhub_show_tags.pyto download all tagged versions for all docker images eg.
dockerfiles_check_git*.py- check Git tags & branches align with the containing Dockerfile's
spark_avro_to_parquet.py- PySpark Avro => Parquet converter
spark_parquet_to_avro.py- PySpark Parquet => Avro converter
spark_csv_to_avro.py- PySpark CSV => Avro converter, supports both inferred and explicit schemas
spark_csv_to_parquet.py- PySpark CSV => Parquet converter, supports both inferred and explicit schemas
spark_json_to_avro.py- PySpark JSON => Avro converter
spark_json_to_parquet.py- PySpark JSON => Parquet converter
xml_to_json.py- XML to JSON converter
json_to_xml.py- JSON to XML converter
json_to_yaml.py- JSON to YAML converter
json_docs_to_bulk_multiline.py- converts json files to bulk multi-record one-line-per-json-document format for pre-processing and loading to big data systems like Hadoop and MongoDB, can recurse directory trees, and mix json-doc-per-file / bulk-multiline-json / directories / standard input, combines all json documents and outputs bulk-one-json-document-per-line to standard output for convenient command line chaining and redirection, optionally continues on error, collects broken records to standard error for logging and later reprocessing for bulk batch jobs, even supports single quoted json while not technically valid json is used by MongoDB and even handles embedded double quotes in 'single quoted json'
yaml_to_json.py- YAML to JSON converter (because some APIs like GitLab CI Validation API require JSON)
validate_*.pyfurther down for all these formats and more
ambari_blueprints.py- Blueprint cluster templating and deployment tool using Ambari API
ambari_blueprints/directory for a variety of Ambari blueprint templates generated by and deployable using this tool
ambari_ams_*.sh- query the Ambari Metrics Collector API for a given metrics, list all metrics or hosts
ambari_cancel_all_requests.sh- cancel all ongoing operations using the Ambari API
ambari_trigger_service_checks.py- trigger service checks using the Ambari API
hdfs_find_replication_factor_1.py- finds HDFS files with replication factor 1, optionally resetting them to replication factor 3 to avoid missing block alerts during datanode maintenance windows
hdfs_time_block_reads.jy- HDFS per-block read timing debugger with datanode and rack locations for a given file or directory tree. Reports the slowest Hadoop datanodes in descending order at the end. Helps find cluster data layer bottlenecks such as slow datanodes, faulty hardware or misconfigured top-of-rack switch ports.
hdfs_files_native_checksums.jy- fetches native HDFS checksums for quicker file comparisons (about 100x faster than doing hdfs dfs -cat | md5sum)
hdfs_files_stats.jy- fetches HDFS file stats. Useful to generate a list of all files in a directory tree showing block size, replication factor, underfilled blocks and small files
impala_schemas_csv.py- dumps all databases, tables, columns and types out in CSV format to standard output
The following programs can all optionally filter by database / table name regex:
impala_foreach_table.py- execute any query or statement against every Hive / Impala table
impala_tables_row_counts.py- outputs tables row counts. Useful for reconciliation between cluster migrations
impala_tables_column_counts.py- outputs tables column counts. Useful for finding unusually wide tables
impala_tables_row_column_counts.py- outputs tables row and column counts. Useful for finding unusually big tables
impala_tables_row_counts_any_nulls.py- outputs tables row counts where any field is NULL. Useful for reconciliation between cluster migrations or catching data quality problems or subtle ETL bugs
impala_tables_null_columns.py- outputs tables columns containing only NULLs. Useful for catching data quality problems or subtle ETL bugs
impala_tables_null_rows.py- outputs tables row counts where all fields contain NULLs. Useful for catching data quality problems or subtle ETL bugs
impala_tables_metadata.py- outputs for each table the matching regex metadata DDL property from describe table
impala_tables_locations.py- outputs for each table its data location
hbase_generate_data.py- inserts random generated data in to a given HBase table, with optional skew support with configurable skew percentage. Useful for testing region splitting, balancing, CI tests etc. Outputs stats for number of rows written, time taken, rows per sec and volume per sec written.
hbase_show_table_region_ranges.py- dumps HBase table region ranges information, useful when pre-splitting tables
hbase_table_region_row_distribution.py- calculates the distribution of rows across regions in an HBase table, giving per region row counts and % of total rows for the table as well as median and quartile row counts per regions
hbase_table_row_key_distribution.py- calculates the distribution of row keys by configurable prefix length in an HBase table, giving per prefix row counts and % of total rows for the table as well as median and quartile row counts per prefix
hbase_compact_tables.py- compacts HBase tables (for off-peak compactions). Defaults to finding and iterating on all tables or takes an optional regex and compacts only matching tables.
hbase_flush_tables.py- flushes HBase tables. Defaults to finding and iterating on all tables or takes an optional regex and flushes only matching tables.
hbase_regions_by_*size.py- queries given RegionServers JMX to lists topN regions by storeFileSize or memStoreSize, ascending or descending
hbase_region_requests.py- calculates requests per second per region across all given RegionServers or average since RegionServer startup, configurable intervals and count, can filter to any combination of reads / writes / total requests per second. Useful for watching more granular region stats to detect region hotspotting
hbase_regionserver_requests.py- calculates requests per regionserver second across all given regionservers or average since regionserver(s) startup(s), configurable interval and count, can filter to any combination of read, write, total, rpcScan, rpcMutate, rpcMulti, rpcGet, blocked per second. Useful for watching more granular RegionServer stats to detect RegionServer hotspotting
hbase_regions_least_used.py- finds topN biggest/smallest regions across given RegionServers than have received the least requests (requests below a given threshold)
opentsdb_import_metric_distribution.py- calculates metric distribution in bulk import file(s) to find data skew and help avoid HBase region hotspotting
opentsdb_list_metrics*.sh- lists OpenTSDB metric names, tagk or tagv via OpenTSDB API or directly from HBase tables with optionally their created date, sorted ascending
find_active_server.py- returns first available healthy server or active master in high availability deployments, useful for chaining with single argument tools. Configurable tests include socket, http, https, ping, url and/or regex content match, multi-threaded for speed. Designed to extend tools that only accept a single
--hostoption but for which the technology has later added multi-master support or active-standby masters (eg. Hadoop, HBase) or where you want to query cluster wide information available from any online peer (eg. Elasticsearch)
find_active_hadoop_namenode.py- returns active Hadoop Namenode in HDFS HA
find_active_hadoop_resource_manager.py- returns active Hadoop Resource Manager in Yarn HA
find_active_hbase_master.py- returns active HBase Master in HBase HA
find_active_hbase_thrift.py- returns first available HBase Thrift Server (run multiple of these for load balancing)
find_active_hbase_stargate.py- returns first available HBase Stargate rest server (run multiple of these for load balancing)
find_active_apache_drill.py- returns first available Apache Drill node
find_active_cassandra.py- returns first available Apache Cassandra node
find_active_impala*.py- returns first available Impala node of either Impalad, Catalog or Statestore
find_active_presto_coordinator.py- returns first available Presto Coordinator
find_active_kubernetes_api.py- returns first available Kubernetes API server
find_active_oozie.py- returns first active Oozie server
find_active_solrcloud.py- returns first available Solr / SolrCloud node
find_active_elasticsearch.py- returns first available Elasticsearch node
travis_last_log.py- fetches Travis CI latest running / completed / failed build log for given repo - useful for quickly getting the log of the last failed build when CCMenu or BuildNotify applets turn red
travis_debug_session.py- launches a Travis CI interactive debug build session via Travis API, tracks session creation and drops user straight in to the SSH shell on the remote Travis build, very convenient one shot debug launcher for Travis CI
selenium_hub_browser_test.py- checks Selenium Grid Hub / Selenoid is working by calling browsers such as Chrome and Firefox to fetch a given URL and content/regex match the result
validate_*.py- validate files, directory trees and/or standard input streams
The automated build will use 'sudo' to install required Python PyPI libraries to the system unless running as root or it detects being inside a VirtualEnv. If you want to install some of the common Python libraries using your OS packages instead of installing from PyPI then follow the Manual Build section below.
Enter the pytools directory and run git submodule init and git submodule update to fetch my library repo:
git clone https://github.com/harisekhon/devops-python-tools pytools cd pytools git submodule init git submodule update sudo pip install -r requirements.txt
Download the DevOps Python Tools and Pylib git repos as zip files:
Unzip both and move Pylib to the
pylib folder under DevOps Python Tools.
unzip devops-python-tools-master.zip unzip pylib-master.zip mv -v devops-python-tools-master pytools mv -v pylib-master pylib mv -vf pylib pytools/
Proceed to install PyPI modules for whichever programs you want to use using your usual procedure - usually an internal mirror or proxy server to PyPI, or rpms / debs (some libraries are packaged by Linux distributions).
All PyPI modules are listed in the
Internal Mirror example (JFrog Artifactory or similar):
sudo pip install --index https://host.domain.com/api/pypi/repo/simple --trusted host.domain.com -r requirements.txt
sudo pip install --proxy hari:[email protected]:8080 -r requirements.txt
The automated build also works on Mac OS X but you'll need to install Apple XCode (on recent Macs just typing
git is enough to trigger Xcode install).
I also recommend you get HomeBrew to install other useful tools and libraries you may need like OpenSSL for development headers and tools such as wget (these are installed automatically if Homebrew is detected on Mac OS X):
brew install openssl wget
If failing to build an OpenSSL lib dependency, just prefix the build command like so:
sudo OPENSSL_INCLUDE=/usr/local/opt/openssl/include OPENSSL_LIB=/usr/local/opt/openssl/lib ...
You may get errors trying to install to Python library paths even as root on newer versions of Mac, sometimes this is caused by pip 10 vs pip 9 and downgrading will work around it:
sudo pip install --upgrade pip==9.0.1 make sudo pip install --upgrade pip make
The 3 Hadoop utility programs listed below require Jython (as well as Hadoop to be installed and correctly configured)
hdfs_time_block_reads.jy hdfs_files_native_checksums.jy hdfs_files_stats.jy
Run like so:
jython -J-cp $(hadoop classpath) hdfs_time_block_reads.jy --help
-J-cp $(hadoop classpath) part dynamically inserts the current Hadoop java classpath required to use the Hadoop APIs.
See below for procedure to install Jython if you don't already have it.
This will download and install jython to /opt/jython-2.7.0:
Jython is a simple download and unpack and can be fetched from http://www.jython.org/downloads.html
Then add the Jython install bin directory to the $PATH or specify the full path to the
jython binary, eg:
/opt/jython-2.7.0/bin/jython hdfs_time_block_reads.jy ...
Strict validations include host/domain/FQDNs using TLDs which are populated from the official IANA list is done via my PyLib library submodule - see there for details on configuring this to permit custom TLDs like
.cloud etc. (all already included in there because they're common across companies internal environments).
If you end up with an error like:
./dockerhub_show_tags.py centos ubuntu [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)
It can be caused by an issue with the underlying Python + libraries due to changes in OpenSSL and certificates. One quick fix is to do the following:
sudo pip uninstall -y certifi && sudo pip install certifi==2015.04.28
make update. This will git pull and then git submodule update which is necessary to pick up corresponding library updates.
If you update often and want to just quickly git pull + submodule update but skip rebuilding all those dependencies each time then run
make update-no-recompile (will miss new library dependencies - do full
make update if you encounter issues).
Continuous Integration is run on this repo with tests for success and failure scenarios:
To trigger all tests run:
which will start with the underlying libraries, then move on to top level integration tests and functional tests using docker containers if docker is available.
Patches, improvements and even general feedback are welcome in the form of GitHub pull requests and issue tickets.
DevOps Bash Tools - 550+ DevOps Bash Scripts, Advanced
.gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies
SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery
Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.
Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments
The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)
DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...
HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.
Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3
PyLib - Python library leveraged throughout the programs in this repo as a submodule
Perl Lib - Perl version of above library
You might also be interested in the following really nice Jupyter notebook for HDFS space analysis created by another Hortonworks guy Jonas Straub: