Miller

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Alternatives To Miller
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Miller8,138
7 days ago65November 26, 202298otherGo
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Visidata6,824573 days ago52July 17, 202391gpl-3.0Python
A terminal spreadsheet multitool for discovering and arranging data
Structured Text Tools6,759
2 days ago3
A list of command line tools for manipulating structured text data
Dsq3,401
2 days ago2October 20, 202219otherGo
Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.
Sqlitebiter780
12 months ago85March 27, 20227mitPython
A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
Pytablewriter57139388 days ago121March 21, 20226mitPython
pytablewriter is a Python library to write a table in various formats: AsciiDoc / CSV / Elasticsearch / HTML / JavaScript / JSON / LaTeX / LDJSON / LTSV / Markdown / MediaWiki / NumPy / Excel / Pandas / Python / reStructuredText / SQLite / TOML / TSV.
Simple Excel Php5353322 years ago10March 21, 201314PHP
Simplexcel.php - Easily read / parse / convert / write between Microsoft Excel XML / CSV / TSV / HTML / JSON / etc spreadsheet tabular file formats
Sq477
2 days ago25August 10, 202213mitGo
sq data wrangler
Elasticsearch_loader34912a year ago34December 21, 20191mitPython
A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch
Sqawk272
2 years ago1mitTcl
Like Awk but with SQL and table joins
Alternatives To Miller
Select To Compare


Alternative Project Comparisons
Readme

What is Miller?

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

What can Miller do for me?

With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more.

cover-art

  • Miller operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map.

  • Miller handles a variety of data formats, including but not limited to the familiar CSV, TSV, and JSON/JSON Lines. (Miller can handle positionally-indexed data too!)

In the above image you can see how Miller embraces the common themes of key-value-pair data in a variety of data formats.

Getting started

More documentation links

Installing

There's a good chance you can get Miller pre-built for your system:

Ubuntu Ubuntu 16.04 LTS Fedora Debian Gentoo

Pro-Linux Arch Linux

NetBSD FreeBSD

Anaconda Homebrew/MacOSX MacPorts/MacOSX Chocolatey

OS Installation command
Linux yum install miller
apt-get install miller
Mac brew install miller
port install miller
Windows choco install miller

See also README-versions.md for a full list of package versions. Note that long-term-support (LtS) releases will likely be on older versions.

See also building from source.

Community

GitHub stars Homebrew downloads Conda downloads

All Contributors

Build status

Multi-platform build status CodeQL status Codespell status

Building from source

  • First:
    • cd /where/you/want/to/put/the/source
    • git clone https://github.com/johnkerl/miller
    • cd miller
  • With make:
    • To build: make. This takes just a few seconds and produces the Miller executable, which is ./mlr (or .\mlr.exe on Windows).
    • To run tests: make check.
    • To install: make install. This installs the executable /usr/local/bin/mlr and manual page /usr/local/share/man/man1/mlr.1 (so you can do man mlr).
    • You can do ./configure --prefix=/some/install/path before make install if you want to install somewhere other than /usr/local.
  • Without make:
    • To build: go build github.com/johnkerl/miller/cmd/mlr.
    • To run tests: go test github.com/johnkerl/miller/pkg/... and mlr regtest.
    • To install: go install github.com/johnkerl/miller/cmd/mlr will install to GOPATH/bin/mlr.
  • See also the doc page on building from source.
  • For more developer information please see README-dev.md.

For developers

License

License: BSD2

Features

  • Miller is multi-purpose: it's useful for data cleaning, data reduction, statistical reporting, devops, system administration, log-file processing, format conversion, and database-query post-processing.

  • You can use Miller to snarf and munge log-file data, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing.

  • Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data. While you can do basic statistics entirely in Miller, its streaming-data feature and single-pass algorithms enable you to reduce very large data sets.

  • Miller complements SQL databases: you can slice, dice, and reformat data on the client side on its way into or out of a database. You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.

  • Miller also goes beyond the classic Unix tools by stepping fully into our modern, no-SQL world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.

  • Miller is streaming: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (sort, tac, stats1), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system’s available RAM, and you can use Miller in tail -f contexts.

  • Miller is pipe-friendly and interoperates with the Unix toolkit.

  • Miller's I/O formats include tabular pretty-printing, positionally indexed (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.

  • Miller does conversion between formats.

  • Miller's processing is format-aware: e.g. CSV sort and tac keep header lines first.

  • Miller has high-throughput performance on par with the Unix toolkit.

  • Miller is written in portable, modern Go, with zero runtime dependencies. You can download or compile a single binary, scp it to a faraway machine, and expect it to work.

What people are saying about Miller

Contributors

Thanks to all the fine people who help make Miller better (emoji key):


Andrea Borruso


Shaun Jackman


Fred Trotter


komosa


jungle-boogie


Thomas Klausner


Stephen Kitt


Leah Neukirchen


Luigi Baldoni


Hiroaki Yutani


Daniel M. Drucker


Nikos Alexandris


kundeng


Victor Sergienko


Adrian Ho


zachp


David Selassie


Joel Parker Henderson


Michel Ace


Matus Goljer


Richard Patel


Jakub Podlaha


Miodrag Mili


Derek Mahar


spmundi


Peter Krner


rubyFeedback


rbolsius


awildturtok


agguser


jganong


Fulvio Scapin


Jordan Torbiak


Andreas Weber


vapniks


Zombo


Brian Fulton-Howard


ChCyrill


Jauder Ho


Pawe Sacawa


schragge


Jordi

This project follows the all-contributors specification. Contributions of any kind are welcome!

Popular Json Projects
Popular Tsv Projects
Popular Data Formats Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Go
Command Line
Json
Csv
Unix
Statistics
Command Line Tool
Json Data
Awk
Tsv
Statistical Analysis
Devops Tools
Data Processing
Data Cleaning
Tabular Data
Streaming Data