mrjob is a Python 2.7/3.4+ package that helps you write and run Hadoop Streaming jobs.
Stable version (v0.7.4) documentation
Development version documentation
mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. mrjob has basic support for Google Cloud Dataproc (Dataproc) which allows you to buy time on a Hadoop cluster on a minute-by-minute basis. It also works with your own Hadoop cluster.
Some important features:
$PYTHONPATH
$TZ
)mrjob.conf
config file$AWS_ACCESS_KEY_ID
and $AWS_SECRET_ACCESS_KEY
$GOOGLE_APPLICATION_CREDENTIALS
pip install mrjob
As of v0.7.0, Amazon Web Services and Google Cloud Services are optional
depedencies. To use these, install with the aws
and google
targets,
respectively. For example:
pip install mrjob[aws]
Code for this example and more live in mrjob/examples
.
"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re
WORD_RE = re.compile(r"[\w']+")
class MRWordFreqCount(MRJob):
def mapper(self, _, line):
for word in WORD_RE.findall(line):
yield (word.lower(), 1)
def combiner(self, word, counts):
yield (word, sum(counts))
def reducer(self, word, counts):
yield (word, sum(counts))
if __name__ == '__main__':
MRWordFreqCount.run()
# locally python mrjob/examples/mr_word_freq_count.py README.rst > counts # on EMR python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts # on Dataproc python mrjob/examples/mr_word_freq_count.py README.rst -r dataproc > counts # on your Hadoop cluster python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts
$AWS_ACCESS_KEY_ID
and
$AWS_SECRET_ACCESS_KEY
accordinglyTo run in other AWS regions, upload your source tree, run make
, and use
other advanced mrjob features, you'll need to set up mrjob.conf
. mrjob looks
for its conf file in:
$MRJOB_CONF
~/.mrjob.conf
/etc/mrjob.conf
See the mrjob.conf documentation for more information.
Thanks to Greg Killion (ROMEO ECHO_DELTA) for the logo.