Scalding

A Scala API for Cascading
Alternatives To Scalding
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Scalding3,43337405 days ago43September 14, 2016319apache-2.0Scala
A Scala API for Cascading
Programmingwithscalding81
7 years ago2otherScala
Programming MapReduce with Scalding
Recommendationengine37
5 years agomitJava
[Deprecated] An optimized MapReduce for item‐based collaborative filtering recommendation algorithm with empirical analysis
Matrix Hadoop Tutorial31
10 years agoPython
A set of tutorial codes about matrix methods in Hadoop
Flexifact18
5 years ago1Java
Run large scale tensor and coupled matrix-tensor factorization on top of stock Hadoop.
Mrtsqr18
11 years agoPython
MapReduce Streaming TSQR Implementation
Matrixmultiply13
a year agoJava
Sparse matrix multiplication for hadoop
Recommendbyitemcf11
4 years ago1Java
Hadoop mapreduce. 基于ItemCF的协同过滤 物品推荐系统 Collaborative filtering goods recommendation system based on ItemCF
Recommender System8
7 years agoJava
movie recommender system based on Item CF
Algorithm Implementation Using Map Reduce8
6 years agomitJava
Page Rank, Inverted Index and Matrix Multiplication
Alternatives To Scalding
Select To Compare


Alternative Project Comparisons
Readme

Scalding

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Current version: 0.15.0

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._

class WordCountJob(args: Args) extends Job(args) {
  TypedPipe.from(TextLine(args("input")))
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedTsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Code of Conduct

This, and all github.com/twitter projects, are under the Twitter Open Source Code of Conduct. Additionally, see the Typelevel Code of Conduct for specific examples of harassing behavior that are not tolerated.

Building

There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.16.0-RC1.

Current published artifacts are

  • scalding-core_2.10
  • scalding-args_2.10
  • scalding-date_2.10
  • scalding-commons_2.10
  • scalding-avro_2.10
  • scalding-parquet_2.10
  • scalding-repl_2.10

The suffix denotes the scala version.

Adopters

  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki

Contact

For user questions or scalding development (internals, extending, release planning): https://groups.google.com/forum/#!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues

Follow @Scalding on Twitter for updates.

Chat: Gitter

Authors:

Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Popular Matrix Projects
Popular Hadoop Projects
Popular Computer Science Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Scala
Matrix
Hadoop
Sbt
Mapreduce