C# and F# language binding and extensions to Apache Spark
Alternatives To Mobius
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Spark35,9402,394882a day ago46May 09, 2021267apache-2.0Scala
Apache Spark - A unified analytics engine for large-scale data processing
2 years ago27
Notes talking about the design and implementation of Apache Spark
Synapseml4,2951a day ago5January 12, 2022304mitScala
Simple and Distributed Machine Learning
Hudi4,2612610 minutes ago13August 16, 2022677apache-2.0Java
Upserts, Deletes And Incremental Processing on Big Data.
Bigdl4,223102 hours ago16April 19, 2021747apache-2.0Jupyter Notebook
Fast, distributed, secure AI for Big Data
a year ago35Scala
酷玩 Spark: Spark 源代码解析、Spark 类库等
Spark Nlp3,2782210 hours ago90March 05, 202137apache-2.0Scala
State of the Art Natural Language Processing
Koalas3,2281126 months ago47October 19, 2021109apache-2.0Python
Koalas: pandas API on Apache Spark
Spark Notebook3,138
24 days ago207apache-2.0JavaScript
Interactive and Reactive Data Science using Scala and Spark.
Deequ2,81743 days ago31February 15, 2022127apache-2.0Scala
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Alternatives To Mobius
Select To Compare

Alternative Project Comparisons

Mobius development is deprecated and has been superseded by a more recent version '.NET for Apache Spark' from Microsoft (Website | GitHub) that runs on Azure HDInsight Spark, Amazon EMR Spark, Azure & AWS Databricks.

Mobius logo

Mobius: C# API for Spark

Mobius provides C# language binding to Apache Spark enabling the implementation of Spark driver program and data processing operations in the languages supported in the .NET framework like C# or F#.

For example, the word count sample in Apache Spark can be implemented in C# as follows :

var lines = sparkContext.TextFile(@"hdfs://path/to/input.txt");  
var words = lines.FlatMap(s => s.Split(' '));
var wordCounts = words.Map(w => new Tuple<string, int>(w.Trim(), 1))  
                      .ReduceByKey((x, y) => x + y);  
var wordCountCollection = wordCounts.Collect();  

A simple DataFrame application using TempTable may look like the following:

var reqDataFrame = sqlContext.TextFile(@"hdfs://path/to/requests.csv");
var metricDataFrame = sqlContext.TextFile(@"hdfs://path/to/metrics.csv");
// C0 - guid in requests DataFrame, C3 - guid in metrics DataFrame  
var joinDataFrame = GetSqlContext().Sql(  
    "SELECT joinedtable.datacenter" +
         ", MAX(joinedtable.latency) maxlatency" +
         ", AVG(joinedtable.latency) avglatency " +
    "FROM (" +
       "SELECT a.C1 as datacenter, b.C6 as latency " +  
       "FROM requests a JOIN metrics b ON a.C0  = b.C3) joinedtable " +   
    "GROUP BY datacenter");

A simple DataFrame application using DataFrame DSL may look like the following:

// C0 - guid, C1 - datacenter
var reqDataFrame = sqlContext.TextFile(@"hdfs://path/to/requests.csv")  
                             .Select("C0", "C1");    
// C3 - guid, C6 - latency   
var metricDataFrame = sqlContext.TextFile(@"hdfs://path/to/metrics.csv", ",", false, true)
                                .Select("C3", "C6"); //override delimiter, hasHeader & inferSchema
var joinDataFrame = reqDataFrame.Join(metricDataFrame, reqDataFrame["C0"] == metricDataFrame["C3"])
var maxLatencyByDcDataFrame = joinDataFrame.Agg(new Dictionary<string, string> { { "C6", "max" } });

A simple Spark Streaming application that processes messages from Kafka using C# may be implemented using the following code:

StreamingContext sparkStreamingContext = StreamingContext.GetOrCreate(checkpointPath, () =>
      var ssc = new StreamingContext(sparkContext, slideDurationInMillis);
      var stream = KafkaUtils.CreateDirectStream(ssc, topicList, kafkaParams, perTopicPartitionKafkaOffsets);
      //message format: [timestamp],[loglevel],[logmessage]
      var countByLogLevelAndTime = stream
                                    .Map(kvp => Encoding.UTF8.GetString(kvp.Value))
                                    .Filter(line => line.Contains(","))
                                    .Map(line => line.Split(','))
                                    .Map(columns => new Tuple<string, int>(
                                                          string.Format("{0},{1}", columns[0], columns[1]), 1))
                                    .ReduceByKeyAndWindow((x, y) => x + y, (x, y) => x - y,
                                                          windowDurationInSecs, slideDurationInSecs, 3)
                                    .Map(logLevelCountPair => string.Format("{0},{1}",
                                                          logLevelCountPair.Key, logLevelCountPair.Value));
      countByLogLevelAndTime.ForeachRDD(countByLogLevel =>
          foreach (var logCount in countByLogLevel.Collect())
      return ssc;

For more code samples, refer to Mobius\examples directory or Mobius\csharp\Samples directory.

API Documentation

Refer to Mobius C# API documentation for the list of Spark's data processing operations supported in Mobius.

API Usage

Mobius API usage samples are available at:

  • Examples folder which contains standalone C# and F# projects that can be used as templates to start developing Mobius applications

  • Samples project which uses a comprehensive set of Mobius APIs to implement samples that are also used for functional validation of APIs

  • Mobius performance test scenarios implemented in C# and Scala for side by side comparison of Spark driver code


Refer to the docs folder for design overview and other info on Mobius

Build Status

Ubuntu 14.04.3 LTS Windows Unit test coverage
Build status Build status

Getting Started

Windows Linux
Build & run unit tests Build in Windows Build in Linux
Run samples (functional tests) in local mode Samples in Windows Samples in Linux
Run examples in local mode Examples in Windows Examples in Linux
Run Mobius app
Run Mobius Shell Not supported yet

Useful Links

Supported Spark Versions

Mobius is built and tested with Apache Spark 1.4.1, 1.5.2, 1.6.* and 2.0.


Mobius releases are available at References needed to build C# Spark driver applicaiton using Mobius are also available in NuGet

NuGet Badge

Refer to for the details on versioning policy and the contents of the release.



Mobius is licensed under the MIT license. See LICENSE file for full license information.


Issue Stats Issue Stats Join the chat at Twitter

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Popular Spark Projects
Popular Apache Spark Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
C Sharp
Big Data
Apache Spark
Spark Streaming