This is a native connector for reading and writing MongoDB collections directly from Apache Spark. In Spark, the data from MongoDB is represented as an RDD[MongoDBObject]. Starting with Release 0.5.0, you can also use NSMC through Spark SQL by registering a MongpDB collection as a temporary table.

Fore more details about this project, check the following blog posts:

NSMC: A Native MongoDB Connector for Apache Spark

Spark SQL Integration for MongoDB

• No Java or Python API bindings
• Can only read (no updates)
• Only tested with the following configurations:
• MongoDB: 2.6
• Scala: 2.10
• Spark: 1.4.0
• Casbah 2.7
• Not tested with MongoDB's hash-based partitioning

• The APIs will probably change several times before an official release

A pre-release alpha version (0.5.3) is available.

See the top-level LICENSE file

Perhaps the easiest way to get started is to use the companion example project. It contains an appropriately configured sbt project, together with sample code for using NSMC from both core Spark and Spark SQL. Be sure to check out that project's README to to make sure you're using the right branch for the NSMC and Apache Spark versions you want to use.

Add the following to your build.sbt for the latest stable release:

scalaVersion := "2.10.4" // any 2.10 is OK -- support for 2.11 coming soon

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.0" % "provided"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.4.0" % "provided"

libraryDependencies += "com.github.spirom" %% "spark-mongodb-connector" % "0.5.3"

In your code, you need to add:

import nsmc._

import com.mongodb.DBObject

Then to actually read from a collection:

val conf = new SparkConf()
    .setAppName("My MongoApp").setMaster("local[4]")
    .set("spark.nsmc.connection.host", "myMongoHost")
    .set("spark.nsmc.connection.port", "myMongoPort")
    .set("spark.nsmc.user", "yourUsernameHere")
    .set("spark.nsmc.password", "yourPasswordHere")
val sc = new SparkContext(conf)
val data: MongoRDD[DBObject] =
    sc.mongoCollection[DBObject]("myDB", "myCollection")

Register a temporary table as follows

sqlContext.sql(
    s"""
    |CREATE TEMPORARY TABLE dataTable
    |USING nsmc.sql.MongoRelationProvider
    |OPTIONS (db 'myDB', collection 'myCollection')
  """.stripMargin)

Query the temporary table as follows

val data = sqlContext.sql("SELECT * FROM dataTable")