To add Mongodb Almaren dependency to your sbt build:
libraryDependencies += "com.github.music-of-the-ainur" %% "mongodb-almaren" % "0.0.9-3.4"
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector" % "10.2.0"
To run in spark-shell:
spark-shell --master local[*] --packages "com.github.music-of-the-ainur:almaren-framework_2.12:0.9.10-3.4,com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.4,org.mongodb.spark:mongo-spark-connector:10.2.0"
MongoDB Connector is available in Maven Central repository.
| version | Connector Artifact |
|---|---|
| Spark 3.4.x and scala 2.13 | com.github.music-of-the-ainur:mongodb-almaren_2.13:0.0.9-3.4 |
| Spark 3.4.x and scala 2.12 | com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.3 |
| Spark 3.3.x and scala 2.13 | com.github.music-of-the-ainur:mongodb-almaren_2.13:0.0.9-3.3 |
| Spark 3.3.x and scala 2.12 | com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.3 |
| Spark 3.2.x and scala 2.12 | com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.2 |
| Spark 3.1.x and scala 2.12 | com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.1 |
| Spark 2.4.x and scala 2.12 | com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-2.4 |
| Spark 2.4.x and scala 2.11 | com.github.music-of-the-ainur:mongodb-almaren_2.11:0.0.9-2.4 |
Connector was implemented using: https://github.com/mongodb/mongo-spark.
Parameters:
| Parameters | Description |
|---|---|
| hosts | localhost:27017 |
| database | foo |
| collection | bar |
| user | username |
| password | password |
| stringPrefix | this is used to specify MongoDb Connection type(srv) https://docs.mongodb.com/manual/reference/connection-string/ |
| options | extra connector options |
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.sourceMongoDb("localhost","foo","bar")import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.sourceMongoDb("localhost","foo","bar",None,None,Some("srv"))Parameters for Uri:
| Parameters | Description |
|---|---|
| uri | mongodb://localhost:27017/foo |
| collection | bar |
| options | extra connector options |
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.sourceMongoDbUri("mongodb://localhost:27017/foo","bar")Parameters:
| Parameters | Description |
|---|---|
| hosts | localhost:27017 |
| database | foo |
| collection | bar |
| user | username |
| password | password |
| stringPrefix | this is used to specify MongoDb Connection type(srv) https://docs.mongodb.com/manual/reference/connection-string/ |
| options | extra connector options |
| saveMode | SaveMode.Overwrite |
import org.apache.spark.sql.SaveMode
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.targetMongoDb("localhost","foo","bar", saveMode = SaveMode.Overwrite)import org.apache.spark.sql.SaveMode
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.targetMongoDb("localhost","foo","bar",None,None,Some("srv"), saveMode = SaveMode.Overwrite)Parameters for Uri:
| Parameters | Description |
|---|---|
| uri | mongodb://localhost:27017/foo |
| collection | bar |
| options | extra connector options |
| saveMode | SaveMode.Overwrite |
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit
almaren.builder.targetMongoDbUri("mongodb://localhost:27017/foo","bar", saveMode = SaveMode.Overwrite)