Microsoft Machine Learning for Apache Spark
MMLSpark is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. MMLSpark adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.
MMLSpark also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, MMLSpark provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.
Table of Contents
- Create a deep image classifier with transfer learning (example 9)
- Fit a LightGBM classification or regression model on a biochemical dataset (example 3), to learn more check out the LightGBM documentation page.
- Deploy a deep network as a distributed web service with MMLSpark Serving
- Use web services in Spark with HTTP on Apache Spark
- Use Bi-directional LSTMs from Keras for medical entity extraction (example 8)
- Create a text analytics system on Amazon book reviews (example 4)
- Perform distributed hyperparameter tuning to identify Breast Cancer (example 5)
- Easily ingest images from HDFS into Spark
- Use OpenCV on Spark to manipulate images (example 7)
- Train classification and regression models easily via implicit featurization of data (example 1)
- Train and evaluate a flight delay prediction system (example 2)
See our notebooks for all examples.
A short example
Below is an excerpt from a simple example of using a pre-trained CNN to classify images in the CIFAR-10 dataset. View the whole source code in notebook example 9.
... import mmlspark # Initialize CNTKModel and define input and output columns cntkModel = mmlspark.cntk.CNTKModel() \ .setInputCol("images").setOutputCol("output") \ .setModelLocation(modelFile) # Train on dataset with internal spark pipeline scoredImages = cntkModel.transform(imagesWithLabels) ...
Setup and installation
MMLSpark can be conveniently installed on existing Spark clusters via the
--packages option, examples:
spark-shell --packages com.microsoft.ml.spark:mmlspark_2.11:0.18.1 pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:0.18.1 spark-submit --packages com.microsoft.ml.spark:mmlspark_2.11:0.18.1 MyApp.jar
For the coordinates use:
com.microsoft.ml.spark:mmlspark_2.11:0.18.1. Ensure this library is attached to all clusters you create.
Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.
You can use MMLSpark in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:
The easiest way to evaluate MMLSpark is via our pre-built Docker container. To do so, run the following command:
docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release
To read the EULA for using the docker image, run \
docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula
GPU VM Setup
MMLSpark can be used to train deep learning models on GPU nodes from a Spark application. See the instructions for setting up an Azure GPU VM.
To try out MMLSpark on a Python (or Conda) installation you can get Spark installed via pip with
pip install pyspark. You can then use
pyspark as in the above example, or from python:
import pyspark spark = pyspark.sql.SparkSession.builder.appName("MyApp") \ .config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:0.18.1") \ .getOrCreate() import mmlspark
If you are building a Spark application in Scala, add the following lines to your
libraryDependencies += "com.microsoft.ml.spark" %% "mmlspark" % "0.18.1"
Building from source
MMLSpark has recently transitioned to a new build infrastructure. For detailed developer docs please see the Developer Readme
If you are an existing mmlspark developer, you will need to reconfigure your development setup. We now support platform independent development and better integrate with intellij and SBT. If you encounter issues please reach out to our support email!
To try out MMLSpark using the R autogenerated wrappers see our instructions. Note: This feature is still under development and some necessary custom wrappers may be missing.
Visit our new website.
Read our paper for a deep dive on MMLSpark.
See how MMLSpark is used to help endangered species.
Explore our collaboration with Apache Spark on image analysis.
Watch MMLSpark at the Spark Summit.
Contributing & feedback
See CONTRIBUTING.md for contribution guidelines.
To give feedback and/or report an issue, open a GitHub Issue.
Other relevant projects
Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.