videoamp / ammonium-util

Ammonium utils, principally for bootstrapping a Flint Spark session

Website GitHub

VideoAmp's Ammonium Utils

Travis Maven Central

These utilities consist principally of a bootstrapping system for connecting to a Flint Spark cluster from an Ammonium shell or Jupyter Scala notebook.

Usage

Assuming you have a Flint Spark master running at ip address masterIP, run

interp.load.ivy("com.videoamp" %% "ammonium-util" % "2.0.2")
# dc3
vamp.ammonium.bootstrap(masterIP, env="dc3")
# use1
vamp.ammonium.bootstrap(masterIP, env="use1")

in a Jupyter Scala notebook to bind a SparkConf to the sparkConf variable in your notebook environment. After configuring Spark through sparkConf, you can access a bootstrapped and configured SparkSession with the spark notebook variable.