andyglow / spark-option-setter

Spark's Reader/Writer Frame and Stream type-safe option setter

Version Matrix

spark-option-setter

Build Status Coverage Status Maven Central Maven Central

When it comes to specifying options on spark readers and writers sometimes it may look inconvenient when you need to work with effects like Option, Either, Try.. Although last 2 might mot be considered as a good candidate, the first, which is Option often is taken into the loop.

And code becomes messy...

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()

So with this simple library you can acheave this level of readiness of code

import com.github.andyglow.spark.options._

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

spark.read
  .format("parquet")
  .option("foo", "bar")
  .option("extra-option-1", extra1Option)
  .option("extra-option-2", extra2Option)
  .load()

Install

libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"