andyglow / spark-option-setter   0.3.1

GitHub

Spark's Reader/Writer Frame and Stream type-safe option setter

Scala versions: 2.12

spark-option-setter

Build Status codecov mvn

since 0.3.1 it is compiled against scala 2.12 for spark 3.x only

earlier versions compiled against both scala 2.11 and 2.12 and aimed to cover spark 2.4.x

When it comes to specifying options on spark readers and writers sometimes it may look inconvenient when you need to work with effects like Option, Either, Try.. Although last 2 might mot be considered as a good candidate, the first, which is Option often is taken into the loop.

And code becomes messy...

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()

So with this simple library you can acheave this level of readiness of code

import com.github.andyglow.spark.options._

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

spark.read
  .format("parquet")
  .option("foo", "bar")
  .option("extra-option-1", extra1Option)
  .option("extra-option-2", extra2Option)
  .load()

Install

libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"

Statistics

4 Dependencies

No Dependent