since
0.3.1it is compiled against scala2.12for spark3.xonly
earlier versions compiled against both scala
2.11and2.12and aimed to cover spark2.4.x
When it comes to specifying options on spark readers and writers sometimes it may look inconvenient
when you need to work with effects like Option, Either, Try.. Although last 2 might mot be considered as a
good candidate, the first, which is Option often is taken into the loop.
And code becomes messy...
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()So with this simple library you can acheave this level of readiness of code
import com.github.andyglow.spark.options._
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
spark.read
.format("parquet")
.option("foo", "bar")
.option("extra-option-1", extra1Option)
.option("extra-option-2", extra2Option)
.load()libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"
