A functional wrapper around Spark to make it work with ZIO
You can find the documentation of zio-spark here.
You can ask us (Dylan, Jonathan) for some help if you want to use the lib or have questions around it : https://calendly.com/zio-spark/help
If you want to get the very last version of this library you can still download it using:
libraryDependencies += "io.univalence" %% "zio-spark" % "0.8.0"
If you want to get the latest snapshots (the version associated with the last commit on master), you can still download it using:
resolvers += Resolver.sonatypeRepo("snapshots"), libraryDependencies += "io.univalence" %% "zio-spark" % "<SNAPSHOT-VERSION>"
You can find the latest version on nexus repository manager.
ZIO-Spark is compatible with Scala 2.11, 2.12 and 2.13. Spark is provided, you must add your own Spark version in build.sbt (as you would usually).
We advise you to use the latest version of Spark for your scala version.
There are many reasons why we decide to build this library, such as:
- allowing user to build Spark pipeline with ZIO easily.
- making better code, pure FP, more composable, more readable Spark code.
- stopping the propagation of
- improving some performances.
- taking advantage of ZIO allowing our jobs to retry and to be run in parallel.
- ZparkIO a framework for Spark, ZIO
Pull requests are welcomed. We are open to organize pair-programming session to tackle improvements. If you want to add
new things in
zio-spark, don't hesitate to open an issue!
You can also talk to us directly using this link if you are interested to contribute https://calendly.com/zio-spark/contribution.