A functional wrapper around Spark to make it works with ZIO

zio-spark

A functional wrapper around Spark to make it work with ZIO

Documentation

You can find the documentation of zio-spark here.

Help

You can ask us (Dylan, Jonathan) for some help if you want to use the lib or have questions around it : https://calendly.com/zio-spark/help

Latest version

If you want to get the very last version of this library you can still download it using:

libraryDependencies += "io.univalence" %% "zio-spark" % "0.8.0"

Snapshots

If you want to get the latest snapshots (the version associated with the last commit on master), you can still download it using:

resolvers += Resolver.sonatypeRepo("snapshots"),
libraryDependencies += "io.univalence" %% "zio-spark" % "<SNAPSHOT-VERSION>"

You can find the latest version on nexus repository manager.

Spark Version

ZIO-Spark is compatible with Scala 2.11, 2.12 and 2.13. Spark is provided, you must add your own Spark version in build.sbt (as you would usually).

We advise you to use the latest version of Spark for your scala version.

Why ?

There are many reasons why we decide to build this library, such as:

  • allowing user to build Spark pipeline with ZIO easily.
  • making better code, pure FP, more composable, more readable Spark code.
  • stopping the propagation of implicit SparkSessions.
  • improving some performances.
  • taking advantage of ZIO allowing our jobs to retry and to be run in parallel.

Alternatives

  • ZparkIO a framework for Spark, ZIO

Contributions

Pull requests are welcomed. We are open to organize pair-programming session to tackle improvements. If you want to add new things in zio-spark, don't hesitate to open an issue!

You can also talk to us directly using this link if you are interested to contribute https://calendly.com/zio-spark/contribution.