Spark Records is a data processing pattern with an associated lightweight, dependency-free framework for Apache Spark v2+ that enables:

1. Bulletproof data processing with Spark
   Your jobs will never unpredictably fail midway due to data transformation bugs. Spark records give you predictable failure control through instant data quality checks performed on metrics automatically collected during job execution, without any additional querying.

2. Automatic row-level structured logging
   Exceptions generated during job execution are automatically associated with the data that caused the exception, down to nested exception causes and full stack traces. If you need to reprocess data, you can trivially and efficiently choose to only process the failed inputs.

3. Lightning-fast root cause analysis
   Get answers to any questions related to exceptions or warnings generated during job execution directly using SparkSQL or your favorite Spark DSL. Would you like to see the top 5 issues encountered during job execution with example source data and the line in your code that caused the problem? You can.

Spark Records has been tested with petabyte-scale data at Swoop. The library was extracted out of Swoop's production systems to share with the Spark community.

See the documentation for more information or watch the Spark Summit talk (slides).

Just add the following to your libraryDependencies in SBT:

resolvers += Resolver.bintrayRepo("swoop-inc", "maven")

libraryDependencies += "com.swoop" %% "spark-records" % "<version>"

You can find all released versions here.

Contributions and feedback of any kind are welcome.

Spark Records is maintained by Sim Simeonov and the team at Swoop.

Special thanks to Reynold Xin and Michael Armbrust for many interesting conversations about better ways to use Spark.

Build docs microsite

sbt "project docs" makeMicrosite

Run docs microsite locally (run under target/site folder)

jekyll serve -b /spark-records

More details

spark-records is Copyright © 2017 Simeon Simeonov and Swoop, Inc. It is free software, and may be redistributed under the terms of the LICENSE.