A library you can include in your Spark job to validate the counters and perform operations on success. Goal is scala/java/python support.

spark-validator 0.0.1


Group ID:
com.holdenkarau
Artifact ID:
spark-validator_2.10
Version:
0.0.1
Release Date:
Oct 5, 2015
Licenses:

libraryDependencies += "com.holdenkarau" %% "spark-validator" % "0.0.1"
resolvers += Resolver.bintrayJCenter

Mill build tool

ivy"com.holdenkarau::spark-validator:0.0.1"
MavenRepository("https://jcenter.bintray.com")

Scala CLI

//> using lib "com.holdenkarau::spark-validator:0.0.1"

Ammonite REPL

import $ivy.`com.holdenkarau::spark-validator:0.0.1`
import ammonite._, Resolvers._
val res = Resolver.Http(
  "Bintray's JCenter",
  "Some(https://jcenter.bintray.com)",
  IvyPattern,
  false)
interp.resolvers() = interp.resolvers() :+ res

<dependency>
  <groupId>com.holdenkarau</groupId>
  <artifactId>spark-validator_2.10</artifactId>
  <version>0.0.1</version>
</dependency>

compile group: 'com.holdenkarau', name: 'spark-validator_2.10', version: '0.0.1'