Spark, Hadoop, and Kryo utilities
Classes that implement the Registrar interface can use various shorthands for registering classes with Kryo.
Adapted from RegistrationTest:
cls[A], // comes with an AlsoRegister that loops in other classes
arr[Foo], // register a class and an Array of that class
cls[B] → BSerializer(), // use a custom Serializer
CDRegistrar // register all of another Registrar's registrations
AlsoRegisters are picked up implicitly if not provided explicitly.
AlsoRegisters are recursive, allowing for much easier and more robust accountability about what is registered and why, and ensurance that needed registrations aren't overlooked.
Configuration: serializable Hadoop-
SparkContextwrapper that is also a Hadoop
Configuration, for unification of "global configuration access" patterns
Conf: load a
SparkConfwith settings from file(s) specified in the
SparkConfBase: trait that brokers setting config key-values and creating a
- many mix-ins for common spark-configuration groups: