whylabs/whylogs-java   0.1.3

Profile and monitor your ML data pipeline end-to-end

isarn/isarn-sketches-spark   0.6.0-sp3.2

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

arcizon/spark-filetransfer   0.3.0

API for reading and writing data via various file transfer protocols from Apache Spark.

florentf9/sparkml-som   0.2

:sparkles: Spark ML implementation of SOM algorithm (Kohonen self-organizing map)

kotlin/kotlin-spark-api   1.2.4

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x

derrickoswald/cimspark   2.12-3.0.1-5.1.1

Spark access to Common Information Model (CIM) files

romans-weapon/spear-framework   3.1.1-3.0

Rapid ETL/ELT-connectors/pipeline development leveraged on top of Apache Spark

woltapp/spark-osm-datasource   0.3.0

Native Spark OSM PBF data source

s22s/pre-lt-raster-frames   0.6.1

Spark DataFrames for earth observation data

qubole/streaminglens   0.5.3

Qubole Streaminglens tool for tuning Spark Structured Streaming Pipelines

qubole/s3-sqs-connector   0.5.1

A library for reading data from Amzon S3 with optimised listing using Amazon SQS using Spark SQL Streaming ( or Structured streaming).

hammerlab/spark-util   3.1.0

low-level helpers for Apache Spark libraries and tests

getsentry/sentry-spark   0.0.1-alpha04

Apache Spark Sentry Integration

hunters-ai/spark-adaptive-file-connector   1.0.0

Adaptive File Source Connector for Spark, optimised for reading from object stores

piotr-kalanski/data-quality-monitoring   0.3.8

Data Quality Monitoring Tool

jtnystrom/discount   3.0.1

Very large scale k-mer counting and analysis on Apache Spark.

absaoss/spark-hofs   0.5.0

Scala API for Apache Spark SQL high-order functions

qubole/spark-state-store   1.0.0

Rocksdb state storage implementation for Structured Streaming.

fqaiser94/mse   0.2.4

Make Structs Easy (MSE)

data-tools/big-data-types   1.4.1

A library to transform Scala product types and Schemes from different systems into other Schemes. Any implemented type automatically gets methods to convert it into the rest of the types and vice versa. E.g: a Spark Schema can be transformed into a BigQuery table.