databrickslabs/automl-toolkit   0.7.2

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

pishen/sbt-lighter   1.2.0

SBT plugin for Apache Spark on AWS EMR

yaooqinn/itachi   0.3.0

A library that brings useful functions from various modern database management systems to Apache Spark

potix2/spark-google-spreadsheets   0.6.3

Google Spreadsheets datasource for SparkSQL and DataFrames

uosdmlab/spark-nkp   0.3.3

Natural Korean Processor for Apache Spark

cerndb/sparkplugins   0.3

Code and examples of how to write and deploy Apache Spark Plugins. Spark plugins allow runnig custom code on the executors as they are initialized. This also allows extending the Spark metrics systems with user-provided monitoring probes.

starlake-ai/starlake   1.3.0

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.

hydrospheredata/spark-ml-serving   0.3.3

Spark ML Lib serving library

hablapps/sparkoptics   0.1.1

Optics for Spark DataFrames

locationtech-labs/geopyspark   0.3.0

GeoTrellis for PySpark

coxautomotivedatasolutions/spark-distcp   0.2.5

A re-implementation of Hadoop DistCP in Apache Spark

benfradet/struct-type-encoder   0.6.0

Deriving Spark DataFrame schemas from case classes

absaoss/hyperdrive   4.7.0

Extensible streaming ingestion pipeline on top of Apache Spark

tharwaninitin/etlflow   1.7.3

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

univalence/zio-spark   0.12.0

A functional wrapper around Spark to make it works with ZIO

g-research/spark-dgraph-connector   0.2.0

A connector for Apache Spark and PySpark to Dgraph databases.

zuinnote/spark-hadoopoffice-ds   1.7.0

A Spark datasource for the HadoopOffice library

xskipper-io/xskipper   1.3.0

An Extensible Data Skipping Framework

heartsavior/spark-sql-kafka-offset-committer   0.2.0

Kafka offset committer for structured streaming query

absaoss/spark-hats   0.3.0

Nested array transformation helper extensions for Apache Spark