smart-data-lake/smart-data-lake

Smart Automation Tool for building modern Data Lakes and Data Pipelines

kakao/cuesheet

A framework for writing Spark 2.x applications in a pretty way

absaoss/cobrix

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

sansa-stack/sansa-stack

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

g-research/spark-extension

A library that provides useful extensions to Apache Spark and PySpark.

dwestheide/kontextfrei

Writing application logic for Spark jobs that can be unit-tested without a SparkContext

galliaproject/gallia-core

A schema-aware Scala library for data transformation

coxautomotivedatasolutions/waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

swoop-inc/spark-records

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

azure/azure-kusto-spark

Apache Spark Connector for Azure Kusto

emesday/spark-annoy

Building Annoy Index on Apache Spark

samelamin/spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

googleclouddataproc/spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

jelmerk/hnswlib

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

simplexspatial/osm4scala

Scala and Spark library focused on reading OpenStreetMap Pbf files.

hablapps/doric

Type safety for spark columns

pishen/sbt-lighter

SBT plugin for Apache Spark on AWS EMR

databrickslabs/automl-toolkit

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

mrpowers/spark-stringmetric

Spark functions to run popular phonetic and string matching algorithms

potix2/spark-google-spreadsheets

Google Spreadsheets datasource for SparkSQL and DataFrames