Search *

treeverse/lakefs

Git-like capabilities for your object storage

data-versioning datalakes go google-cloud-storage azure-blob-storage lakefs object-storage apache-spark hadoop-filesystem git-for-data aws-s3 data-quality data-lake datalake data-engineering golang apache-sparksql

1646 172 45

microsoft/mobius

C# and F# language binding and extensions to Apache Spark

Scala (JVM): 2.10 2.11

mapreduce bigdata near-real-time streaming dataset apache-spark csharp fsharp dstream dataframe spark kafka-streaming rdd spark-streaming mobius eventhubs

737 173 32

lucacanali/sparkmeasure

This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.

Scala (JVM): 2.11 2.12

spark performance-troubleshooting apache-spark performance-metrics spark-troubleshooting

323 74 7

hydrospheredata/mist

Serverless proxy for Spark cluster

Scala (JVM): 2.10 2.11 2.12

apache-spark api big-data serverless

247 59 24

whylabs/whylogs-java

Profile and monitor your ML data pipeline end-to-end

Scala (JVM): 2.11 2.12

approximate-statistics java calculate-statistics aiops dataset apache-spark statistics ai-pipelines mlops spark statistical-properties data-quality whylogs

179 8 7

azure/azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB

Scala (JVM): 2.10 2.11

azure-databricks connector azure-cosmos-db lambda-architecture jupyter-notebook pyspark spark changefeed databricks-notebooks apache-spark databricks cosmos-db

176 114 22

azure/azure-event-hubs-spark

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Scala (JVM): 2.11 2.12

spark structured-streaming stream streaming connector eventhubs kafka bigdata apache scala spark-streaming continuous real-time microsoft azure ingestion apache-spark databricks event-hubs

161 148 33

kakao/cuesheet

A framework for writing Spark 2.x applications in a pretty way

Scala (JVM): 2.10 2.11

spark scala yarn mango magic apache-spark

78 21 2

streamnative/pulsar-spark

When Apache Pulsar meets Apache Spark

Scala (JVM): 2.12

stream-processing spark structured-streaming spark-sql flink batch-processing apache-spark data-science apache-pulsar data-processing

69 24 7

swoop-inc/spark-records

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

Scala (JVM): 2.12

big-data spark scala swoop apache-spark sparksql spark-records

58 14 4

chermenin/spark-states

Custom state store providers for Apache Spark

Scala (JVM): 2.11 2.12

stateful spark apache spark-streaming structured-streaming state-store spark-structured-streaming state apache-spark

53 17 3

uosdmlab/spark-nkp

Natural Korean Processor for Apache Spark

Scala (JVM): 2.11

spark apache-spark text-mining nlp spark-mllib natural-language-processing korean-nlp

41 13 2

bizreach/aws-kinesis-scala

Scala client for Amazon Kinesis. Also provides write to Kinesis capability for Apache Spark or Spark Streaming.

Scala (JVM): 2.11 2.12 2.13

scala apache-spark kinesis aws spark aws-kinesis

26 11 4

flipkart-incubator/spark-transformers

Spark-Transformers: Library for exporting Apache Spark MLLIB models in to use them in any Java application with no other dependencies.

Scala (JVM): 2.10 2.11

mllib java spark machine-learning-algorithms machine-learning transformers export apache-spark machine-learning-library data-pipelines scala

25 20 8

absaoss/hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark

Scala (JVM): 2.11 2.12

spark framework pipeline spark-structured-streaming streaming kafka streaming-etl ingestion apache-spark

25 9 10

tupol/spark-utils

Basic framework utilities to quickly start writing production ready Apache Spark applications

Scala (JVM): 2.11 2.12

spark scala spark-streaming framework data-sink apache-spark data-source spark-applications convenience

24 5 1

isarn/isarn-sketches-spark

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

Scala (JVM): 2.10 2.11 2.12

spark scala datasets sketching-algorithm data-sketches aggregator udaf variable-importance data-sketching pyspark feature-importance dataframe python dataframes spark-ml dataset t-digest apache-spark

23 6 2

heartsavior/spark-state-tools

Spark Structured Streaming State Tools

Scala (JVM): 2.11 2.12

apache-spark structured-streaming

22 4 2

itspawanbhardwaj/spark-fuzzy-matching

Fuzzy matching function in spark (https://spark-packages.org/package/itspawanbhardwaj/spark-fuzzy-matching)

Scala (JVM): 2.10 2.11

fuzzy-matching apache-spark scala

16 5 1

liquidsvm/liquidsvm

Support vector machines (SVMs) and related kernel-based learning algorithms are a well-known class of machine learning algorithms, for non-parametric classification and regression. liquidSVM is an implementation of SVMs whose key features are: fully integrated hyper-parameter selection, extreme speed on both small and large data sets, full flexibility for experts, and inclusion of a variety of different learning scenarios: multi-class classification, ROC, and Neyman-Pearson learning, and least-squares, quantile, and expectile regression.

Scala (JVM): 2.11

c-plus-plus machine-learning expectile-regression r-package octave ml rstats r python svm apache-spark regression classification matlab quantile-regression

14 2 3

Sorting

Filters

treeverse/lakefs

microsoft/mobius

lucacanali/sparkmeasure

hydrospheredata/mist

whylabs/whylogs-java

azure/azure-cosmosdb-spark

azure/azure-event-hubs-spark

kakao/cuesheet

streamnative/pulsar-spark

swoop-inc/spark-records

chermenin/spark-states

uosdmlab/spark-nkp

bizreach/aws-kinesis-scala

flipkart-incubator/spark-transformers

absaoss/hyperdrive

tupol/spark-utils

isarn/isarn-sketches-spark

heartsavior/spark-state-tools

itspawanbhardwaj/spark-fuzzy-matching

liquidsvm/liquidsvm