Search *

35 results

microsoft/synapseml

Simple and Distributed Machine Learning

Scala versions: 2.11

deep-learning databricks apache-spark onnx lightgbm azure cognitive-services opencv big-data data-science spark ai microsoft machine-learning ml pyspark synapse scala http model-deployment

3770 86
feathr-ai/feathr

Feathr – An Enterprise-Grade, High Performance Feature Store

Scala versions: 2.12

feature-governance feature-management apache-spark azure data-science feature-engineering artificial-intelligence feature-store machine-learning feature-metadata mlops data-engineering feature-marketplace data-quality feature-platform

1053 372 25
lucacanali/sparkmeasure

This is the development repository for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.

Scala versions: 2.11 2.12 2.13

apache-spark performance-metrics performance-troubleshooting spark-troubleshooting spark

496 10
hydrospheredata/mist

Serverless proxy for Spark cluster

Scala versions: 2.10 2.11 2.12

apache-spark api big-data serverless

318 23
azure/azure-event-hubs-spark

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Scala versions: 2.10 2.11 2.12

apache-spark streaming azure eventhubs kafka stream scala real-time microsoft spark-streaming bigdata continuous databricks structured-streaming spark apache ingestion connector event-hubs

208 36
azure/azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB

Scala versions: 2.10 2.11

databricks apache-spark lambda-architecture cosmos-db databricks-notebooks azure-databricks connector azure-cosmos-db jupyter-notebook pyspark spark changefeed

187 23
treeverse/lakefs

Git-like capabilities for your object storage

Scala versions: 2.11 2.12

apache-sparksql data-version-control data-engineering datalake aws-s3 data-quality go lakefs data-lake git-for-data object-storage hadoop-filesystem golang apache-spark hacktoberfest data-versioning datalakes google-cloud-storage azure-blob-storage

2847 812 66
microsoft/mobius

C# and F# language binding and extensions to Apache Spark

Scala versions: 2.10 2.11

bigdata apache-spark streaming mapreduce dataframe eventhubs kafka-streaming rdd dstream spark-streaming near-real-time csharp mobius spark dataset fsharp

931 36
chermenin/spark-states

Custom state store providers for Apache Spark

Scala versions: 2.11 2.12

structured-streaming apache-spark state spark-structured-streaming stateful state-store spark-streaming spark apache

90 3
streamnative/pulsar-spark

When Apache Pulsar meets Apache Spark

Scala versions: 2.11 2.12

structured-streaming apache-spark apache-pulsar flink data-science batch-processing stream-processing spark-sql spark data-processing

88 12
kakao/cuesheet

A framework for writing Spark 2.x applications in a pretty way

Scala versions: 2.10 2.11

apache-spark yarn scala magic spark mango

85 2
sansa-stack/sansa-stack

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

Scala versions: 2.11 2.12

rdf apache-spark flink spark semantic-web apache-jena distributed-computing

128 35
swoop-inc/spark-records

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

Scala versions: 2.12

apache-spark big-data sparksql scala spark-records spark swoop

68 4
databrickslabs/automl-toolkit

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

Scala versions: 2.11

apache-spark feature-engineering spark scala ml pyspark machinelearning

184 7
uosdmlab/spark-nkp

Natural Korean Processor for Apache Spark

Scala versions: 2.11

nlp apache-spark text-mining korean-nlp spark natural-language-processing spark-mllib

53 2
bizreach/aws-kinesis-scala

Scala client for Amazon Kinesis. Also provides write to Kinesis capability for Apache Spark or Spark Streaming.

Scala versions: 2.11 2.12 2.13

aws apache-spark spark scala kinesis aws-kinesis

36 5
heartsavior/spark-state-tools

Spark Structured Streaming State Tools

Scala versions: 2.11 2.12

apache-spark structured-streaming

34 2
absaoss/hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark

Scala versions: 2.11 2.12

apache-spark streaming spark-structured-streaming framework pipeline kafka streaming-etl spark ingestion

32 10
tupol/spark-utils

Basic framework utilities to quickly start writing production ready Apache Spark applications

Scala versions: 2.11 2.12

apache-spark convenience data-source framework data-sink spark scala spark-applications spark-streaming

28 1
coxautomotivedatasolutions/spark-distcp

A re-implementation of Hadoop DistCP in Apache Spark

Scala versions: 2.11 2.12 2.13

apache-spark data-engineering distcp hadoop spark

25 3