spotify/scio

A Scala API for Apache Beam and Google Cloud Dataflow.

spotify/ratatool

A tool for data sampling, data generation, and data diffing

spotify/magnolify

A collection of Magnolia add-on modules

nevillelyh/shapeless-datatype

Shapeless utilities for common data types

samelamin/spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

googleclouddataproc/spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

tharwaninitin/etlflow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases and more.

wix/quix

Quix Notebook Manager

starlake-ai/starlake

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

ocadaruma/scalikejdbc-bigquery

ScalikeJDBC extension for Google BigQuery

cartodb/analytics-toolbox-core

A set of UDFs and Procedures to extend BigQuery, Snowflake, Redshift, Postgres and Databricks with Spatial Analytics capabilities

data-tools/big-data-types

A library to transform Scala product types and Schemas from different databases into other Schemas. Any implemented type gets automatically methods to convert it into the rest of the types and vice versa. For example, an Spark Schema can be transformed into a BigQuery table.

spotify/scio-contrib

Community-supported add-ons for Scio

armanbilge/gcp4s

Cross-platform JVM/JS Google Cloud Platform integrations for fs2 and friends

fpopic/bigquery-schema-select

(Script) Generates SQL query that selects all fields (recursively for nested fields) from the provided BigQuery schema file.