spotify/scio

A Scala API for Apache Beam and Google Cloud Dataflow.

spotify/ratatool

A tool for data sampling, data generation, and data diffing

spotify/magnolify

A collection of Magnolia add-on modules

nevillelyh/shapeless-datatype

Shapeless utilities for common data types

googleclouddataproc/spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

samelamin/spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

tharwaninitin/etlflow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

wix-incubator/quix

Quix Notebook Manager

starlake-ai/starlake

Starlake is an On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

ocadaruma/scalikejdbc-bigquery

ScalikeJDBC extension for Google BigQuery

data-tools/big-data-types

A library to transform Scala product types and Schemas from different systems into other Schemas. Any implemented type gets automatically methods to convert it into the rest of the types and vice versa. For example, an Spark Schema can be transformed into a BigQuery table.

cartodb/analytics-toolbox-core

A set of UDFs and Procedures to extend BigQuery, Snowflake, Redshift, Postgres and Databricks with Spatial Analytics capabilities

spotify/scio-contrib

Community-supported add-ons for Scio

armanbilge/gcp4s

Cross-platform JVM/JS Google Cloud Platform integrations for fs2 and friends

fpopic/bigquery-schema-select

(Script) Generates SQL query that selects all fields (recursively for nested fields) from the provided BigQuery schema file.