tharwaninitin / etlflow Edit

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.

Version Matrix

EtlFlow

License Core Workflow Server Workflow Maven Central javadoc

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.

Documentation

Library Documentation https://tharwaninitin.github.io/etlflow/site/docs

Examples

You can use this library in different ways mentioned below.

  • Core Module:
    Using this module you can use features of Step API into your project.
  • Spark Module (spark steps):
    Using this addon module along with core you can use Apache Spark steps into your project.
  • GCP Module (spark steps):
    Using this addon module along with core you can use GCP steps into your project.
  • Server Module:
    Using this module along with job you can use additional features of Server API into your project.

Modules Dependency Graph

ModuleDepGraph

Scala Version Compatibility Matrix

Main Modules

Module Name Scala 2.12 Scala 2.13 Scala 3.1
Core
Server

Server Internal Modules

Module Name Scala 2.12 Scala 2.13 Scala 3.1
Json

Addon Modules For Core

Module Name Scala 2.12 Scala 2.13 Scala 3.1
Db
Http
Cloud
Email
Aws
Gcp
Redis
Spark

Requirements and Installation

This project is compiled with scala versions 2.12.15, 2.13.7, 3.1.0 Available via maven central. Add the latest release as a dependency to your project

Latest Version

SBT

libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-server" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-cloud" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "x.x.x"

Maven

<dependency>
    <groupId>com.github.tharwaninitin</groupId>
    <artifactId>etlflow-core_2.12</artifactId>
    <version>x.x.x</version>
</dependency>

Contributions

Please feel free to add issues to report any bugs or to propose new features.