tharwaninitin / etlflow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.

Version Matrix

EtlFlow

Core Workflow Job Workflow Server Workflow Maven Central javadoc

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.

Documentation

Library Documentation https://tharwaninitin.github.io/etlflow/site/docs

Examples

You can use this library in different ways mentioned below

  • Core Module :
    Using just this module you can use Step API into your project.
  • Spark Module (spark steps) :
    Using this addon module along with core you can use spark steps into your project.
  • Job Module :
    Using this module along with core you can use Job API into your project.
  • Server Module :
    Using this module along with job you can use Server API into your project.

Modules Dependency Graph

ModuleDepGraph

Scala Version Compatibility Matrix

Internal Modules

Module Name Scala 2.12 Scala 2.13 Scala 3.0
Utils
Json
Cache
Crypto
Db

Main Modules

Module Name Scala 2.12 Scala 2.13 Scala 3.0
Core
Job
Server

Addon Modules For Core

Module Name Scala 2.12 Scala 2.13 Scala 3.0
Http
Cloud
Email
Aws
Gcp
Redis
Spark

Requirements and Installation

This project is compiled with scala versions 2.12.15, 2.13.7, 3.1.0 Available via maven central. Add the latest release as a dependency to your project

Latest Version

SBT

libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-job" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-server" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-cloud" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "x.x.x"

Maven

<dependency>
    <groupId>com.github.tharwaninitin</groupId>
    <artifactId>etlflow-core_2.12</artifactId>
    <version>x.x.x</version>
</dependency>

Contributions

Please feel free to add issues to report any bugs or to propose new features.