Input Sources is an abstraction for loading Spark data via configuration files. Currently, it can handle
- file path sources
- table sources
- SQL sources
- BigQuery sources
This library aims to be easily extended to other sources by using sealed trait with case classes for each new sources.
// https://central.sonatype.com/artifact/com.growingintech/spark-input-sources_2.12/1.0.1
libraryDependencies += "com.growingintech" %% "spark-input-sources" % "1.0.1"Feel free to submit a PR for any new sources you would like to add. I don't plan on creating cloud accounts for all clouds, so it will be helpful if others can work on Amazon and Azure.
In this simple example, we have a HOCON pipeline configuration string which can have as many parameters as needed for
the user's use case. For our data definition, I am using a TableSource example.
/*
* Copyright 2023 GrowingInTech.com. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not
* use this file except in compliance with the License. A copy of the License
* is located at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* or in the "license" file accompanying this file. This file is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
import com.growingintech.datasources.InputSources
import com.typesafe.config.ConfigFactory
import pureconfig._
import pureconfig.generic.auto._
import org.apache.spark.sql.DataFrame
val strConfig: String =
"""
|{
| pipeline-name: Data Runner
| date: 20230216
| data: {
| type: table-source
| table-name: default.test_data
| filter: "date = 20230101 AND x > 2"
| }
|}
|""".stripMargin
case class Params(
pipelineName: String,
date: Int,
data: InputSources
)
val config: Params = ConfigSource.fromConfig(ConfigFactory.parseString(strConfig)).loadOrThrow[Params]
val df: Dataframe = Params.data.loadData