exasol / spark-exasol-connector

A connector for Apache Spark to access Exasol

Version Matrix

Spark Exasol Connector

spark-exasol-connector logo

Build Status Codecov Coveralls Maven Central

Overview

Spark Exasol Connector supports an integration between Exasol and Apache Spark. It allows creating Spark DataFrame from Exasol queries and saving Spark DataFrame as an Exasol table.

Features

  • Creates Spark DataFrame from Exasol query results
  • Saves Spark DataFrame to an Exasol table
  • Allows configuring the Spark tasks for parallel connection
  • Supports Spark DataFrame column pruning and filter push down

Information for Users

Information for Contributors

Dependencies

In this section, we define all the dependencies together with their licenses that are required for building, testing and running the connector.

The Java 8 is required for compiling and building the project. In the future versions of Spark, we are planning to change to the newer JVM versions.

Runtime Dependencies

Dependency Purpose License
Exasol JDBC Accessing Exasol using JDBC and sub-connections MIT License
Spark Core Apache Spark core libraries for optimized computation Apache License 2.0
Spark SQL Apache Spark higher-level SQL and Dataframe interface libraries Apache License 2.0

Test Dependencies

Dependency Purpose License
Scalatest Testing tool for Scala and Java developers Apache License 2.0
Scalatest Plus Integration support between Scalatest and Mockito Apache License 2.0
Mockito Core Mocking framework for unit tests MIT License
Testcontainers JDBC Testcontainers JDBC to help create JDBC based Docker containers MIT License
Testcontainers Scala Scala wrapper for testcontainers-java MIT License
Spark Testing Base Library that helps to create tests for Spark applications Apache License 2.0

Compiler Plugin Dependencies

These plugins help with project development.

Plugin Name Purpose License
SBT Coursier Pure Scala artifact fetching Apache License 2.0
SBT Wartremover Flexible Scala code linting tool Apache License 2.0
SBT Wartremover Contrib Community managed additional warts for wartremover Apache License 2.0
SBT Assembly Create fat jars with all project dependencies MIT License
SBT API Mappings Plugin that fetches API mappings for common Scala libraries Apache License 2.0
SBT Scoverage Integrates the scoverage code coverage library Apache License 2.0
SBT Updates Checks Maven and Ivy repositories for dependency updates BSD 3-Clause License
SBT Scalafmt Plugin for https://scalameta.org/scalafmt/ formatting Apache License 2.0
SBT Scalastyle Plugin for http://www.scalastyle.org/ Scala style checker Apache License 2.0
SBT Dependency Graph Plugin for visualizing dependency graph of your project Apache License 2.0
SBT Sonatype Sbt plugin for publishing Scala projects to the Maven central Apache License 2.0
SBT PGP PGP plugin for sbt BSD 3-Clause License
SBT Git Plugin for Git integration, used to version the release jars BSD 2-Clause License