Read Azure Data Lake Storage into Akka Streams
Replay historical data-at-rest into an existing code base that had been designed for streaming.
Current Storage Sources
- GZip files of UTF8
- Other storage implementations TBD
Uses the adslapi.
build.sbt dependencies with:
// https://mvnrepository.com/artifact/tech.navicore/navilake libraryDependencies += "tech.navicore" %% "navilake" % "1.3.0"
This example reads gzip data from Azure Data Lake.
Create a config, a connector, and a source via the example below.
val consumer = ... // some Sink ... ... ... // credentials and location implicit val cfg: LakeConfig = LakeConfig(ACCOUNTFQDN, CLIENTID, AUTHEP, CLIENTKEY, Some(PATH)) val connector: ActorRef = actorSystem.actorOf(GzipConnector.props) val src = NaviLake(connector) ... ... ... src.runWith(consumer) ... ... ...