A Scala-based DSLink implementation for Apache Spark.
- DSAReceiver class for communicating with DSA broker.
To start streaming data from one or more DSA nodes, use StreamingContext.receiverStream()
method.
val sc = new SparkContext(...)
val ssc = new StreamingContext(sc, ...)
val stream = ssc.receiverStream(new DSAReceiver(path1, path2, ..., pathN))
where path1, path2, ..., pathN are paths in the DSA tree.
The type of the returned stream will be DStream[(String, Date, Any)]
with the elements as:
- The node path
- The time of the last update
- The updated value
If you want stream flow to publish the updates in the node tree, use DSAConnector.updateNode()
method:
stream foreachRDD (_ foreach { point =>
DSAHelper.updateNode(path1, point.x)
DSAHelper.updateNode(path2, point.y)
...
})
All paths should be relative to the owner's DSLink's root node.