A library for reading public web news results from Bing Custom Search using Spark Streaming.
Run a demo via:
# set up all the requisite environment variables export BING_SEARCH_INSTANCE_ID="..." export BING_AUTH_TOKEN="..." # compile scala, run tests, build fat jar sbt assembly # run locally java -cp target/scala-2.11/streaming-bing-assembly-0.0.7.jar BingDemo standalone # run on spark spark-submit --class BingDemo --master local target/scala-2.11/streaming-bing-assembly-0.0.7.jar spark
How does it work?
Bing Custom Search doesn't support streamed web results so we currently poll the service based on a polling interval rate. The BingReceiver pings the Bing Search API every few seconds and pushes any newly indexed web results into Spark Streaming for further processing.
- Configure your credentials via the
sbt sonatypeOpen "enter staging description here"