This is a Flink Streaming MQTT Source And MQTT Sink library.

The library for writing and reading data from MQTT Servers using Flink Streaming.

This library comes from Spark SQL Streaming MQTT Data Source changed (https://github.com/apache/bahir/tree/master/sql-streaming-mqtt).

Using SBT:

libraryDependencies += "io.github.kevin4936" %% "kevin-flink-connector-mqtt3" % "{{site.FLINK_VERSION}}"

or

libraryDependencies += "io.github.kevin4936" % "kevin-flink-connector-mqtt3_2.12" % "1.14.6"

Using Maven:

<dependency>
    <groupId>io.github.kevin4936</groupId>
    <artifactId>kevin-flink-connector-mqtt3_2.12</artifactId>
    <version>1.14.6</version>
</dependency>

mvn clean install -DskipTests -Denforcer.skip=true

Unlike using --jars, using --packages ensures that this library and its dependencies will be added to the classpath.

This library is cross-published for Scala 2.12, so users should replace the proper Scala version in the commands listed above.

Please see https://github.com/kevin4936/kevin-flink-connector-mqtt3/blob/main/src/test/java/com/kevin/flink/streaming/connectors/mqtt/MQTTStreamSourceAndSinkTest.java for full example.

Please see https://github.com/kevin4936/flink-mqtt3 for full example.

This connector uses Eclipse Paho Java Client. Client API documentation is located here.

Custom environment variables allowing to manage MQTT connectivity performed by sink connector:

• flink.mqtt.client.connect.attempts Number of attempts sink will try to connect to MQTT broker before failing.
• flink.mqtt.client.connect.backoff Delay in milliseconds to wait before retrying connection to the server.
• flink.mqtt.connection.cache.timeout Sink connector caches MQTT connections. Idle connections will be closed after timeout milliseconds.
• flink.mqtt.client.publish.attempts Number of attempts to publish the message before failing the task.
• flink.mqtt.client.publish.backoff Delay in milliseconds to wait before retrying send operation.

1. Turn Mqtt into a more reliable messaging service.

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport.

The design of Mqtt and the purpose it serves goes well together, but often in an application it is of utmost value to have reliability. Since mqtt is not a distributed message queue and thus does not offer the highest level of reliability features. It should be redirected via a kafka message queue to take advantage of a distributed message queue. In fact, using a kafka message queue offers a lot of possibilities including a single kafka topic subscribed to several mqtt sources and even a single mqtt stream publishing to multiple kafka topics. Kafka is a reliable and scalable message queue.

2. What is the solution for a situation when there are a large number of varied mqtt sources, each with different schema and throughput characteristics.

Generally, one would create a lot of streaming pipelines to solve this problem. This would either require a very sophisticated scheduling setup or will waste a lot of resources, as it is not certain which stream is using more amount of data.

The general solution is both less optimum and is more cumbersome to operate, with multiple moving parts incurs a high maintenance overall. As an alternative, in this situation, one can setup a single topic kafka-flink stream, where message from each of the varied stream contains a unique tag separating one from other streams. This way at the processing end, one can distinguish the message from one another and apply the right kind of decoding and processing. Similarly while storing, each message can be distinguished from others by a tag that distinguishes.