PagerDuty Kafka Consumer
This is an open source project!
This is an opinionated wrapper around the Kafka consumer, adding a default set of policies/interactions. It also specifies the version of the Kafka client library to use as a transitive dependency (so don't double-specify it if you're depending on this library) and provides a base test class for Kafka-eske integration tests.
*** NOTE: This library uses Java 8 functionality. ***
Installation
This library is published as a number of different artifacts to the PagerDuty Bintray OSS Maven repository:
resolvers += "bintray-pagerduty-oss-maven" at "https://dl.bintray.com/pagerduty/oss-maven"
Currently, there are two different consumers available:
-
SimpleKafkaConsumer
- As the name suggests, this consumer is very simple to use and has few dependencies. It is available as follows:libraryDependencies += "com.pagerduty" %% "kafka-consumer" % "<version>"
-
PartitionedKafkaConsumer
- This consumer improves onSimpleKafkaConsumer
at the expense of an additional dependency (Akka Streams) and no support for 2.10. This consumer processes and commits Kafka messages for each assigned partition independently and in parallel. Therefore, this consumer is both faster and more tolerant of transient processing failures. It is available as follows:libraryDependencies += "com.pagerduty" %% "kafka-consumer-partitioned" % "<version>"
Usage
This library is published as a number of separate artifacts
Create a new SimpleKafkaConsumer
or PartitionedKafkaConsumer
and get cranking. Usually you need three items to get a default consumer up and running:
- The topic the consumer works on (constructor argument)
- The consumer group your consumer belongs to (passed through properties)
- A bootstrap server to start talking to the Kafka cluster (passed through properties)
For testing, you can re-use/build on the classes in the com.pagerduty.kafkaconsumer.testsupport
package, which contains a base Spec to inherit from if you need a Kafka integration test.
Release
Follow these steps to release a new version:
- Update version.sbt in your PR
- Update CHANGELOG.md in your PR
- When the PR is approved, merge it to master, and delete the branch
- Travis will run all tests, publish to Artifactory, and create a new version tag in Github