TL;DR: Kafka Streams Scala Application: sbt new monksy/kafka-streams.g8

I’ve been working on a new project, which I’ll give more details on later, that involves a few Kafka Streams applications. As with any new Scala project, you should use a G8 template to start out with it. Starting out with a premade template is just a good practice. Additionally, templates can do a lot of work for you when it comes to good practices, structures, and extra compilation tools.

Unfortunately, with the Confluence/Kafka world, there aren’t many Kafka-Streams Scala based templates. On Github I found two that came up in the search:

idarlington/kafka-streams.g8 sv3ndk/kafka-streams-scala.g8

The sv3ndk template used the Lightbend version of the kafka-streams-scala. It’s very out of date. So that was out. The idarlington template used Kafka Streams 2.0. Not ideal, but not unworkable. So I forked it.

What was done:

  • Upgrade the Kafka version from 2 to 2.5 (Latest Release)
  • Upgrade the testing utilties to use the non-depricated (post 2.4) functionality
  • Improved the file layout
  • Added Assembly support
  • Added a larger gitingore file
  • Upgraded the project to Scala 2.13
  • Upgraded the other libraries in the project
  • Added dependency tree plugin support
  • Upgraded the g8 build file

What did I learn about?

  • I learned a lot about creating a G8 template and how the variables are substituted. The existing project had a lot of this work already done, however, I did have to do some of my own substitutions.
  • MergeStrategies and dealing with “module-info.class” in the assembly plugin. (Hint: Merge strategy rule for: case “module-info.class” => MergeStrategy.discard) The module-info.class is a new addition that came from Java 9’s Jigsaw to define JVM modules. They’ll pop up in the Jackson libraries.
  • Some of the built-in Kafka Streams Test utilities. For the most part, I’ve been using the mocked streams library.
  • G8 Sbt plugin. Use sbt g8 to build an example copy of the application in target/g8. From there it’s a lot easier to test and build up your template. Use sbt g8Test to run an automated test. I’m not sure how to customize the sbt tasks. I’m sure it’s a configuration option.