Hey everyone,
In 1.1, kafka-streams adds an artifact called 'kafka-streams-test-utils'
(see
https://kafka.apache.org/11/documentation/streams/developer-guide/testing.html
).
The basic idea is to provide first-class support for testing Kafka Streams
applications. Without that, users were forced to either depend on our
internal test artifacts or develop their own test utilities, neither of
which is ideal.
I think it would be great if all our APIs offered a similar module, and it
would all be good if we followed a similar pattern, so I'll describe the
streams approach along with one challenge we had to overcome:
=====================
= Project Structure =
=====================
The directory structure goes:
kafka/streams/ <- main module code here
/test-utils/ <- test utilities module here
/examples/ <- example usages here
Likewise, the artifacts are:
kafka-streams
kafka-streams-test-utils
kafka-streams-examples
And finally, the Gradle build structure is:
:streams
:streams:test-utils
:streams:examples
=============================
= Problem 1: circular build =
=============================
In eat-your-own-dogfood tradition, we wanted to depend on our own
test-utils in our streams tests, but :streams:test-utils (obviously)
depends on :streams already.
(:streams) <-- (:streams:test-utils)
\--->
Luckily, Filipe Agapito found a way out of the conundrum (
https://issues.apache.org/jira/browse/KAFKA-6474?focusedCommentId=16402326&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16402326).
Many thanks to him for this contribution.
* Add this to the ':streams' definition:
testCompile project(':streams:test-utils').sourceSets.main.output
* And this to the ':streams:test-utils' definition:
compile project(':streams').sourceSets.main.output
* And finally (because we also have tests for the examples), add this to
the ':streams:examples' definition:
testCompile project(':streams:test-utils')
By scoping the dependencies to 'sourceSets.main', we break the cyclic
dependency:
(:streams main) <-- (:streams:test-utils main)
^ ^ ^
| / |
| / |
(:streams test) (:streams:test-utils test)
==============================================
= Problem 2: missing transitive dependencies =
==============================================
Scoping the dependency to source-only skips copying transitive library
dependencies into the build & test environment, so we ran into the
following error in our tests for ':streams:test-utils' :
java.lang.ClassNotFoundException: org.rocksdb.RocksDBException
This kind of thing is easy to resolve, once you understand why it happens.
We just added this to the :test-utils build definition:
testCompile libs.rocksDBJni
It's a little unfortunate to have to manually pull in transitive
dependencies for testing, but it's really the only downside of this
approach (so far).
That's about it! This is partly to propose a similar model across other
parts of Kafka's API and partly to collect feedback on this approach.
Thoughts?
-John
