Richard,
There is; but it doesn't look like what you would do in a typical
messaging technology. As you may know Apache Kafka® is not messaging but
a distributed streaming platform based on the concept of commit log. The
main characteristic of a commit log is that they keep records in the
same order that they were written (hence the immutable nature) and
therefore the ordering cannot be changed in a broker level. With this in
mind what you can do is:
A. *Ensure Priority using Partitions*: Break down the topic into
multiple partitions where each partition represents a priority. Change
your producers to use a custom partitioner that writes records into a
given partition depending on the priority logic such as: "if the order
is from North America then send to partition 0 that represents platinum
orders.". This way your consumer groups will consume them as they come
based on their natural order without worrying about handling priority
themselves.
B. *Using the Resequencer Pattern
<https://www.enterpriseintegrationpatterns.com/patterns/messaging/Resequencer.html>*:
Have a dedicated stream processor (implemented as a Kafka consumer, as a
Kafka Streams app, or as a ksqlDB app) that reads all the records from
the input topic and perform a routing to specific topics based on the
data contained in the records -- preferably data from the headers for
improved performance. This way you can have dedicated consumer groups
each one reading from specific topics without worrying about handling
priority themselves.
Either option works, though personally I fancy option A since it
provides a less intrusive solution with less layers being added to the
final architecture.
-- Ricardo
On 7/21/20 12:52 PM, Richard Ney wrote:
Trying to see if there is a way to implement priority topics where a single
consumer group is reading from multiple topics each denoting a level of
priority and having the Kafka client automatically prioritize reads against
the message in the higher priority topics.
