> On Jan 16, 2019, at 9:51 PM, n...@afshartous.com wrote:
>
> Hi all,
>
> On the dev list we’ve been discussing a proposed new feature (prioritized
> topics). In a nutshell, when consuming from a set of topics with assigned
> priorities, consumption from lower-priority topics only occurs if
Hi,
This sounds like a great idea, and thanks for reaching out for feedback.
Here are two use cases I've worked on that I'd seriously consider using
such feature for:
1. Priority Republish of Data - in an event driven system, there's a
"republish" functionality used for e.g. fixing data affected
Nick, I think it's worth noting that Kafka is not a real-time system, and
most of these use cases (and TBH any that I can imagine) for prioritized
topics are real-time use cases.
For example, you wouldn't want to pilot a jet (a canonical real-time
system) based on Kafka events, as there is no guar
Use case: we process documents from a variety of sources. We want to
process some of these sources in a priority order, but we don’t want to
necessarily finish all the higher priority sources before going to lower
priority because the volume of higher priority sources can be extremely
high.
We hav
Use case:
I work for a company that ingests events that come from both real-time
sources (which spike during the day) and historical log data.
We want the real-time data processed in minutes, and the historical log
data processed within hours. The consumer's business logic is the same.
Our curren
Use cases: prioritise current data
When processing messages sometimes there is a need to re process old data.
It would be nice to be abled to send the old data as messages to a separate
topic and that would only be processed when the current topic doesn’t have any
messages left to process.
This
Use cases: processing alerts.
High priority alerts ("a large chunk of your system has stopped providing
service, immediate action essential") should be processed before low priority
alerts ("some minor component has put out a not-very serious warning, somebody
should probably have a look at it
