[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770704#comment-17770704
]
Domenico Di Giulio edited comment on KAFKA-6690 at 9/30/23 12:54 PM:
---------------------------------------------------------------------
I see this issue was closed but we also really need it A LOT.
In our case two co-partitioned source topics are used in the same topology, one
(let's call it topic "A") to accept new business requests, the other (let's
call it topic "B") as a loopback (from downstream components) about business
transactions already in progress, which come back to the same topology to
update a transaction status (in a store) and proceed.
We found that when the event rate on topic A is too high, Kafka Streams has no
way to apply any backpressure on it and favor topic B. As a result, the
processing on topic B progressively goes in starvation and the system proceeds
much slower than it could. We're looking at how we could apply a backpressure
on topic A, fetching more events from topic B so that the processing of ongoing
business transactions can proceed with higher priority.
So far we only thought about adding another upstream component (may be just a
simple consumer and producer), which should read topic A and forward events to
our topology based on a metric which looks for symptoms of congestion (business
latency or whatever). But it would be very beneficial instead to just define B
as a priority topic on Kafka Streams.
Any news on this issue ? Does the above use case look meaningful ?
was (Author: domenico74):
I see this issue was closed but we also really need it A LOT.
In our case two co-partitioned source topics are used in the same topology, one
(let's call it topic "A") to accept new business requests, the other (let's
call it topic "B") as a loopback (from downstream components) about business
transactions already in progress, which come back to the same topology to
update a transaction status (in a store) and proceed.
We found that when the event rate on topic A is too high, Kafka Streams has no
way to apply any backpressure on it and favor topic B. As a result, the
processing on topic B progressively goes in starvation and the system proceeds
much slower than it could. We're looking at how we could apply a backpressure
on topic A, fetching more events from topic B so that the processing of ongoing
business transactions can proceed with higher priority.
So far we only thought about adding another upstream component (may be just a
simple consumer and producer), which should read topic A and forward events to
our topology based on a metric which looks for symptoms of congestion (business
latency or whatever). But it would be very beneficial instead to just define B
as a priority topic on Kafka Streams.
Any news on this KIP ? Does the above use case look meaningful ?
> Priorities for Source Topics
> ----------------------------
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: consumer
> Reporter: Bala Prassanna I
> Assignee: Nick Afshartous
> Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
