[
https://issues.apache.org/jira/browse/KAFKA-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082871#comment-18082871
]
sanghyeok An commented on KAFKA-20429:
--------------------------------------
[~lianetm]
Sorry to bother you!
When you get a chance, could you take a look?
> Review async consumer single background queue design
> ----------------------------------------------------
>
> Key: KAFKA-20429
> URL: https://issues.apache.org/jira/browse/KAFKA-20429
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: sanghyeok An
> Priority: Major
> Fix For: 4.4.0
>
>
> The async consumer relies on a single queue to pass events from the
> background to the app thread. But this queue mixes different kinds of events
> (coord errors, metadata errors, callbacks, internal updates events like
> applyAssignment).
> This makes it problematic and complicated to use. When
> processingBackgroundEvents from the app thread, we need to filter-out events
> depending on what we're processing background events for. E.g., :
> * unsubscribe cares about callbacks only, needs to ignore all others (error
> prone as new events are added, see
> https://issues.apache.org/jira/browse/KAFKA-20428)
> * poll cares about all, but actually could maybe benefit from the ability to
> process errors and callbacks at different points of the poll loop (see
> https://issues.apache.org/jira/browse/KAFKA-20397)
> I wonder if re-designing the background events queue would simplify it's
> usage from the AsyncKafkaConsumer, and avoid continuously falling into errors
> when new events are added. Also we could have a simpler/clearer intention
> from the app thread, consuming from each func what it needs (vs consume all
> and filter out).
> One option that comes to mind is to have separate queues for errors,
> callbacks, internal ops. So app thread should be simpler (explicit intention
> when processing events from a queue), but to review in detail to see if we
> could get a simpler design, less error prone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
