John Roesler created KAFKA-13523:
------------------------------------
Summary: Implement IQv2 support in global stores
Key: KAFKA-13523
URL: https://issues.apache.org/jira/browse/KAFKA-13523
Project: Kafka
Issue Type: Sub-task
Reporter: John Roesler
Global stores pose one significant problem for IQv2: when they start up, they
skip the regular ingest pipeline and instead use the "restoration" pipeline to
read up until the current end offset. Then, they switch over to the regular
ingest pipeline.
IQv2 position tracking expects to track the position of each record from the
input topic through the ingest pipeline and then get the position headers
through the restoration pipeline via the changelog topic. The fact that global
stores "restore" the input topic instead of ingesting it violates our
expectations.
It has also caused other problems, so we may want to consider switching the
global store processing to use the normal paradigm rather than adding
special-case logic to track positions in global stores.
Note: there are two primary reasons that global stores behave this way:
# We can write in batches during restoration, so the I/O may be more efficient
# The global thread does not transition to RUNNING state until it reaches the
(current) end of the input topic, which blocks other threads from joining
against it, thereby improving the time synchronization of global KTable joins.
If we want to propose changing the bootstrapping pipeline for global threads,
we should have some kind of answer to these concerns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
