Matthias J. Sax created KAFKA-16508:
---------------------------------------
Summary: Infinte loop if output topic does not exisit
Key: KAFKA-16508
URL: https://issues.apache.org/jira/browse/KAFKA-16508
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
Kafka Streams supports `ProductionExceptionHandler` to drop records on error
when writing into an output topic.
However, if the output topic does not exist, the corresponding error cannot be
skipped over because the handler is not called.
The issue is, that the producer internally retires to fetch the output topic
metadata until it times out, an a `TimeoutException` (which is a
`RetriableException`) is returned via the registered `Callback`. However, for
`RetriableException` there is different code path and the
`ProductionExceptionHandler` is not called.
In general, Kafka Streams correctly tries to handle as many errors a possible
internally, and a `RetriableError` falls into this category (and thus there is
no need to call the handler). However, for this particular case, just retrying
does not solve the issue – it's unclear if throwing a retryable
`TimeoutException` is actually the right thing to do for the Producer? Also not
sure what the right way to address this ticket would be (currently, we cannot
really detect this case, except if we would do some nasty error message String
comparison what sounds hacky...)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
