[
https://issues.apache.org/jira/browse/KAFKA-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Coates updated KAFKA-9416:
-------------------------------
Description:
We've noticed that if topics are deleted from under a running topology, e.g.
repartition, changelog or sink topics, then the stream threads transition from
`RUNNING` to `PENDING_SHUTDOWN`, but then do not transition to `ERROR`.
Likewise, if a Kafka cluster has auto topic create disabled and a topology is
started where its sink topic(s) do not exist, then the topology similarly gets
stuck in `PENDING_SHUTDOWN`.
Once the query is stuck in `PENDING_SHUTDOWN` any call to close the topology
blocks, as per https://issues.apache.org/jira/browse/KAFKA-9398.
We would like to see Kafka Streams handle this case and correctly transition to
`ERROR` state.
The root cause of this looks to be that we have a large `max.block.ms` set on
the KS Producer and consumer. The producer can get stuck trying to produce to
a topic that does not exist. Likewise, a consumer can get stuck trying to
commit offsets for a topic that does not exist.
The cause of this is covered in more detail here:
[https://github.com/confluentinc/ksql/issues/4268#issuecomment-573020281]
was:
We've noticed that if topics are deleted from under a running topology, e.g.
repartition, changelog or sink topics, then the stream threads transition from
`RUNNING` to `PENDING_SHUTDOWN`, but then do not transition to `ERROR`.
Likewise, if a Kafka cluster has auto topic create disabled and a topology is
started where its sink topic(s) do not exist, then the topology similarly gets
stuck in `PENDING_SHUTDOWN`.
Once the query is stuck in `PENDING_SHUTDOWN` any call to close the topology
blocks, as per https://issues.apache.org/jira/browse/KAFKA-9398.
We would like to see Kafka Streams handle this case and correctly transition to
`ERROR` state.
The cause of this is covered in more detail here:
https://github.com/confluentinc/ksql/issues/4268#issuecomment-573020281
> Streams get stuck in `PENDING_SHUTDOWN` is underlying topics deleted.
> ---------------------------------------------------------------------
>
> Key: KAFKA-9416
> URL: https://issues.apache.org/jira/browse/KAFKA-9416
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.0
> Reporter: Andy Coates
> Priority: Major
>
> We've noticed that if topics are deleted from under a running topology, e.g.
> repartition, changelog or sink topics, then the stream threads transition
> from `RUNNING` to `PENDING_SHUTDOWN`, but then do not transition to `ERROR`.
> Likewise, if a Kafka cluster has auto topic create disabled and a topology is
> started where its sink topic(s) do not exist, then the topology similarly
> gets stuck in `PENDING_SHUTDOWN`.
> Once the query is stuck in `PENDING_SHUTDOWN` any call to close the topology
> blocks, as per https://issues.apache.org/jira/browse/KAFKA-9398.
> We would like to see Kafka Streams handle this case and correctly transition
> to `ERROR` state.
> The root cause of this looks to be that we have a large `max.block.ms` set on
> the KS Producer and consumer. The producer can get stuck trying to produce
> to a topic that does not exist. Likewise, a consumer can get stuck trying to
> commit offsets for a topic that does not exist.
> The cause of this is covered in more detail here:
> [https://github.com/confluentinc/ksql/issues/4268#issuecomment-573020281]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
