When using the kafka connector, you need to set the topics in advance (by
giving a list of them or a regex pattern for the topic names). Imagine a
situation where the topics are not known in advance, of course you could
use an all-pass regex pattern to match all the topics in the broker but
what I want to know is whether it's possible to read from new topics on
demand.

E.g., initially the source starts without any topics to read from so
nothing is read until it gets a control msg (which could be pushed to a
control topic, for example) specifying the set of topics to subscribe to. I
guess this could be somehow implemented using custom subscribers once this
issue is merged/closed:

https://issues.apache.org/jira/browse/FLINK-24660

but would it be possible to achieve this objective without having to
periodically pull the broker, e.g., in a more reactive (push) way? I guess
if the kafka source (or any other source for what it's worth) were to have
a control signal like that then it would be more of an operator than a
source, really...

Salva

PS: Does anyone know the current state of FLINK-24660? The changes seem to
have been ready to merge for a while.

Reply via email to