When using the kafka connector, you need to set the topics in advance (by giving a list of them or a regex pattern for the topic names). Imagine a situation where the topics are not known in advance, of course you could use an all-pass regex pattern to match all the topics in the broker but what I want to know is whether it's possible to read from new topics on demand.
E.g., initially the source starts without any topics to read from so nothing is read until it gets a control msg (which could be pushed to a control topic, for example) specifying the set of topics to subscribe to. I guess this could be somehow implemented using custom subscribers once this issue is merged/closed: https://issues.apache.org/jira/browse/FLINK-24660 but would it be possible to achieve this objective without having to periodically pull the broker, e.g., in a more reactive (push) way? I guess if the kafka source (or any other source for what it's worth) were to have a control signal like that then it would be more of an operator than a source, really... Salva PS: Does anyone know the current state of FLINK-24660? The changes seem to have been ready to merge for a while.