Hi Gunnar, It's not possible to do this in a generalized fashion with the API provided by the framework today. Trying to hack your way around things by setting a flag or storing the connector name in some shared JVM state wouldn't work in a cluster with more than one worker since that state would obviously not be available across workers.
With the specific case of the Debezium PostgreSQL connector, I'm wondering if you might be able to store the name of the connector in some external system (likely either the database itself or a Kafka topic, as I seem to recall that Debezium connectors create and consume from topics outside of the framework) after successfully claiming the replication slot. Then, during config validation, you could skip the replication slot validation if that stored name matched the name of the connector being validated. There are obviously some edge cases that'd need to be addressed such as sudden death of connectors after claiming the replication slot but before storing their name; just wanted to share the thought in case it leads somewhere useful. Either way, I think a small, simple KIP for this would be fine, as long as we could maintain backwards compatibility for existing connectors and allow connectors that use this new API to work on older versions of Connect that don't have support for it. Cheers, Chris On Thu, Jan 21, 2021 at 6:00 AM Gunnar Morling <gun...@hibernate.org> wrote: > Hi, > > In the Debezium community, we ran into an interesting corner case of > connector config validation [1]. > > The Debezium Postgres connector requires a database resource called a > "replication slot", which identifies this connector to the database and > tracks progress it has made reading the TX log. This replication slot must > not be shared between multiple clients (Debezium connectors, or others), so > we added a validation to make sure that the slot configured by the user > isn't active, i.e. no client is connected to it already. This works as > expected when setting up, or restarting a connector, but when trying to > update the connector configuration, the connector still is running when the > configuration is validated, so the slot is active and validation hence > fails. > > Is there a way we can distinguish during config validation whether the > connector is (re-)started or whether it's a validation upon > re-configuration (allowing us to skip this particular validation in the > re-configuration case)? > > If that's not the case, would there be interest for a KIP for adding such > capability to the Kafka Connect API? > > Thanks for any feedback, > > --Gunnar > > [1] https://issues.redhat.com/browse/DBZ-2952 >
