Hi Team Kafka, This may be a slightly premature discussion, but forgetting about upgrades is a newbie mistake I'd like us to avoid :)
So, we have connector-plugins, and users use them to create connector-instances. This requires some configuration - database connection string, HDFS namenode, etc. Now suppose we have a SystemX connector-plugin, and it takes just "connection string" as argument. And people happily use it. Now imagine that the guy who wrong the plugin wants to release SystemX-plugin-2.0. Ideally, we'd want users to be able to drop 2.0 jar instead of 2.0 and restart their connector-instances and keep on running with the existing configuration. But what if SystemX-plugin-2.0 made changes to the configuration? What if it now has a new mandatory parameter? Or if the connection string format changed? I'd like to give connector developers a way to upgrade existing configuration when they release a new version. My proposal: 1. Connector API now includes 2 new methods - int getVersion() and configuration upgrade(configuration) 2. When the framework persists configuration for the connector (I'm talking mostly about cluster mode where we want to keep the configuration in Kafka), it also persists the version #. 3. When starting a connector-instance, if the version from getVersion() doesn't match the version in the persisted configs, the framework will call upgrade() with existing configs so the connector can upgrade them and return the new configs which will then be persisted with the new version. 4. If the upgrade fails, the connector-instance will not run. Does that make sense? Gwen
