Re: Before creating KIP : Kafka Connect / Add a configuration provider class

2017-10-24 Thread Florian Hussonnois
Yes, the provider classes will need to be installed on each worker (same installation mechanism than a connector-plugin). A new provider instance should be create for each connector instance but will be configure at the worker level. A provider class may have two behaviors : - provide a default

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

2017-10-23 Thread Sönke Liebau
I agree, sounds like an intriguing idea. I think we could probably come up with a few common enough implementations that merit including in Kafka. FileConfigProvider for example, so you can distribute common configs throughout your cluster with some orchestration tool and users simply state the ide

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

2017-10-23 Thread Randall Hauch
Very interesting. Would the proposed configuration provider be set at the connector level or the worker level? The latter would obviously be required to handle all/multiple connector configurations. Either way, the provider class(es) would need to be installed on the worker (really, every worker),

Before creating KIP : Kafka Connect / Add a configuration provider class

2017-10-20 Thread Florian Hussonnois
Hi Team Before submitting a new KIP I would like to open the discussion regarding an enhancement of Kafka Connect. Currently, the only way to configure a connector (in distributed mode) is through REST endpoints while creating or updating a connector. It would be nice to have the possibility to