Hi everyone, In order to not postpone FLIP-95 further, I include the "removing Factory#factoryVersion" in this FLIP. I updated the "Proposed Changes" section to reflect the changes.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory Please let me know if you have other questions. Best, Jark On Wed, 1 Apr 2020 at 00:56, Jark Wu <imj...@gmail.com> wrote: > Hi, Dawid > > Regarding to `connector.property-version`, > I totally agree with you we should implicitly add a "property-version=1" > (without 'connector.' prefix) property for future evolving. So I updated > FLIP for this. > However, I still doubt to use property version to distinguish old/new > factory. Because it will break existing DDLs, unless users manually set > `connector.property-version=1` to their existing DDLs. So I still prefer > to use `connector` vs `connector.type` to distinguish old/new factory. > > ---------------------------- > > Hi Timo, > > +1 to zookeeper.znode-parent > > > sink.bulk-flush -> sink.buffer-flush? > I would like to keep using bulk-flush, because "bulk" is a well-known > Elasticsearch API and terminology [1]. > I think we don't need to align all the terminologies across Flink > connectors. Following the external system's > terminology will be more easy-to-understand for connector users. > > > username -> secrect.username? > That's a good idea to hide secret values in logs. However, is there a > better way to do that? For example, add a secretOptions() method to Factory? > IMO, a `secrect.` prefix is too weak and limit the design of a property > key. For example, we want to support authentication for elasticserch [2], > a possible property keys will be `authentication.enabled=true`, > `authentication.username=jark`, `authentication.password=123456`. > > [1]: > https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html > [2]: https://issues.apache.org/jira/browse/FLINK-16788 > ---------------------------- > > Hi Zhenghua, > > > does this affect descriptors and related validators? > No. As described in the compatiblity section, all the old properties will > be routed to the old factories. > So all the current descriptors (will be translated to old property keys) > are still compatible. > But, we should have a plan to translate current descritors into new > property keys. > However, that is not in the scope of this FLIP and could be done in a > separate simple JIRA issue. > > Best, > Jark > > On Tue, 31 Mar 2020 at 16:08, Zhenghua Gao <doc...@gmail.com> wrote: > >> Hi Jark, >> >> Thanks for the proposal. I'm +1 since it's more simple and clear for sql >> users. >> I have a question about this: does this affect descriptors and related >> validators? >> >> *Best Regards,* >> *Zhenghua Gao* >> >> >> On Mon, Mar 30, 2020 at 2:02 PM Jark Wu <imj...@gmail.com> wrote: >> >> > Hi everyone, >> > >> > I want to start a discussion about further improve and simplify our >> current >> > connector porperty keys, aka WITH options. Currently, we have a >> > 'connector.' prefix for many properties, but they are verbose, and we >> see a >> > big inconsistency between the properties when designing FLIP-107. >> > >> > So we propose to remove all the 'connector.' prefix and rename >> > 'connector.type' to 'connector', 'format.type' to 'format'. So a new >> Kafka >> > DDL may look like this: >> > >> > CREATE TABLE kafka_table ( >> > ... >> > ) WITH ( >> > 'connector' = 'kafka', >> > 'version' = '0.10', >> > 'topic' = 'test-topic', >> > 'startup-mode' = 'earliest-offset', >> > 'properties.bootstrap.servers' = 'localhost:9092', >> > 'properties.group.id' = 'testGroup', >> > 'format' = 'json', >> > 'format.fail-on-missing-field' = 'false' >> > ); >> > >> > The new connector property key set will come together with new Factory >> > inferface which is proposed in FLIP-95. Old properties are still >> compatible >> > with their existing implementation. New properties are only available in >> > new DynamicTableFactory implementations. >> > >> > You can access the detailed FLIP here: >> > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory >> > >> > Best, >> > Jark >> > >> >
