Hi Jark, Thanks for the proposal, I'm +1 to the general idea. However I have a question about "version", in the old design, the version seems to be aimed for tracking property version, with different version, we could evolve these step by step without breaking backward compatibility. But in this design, version is representing external system's version, like "0.11" for kafka, "6" or "7" for ES. I'm not sure if this is necessary, what's the benefit of using two keys instead of one, like "kafka-0.11" or "ES6" directly? And how about the old capability which could let us evolving connector properties?
Best, Kurt On Mon, Mar 30, 2020 at 2:36 PM LakeShen <shenleifight...@gmail.com> wrote: > Hi Jark, > > I am really looking forward to this feature. I think this feature > could simplify flink sql code,and at the same time , > it could make the developer more easlier to config the flink sql WITH > options. > > Now when I am using flink sql to write flink task , sometimes I think the > WITH options is too long for user. > For example,I config the kafka source connector parameter，for consumer > group and brokers parameter: > > 'connector.properties.0.key' = 'group.id' > > , 'connector.properties.0.value' = 'xxx' > > , 'connector.properties.1.key' = 'bootstrap.servers' > > , 'connector.properties.1.value' = 'xxxxx' > > > > I can understand this config , but for the flink fresh man,maybe it > is confused for him. > In my thought, I am really looking forward to this feature,thank you to > propose this feature. > > Best wishes, > LakeShen > > > Jark Wu <imj...@gmail.com> 于2020年3月30日周一 下午2:02写道： > > > Hi everyone, > > > > I want to start a discussion about further improve and simplify our > current > > connector porperty keys, aka WITH options. Currently, we have a > > 'connector.' prefix for many properties, but they are verbose, and we > see a > > big inconsistency between the properties when designing FLIP-107. > > > > So we propose to remove all the 'connector.' prefix and rename > > 'connector.type' to 'connector', 'format.type' to 'format'. So a new > Kafka > > DDL may look like this: > > > > CREATE TABLE kafka_table ( > > ... > > ) WITH ( > > 'connector' = 'kafka', > > 'version' = '0.10', > > 'topic' = 'test-topic', > > 'startup-mode' = 'earliest-offset', > > 'properties.bootstrap.servers' = 'localhost:9092', > > 'properties.group.id' = 'testGroup', > > 'format' = 'json', > > 'format.fail-on-missing-field' = 'false' > > ); > > > > The new connector property key set will come together with new Factory > > inferface which is proposed in FLIP-95. Old properties are still > compatible > > with their existing implementation. New properties are only available in > > new DynamicTableFactory implementations. > > > > You can access the detailed FLIP here: > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > > > > Best, > > Jark > > >
