Hi Dawid, Konstantin, This is an interesting topic. I will list the prons and cons of these 2 options.
1) under a single key: `connector.properties` Cons: - Users can pass in arbitrary properties defined in kafka docs Prons: - The property value is collapsed together and will be long and not readable. - Some values may need to be escaped because of the separator. 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap. servers` Cons: - Better validation and helpful exception message if the value is wrong. - Better documentation about which properties are required and what it means. - Easy to write. Prons: - If users want to pass in other kafka properties, need to support them and wait until the next major release. >From my point of view, we can support both #1 and #2, but #2 will override the values defined in #1. So that, users can define the important properties in falttented keys, and can define unsupported properties in `connector.properties`. What do you think? Best, Jark On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf <konstan...@ververica.com> wrote: > Hi Dawid, Hi Jark, > > in my experience it is very important to be able to forward arbitrary > properties to the underlying KafkaClient. This seems to be possible in both > cases. I am leaning towards Jark's original suggestion. Flink's > documentation would only need to state that it forwards everything under > "connector.properties" to the KafkaClient. > > If the external system has a different formatting scheme, we could still > document this on a per-connector basis (use Flink's formatting scheme and > document how it is translated to the external system's native properties) > or if possible use the external system's scheme under > "connector.properties". I don't really know the limitations on our side > w.r.t this. > > Cheers, > > Konstantin > > > > On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > > > Hi Jark, > > > > Majority of the changes make sense to me. I think they will be helpful > for > > the users. I have a slight concern about the > > > > kafka's connector.properties property :) . I wonder if we should have > > them under a single key: > > > > connector.properties: > `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` > > > > As far as I understand it, these are mostly the properties that are > > forwarded straight to the underlying KafkaClient. Such properties are > > mostly defined and documented by the kafka itself rather than flink. They > > might also follow a different formatting scheme than we have for our > > properties. Moreover how do we decide which properties are put into the > > Properties object and which are not? I would be happy to hear what others > > think about that part, as I am not convinced myself about that part. > > > > Best, > > > > Dawid > > On 13/11/2019 13:22, Jark Wu wrote: > > > > Hi everyone, > > > > Connector properties is a very basic component which is used to > construct a > > connector table via YAML, descriptor API, or DDL. It is also the > > serialization representation when persisting into catalog. However, we > have > > encountered some problems when using connector properties, especially in > > DDL. This FLIP is aiming to solve following two issues: > > > > - FLINK-14645: Data types defined in DDL loses precision and nullability > > when converting to properties. > > - FLINK-14649: Some properties structure is hard to define in DDL, e.g. > > List and Map structure. > > > > This FLIP proposes new properties to represent data types in schema and > to > > flatten properties in existing connectors. > > > > FLIP-86: > https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# > > > > Thanks for any feedback! > > > > Best, > > Jark > > > > > > > > -- > > Konstantin Knauf | Solutions Architect > > +49 160 91394525 > > > Follow us @VervericaData Ververica <https://www.ververica.com/> > > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Tony) Cheng >
