Yes, personally I would also go with the external system's format for the exact reason you mentioned.
Best, Dawid On 14/11/2019 10:32, Jark Wu wrote: > Hi Dawid, > > The `connector.properties.zookeeper.connect` sounds good to me. > > Just to clarify, in this way, we can support arbitrary properties and we > can document the important ones. > And regarding to the value format, shall we use the external system's > format or use Flink's format, e.g. the list separator? > I would prefer the external system's format, so that keys and values are > all forwarded directly. > > Best, > Jark > > > On Thu, 14 Nov 2019 at 17:02, Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > >> Hi Jark, >> >> I see the not readable point and that is also my concern about my >> suggestion with all the properties under a single key as the list might get >> quite long. >> >> What do you think about modifying your suggestion a bit and at least add >> the `properties` prefix? So that we can distinguish the Kafka defined >> properties and the Flink's ones? >> >> What do you think about properties like: >> >> connector.properties.zookeeper.connect: .... >> >> connector.properties.bootstrap-servers: .... >> >> >> I think that's also what Konstantin was suggesting. I might be wrong on >> that though :) >> >> Best, >> >> Dawid >> >> >> On 14/11/2019 04:52, Jark Wu wrote: >> >> Hi Dawid, Konstantin, >> >> This is an interesting topic. I will list the prons and cons of these 2 >> options. >> >> 1) under a single key: `connector.properties` >> Cons: >> - Users can pass in arbitrary properties defined in kafka docs >> Prons: >> - The property value is collapsed together and will be long and not >> readable. >> - Some values may need to be escaped because of the separator. >> >> 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap. >> servers` >> Cons: >> - Better validation and helpful exception message if the value is wrong. >> - Better documentation about which properties are required and what it >> means. >> - Easy to write. >> Prons: >> - If users want to pass in other kafka properties, need to support them and >> wait until the next major release. >> >> From my point of view, we can support both #1 and #2, but #2 will override >> the values defined in #1. >> So that, users can define the important properties in falttented keys, and >> can define unsupported properties in `connector.properties`. >> >> What do you think? >> >> Best, >> Jark >> >> >> >> On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf <konstan...@ververica.com> >> <konstan...@ververica.com> >> wrote: >> >> >> Hi Dawid, Hi Jark, >> >> in my experience it is very important to be able to forward arbitrary >> properties to the underlying KafkaClient. This seems to be possible in both >> cases. I am leaning towards Jark's original suggestion. Flink's >> documentation would only need to state that it forwards everything under >> "connector.properties" to the KafkaClient. >> >> If the external system has a different formatting scheme, we could still >> document this on a per-connector basis (use Flink's formatting scheme and >> document how it is translated to the external system's native properties) >> or if possible use the external system's scheme under >> "connector.properties". I don't really know the limitations on our side >> w.r.t this. >> >> Cheers, >> >> Konstantin >> >> >> >> On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <dwysakow...@apache.org> >> <dwysakow...@apache.org> >> wrote: >> >> >> Hi Jark, >> >> Majority of the changes make sense to me. I think they will be helpful >> >> for >> >> the users. I have a slight concern about the >> >> kafka's connector.properties property :) . I wonder if we should have >> them under a single key: >> >> connector.properties: >> >> `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` >> >> As far as I understand it, these are mostly the properties that are >> forwarded straight to the underlying KafkaClient. Such properties are >> mostly defined and documented by the kafka itself rather than flink. They >> might also follow a different formatting scheme than we have for our >> properties. Moreover how do we decide which properties are put into the >> Properties object and which are not? I would be happy to hear what others >> think about that part, as I am not convinced myself about that part. >> >> Best, >> >> Dawid >> On 13/11/2019 13:22, Jark Wu wrote: >> >> Hi everyone, >> >> Connector properties is a very basic component which is used to >> >> construct a >> >> connector table via YAML, descriptor API, or DDL. It is also the >> serialization representation when persisting into catalog. However, we >> >> have >> >> encountered some problems when using connector properties, especially in >> DDL. This FLIP is aiming to solve following two issues: >> >> - FLINK-14645: Data types defined in DDL loses precision and nullability >> when converting to properties. >> - FLINK-14649: Some properties structure is hard to define in DDL, e.g. >> List and Map structure. >> >> This FLIP proposes new properties to represent data types in schema and >> >> to >> >> flatten properties in existing connectors. >> >> FLIP-86: >> >> https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# >> >> Thanks for any feedback! >> >> Best, >> Jark >> >> >> >> >> -- >> >> Konstantin Knauf | Solutions Architect >> >> +49 160 91394525 >> >> >> Follow us @VervericaData Ververica <https://www.ververica.com/> >> <https://www.ververica.com/> >> >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> <https://flink-forward.org/> >> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Tony) Cheng >> >> >>
signature.asc
Description: OpenPGP digital signature
