Hi Jark, I see the not readable point and that is also my concern about my suggestion with all the properties under a single key as the list might get quite long.
What do you think about modifying your suggestion a bit and at least add the `properties` prefix? So that we can distinguish the Kafka defined properties and the Flink's ones? What do you think about properties like: connector.properties.zookeeper.connect: .... connector.properties.bootstrap-servers: .... I think that's also what Konstantin was suggesting. I might be wrong on that though :) Best, Dawid On 14/11/2019 04:52, Jark Wu wrote: > Hi Dawid, Konstantin, > > This is an interesting topic. I will list the prons and cons of these 2 > options. > > 1) under a single key: `connector.properties` > Cons: > - Users can pass in arbitrary properties defined in kafka docs > Prons: > - The property value is collapsed together and will be long and not > readable. > - Some values may need to be escaped because of the separator. > > 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap. > servers` > Cons: > - Better validation and helpful exception message if the value is wrong. > - Better documentation about which properties are required and what it > means. > - Easy to write. > Prons: > - If users want to pass in other kafka properties, need to support them and > wait until the next major release. > > From my point of view, we can support both #1 and #2, but #2 will override > the values defined in #1. > So that, users can define the important properties in falttented keys, and > can define unsupported properties in `connector.properties`. > > What do you think? > > Best, > Jark > > > > On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf <konstan...@ververica.com> > wrote: > >> Hi Dawid, Hi Jark, >> >> in my experience it is very important to be able to forward arbitrary >> properties to the underlying KafkaClient. This seems to be possible in both >> cases. I am leaning towards Jark's original suggestion. Flink's >> documentation would only need to state that it forwards everything under >> "connector.properties" to the KafkaClient. >> >> If the external system has a different formatting scheme, we could still >> document this on a per-connector basis (use Flink's formatting scheme and >> document how it is translated to the external system's native properties) >> or if possible use the external system's scheme under >> "connector.properties". I don't really know the limitations on our side >> w.r.t this. >> >> Cheers, >> >> Konstantin >> >> >> >> On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <dwysakow...@apache.org> >> wrote: >> >>> Hi Jark, >>> >>> Majority of the changes make sense to me. I think they will be helpful >> for >>> the users. I have a slight concern about the >>> >>> kafka's connector.properties property :) . I wonder if we should have >>> them under a single key: >>> >>> connector.properties: >> `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` >>> As far as I understand it, these are mostly the properties that are >>> forwarded straight to the underlying KafkaClient. Such properties are >>> mostly defined and documented by the kafka itself rather than flink. They >>> might also follow a different formatting scheme than we have for our >>> properties. Moreover how do we decide which properties are put into the >>> Properties object and which are not? I would be happy to hear what others >>> think about that part, as I am not convinced myself about that part. >>> >>> Best, >>> >>> Dawid >>> On 13/11/2019 13:22, Jark Wu wrote: >>> >>> Hi everyone, >>> >>> Connector properties is a very basic component which is used to >> construct a >>> connector table via YAML, descriptor API, or DDL. It is also the >>> serialization representation when persisting into catalog. However, we >> have >>> encountered some problems when using connector properties, especially in >>> DDL. This FLIP is aiming to solve following two issues: >>> >>> - FLINK-14645: Data types defined in DDL loses precision and nullability >>> when converting to properties. >>> - FLINK-14649: Some properties structure is hard to define in DDL, e.g. >>> List and Map structure. >>> >>> This FLIP proposes new properties to represent data types in schema and >> to >>> flatten properties in existing connectors. >>> >>> FLIP-86: >> https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# >>> Thanks for any feedback! >>> >>> Best, >>> Jark >>> >>> >>> >> -- >> >> Konstantin Knauf | Solutions Architect >> >> +49 160 91394525 >> >> >> Follow us @VervericaData Ververica <https://www.ververica.com/> >> >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Tony) Cheng >>
signature.asc
Description: OpenPGP digital signature
