Hi Dawid,

The `connector.properties.zookeeper.connect` sounds good to me.

Just to clarify, in this way, we can support arbitrary properties and we
can document the important ones.
And regarding to the value format, shall we use the external system's
format or use Flink's format, e.g. the list separator?
I would prefer the external system's format, so that keys and values are
all forwarded directly.

Best,
Jark


On Thu, 14 Nov 2019 at 17:02, Dawid Wysakowicz <dwysakow...@apache.org>
wrote:

> Hi Jark,
>
> I see the not readable point and that is also my concern about my
> suggestion with all the properties under a single key as the list might get
> quite long.
>
> What do you think about modifying your suggestion a bit and at least add
> the `properties` prefix? So that we can distinguish the Kafka defined
> properties and the Flink's ones?
>
> What do you think about properties like:
>
> connector.properties.zookeeper.connect: ....
>
> connector.properties.bootstrap-servers: ....
>
>
> I think that's also what Konstantin was suggesting. I might be wrong on
> that though :)
>
> Best,
>
> Dawid
>
>
> On 14/11/2019 04:52, Jark Wu wrote:
>
> Hi Dawid, Konstantin,
>
> This is an interesting topic. I will list the prons and cons of these 2
> options.
>
> 1) under a single key: `connector.properties`
> Cons:
>  - Users can pass in arbitrary properties defined in kafka docs
> Prons:
>  - The property value is collapsed together and will be long and not
> readable.
>  - Some values may need to be escaped because of the separator.
>
> 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap.
> servers`
> Cons:
> - Better validation and helpful exception message if the value is wrong.
> - Better documentation about which properties are required and what it
> means.
> - Easy to write.
> Prons:
> - If users want to pass in other kafka properties, need to support them and
> wait until the next major release.
>
> From my point of view, we can support both #1 and #2, but #2 will override
> the values defined in #1.
> So that, users can define the important properties in falttented keys, and
> can define unsupported properties in `connector.properties`.
>
> What do you think?
>
> Best,
> Jark
>
>
>
> On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf <konstan...@ververica.com> 
> <konstan...@ververica.com>
> wrote:
>
>
> Hi Dawid, Hi Jark,
>
> in my experience it is very important to be able to forward arbitrary
> properties to the underlying KafkaClient. This seems to be possible in both
> cases. I am leaning towards Jark's original suggestion. Flink's
> documentation would only need to state that it forwards everything under
> "connector.properties" to the KafkaClient.
>
> If the external system has a different formatting scheme, we could still
> document this on a per-connector basis (use Flink's formatting scheme and
> document how it is translated to the external system's native properties)
> or if possible use the external system's scheme under
> "connector.properties". I don't really know the limitations on our side
> w.r.t this.
>
> Cheers,
>
> Konstantin
>
>
>
> On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <dwysakow...@apache.org> 
> <dwysakow...@apache.org>
> wrote:
>
>
> Hi Jark,
>
> Majority of the changes make sense to me. I think they will be helpful
>
> for
>
> the users. I have a slight concern about the
>
> kafka's connector.properties property :) .  I wonder if we should have
> them under a single key:
>
> connector.properties:
>
> `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092`
>
> As far as I understand it, these are mostly the properties that are
> forwarded straight to the underlying KafkaClient. Such properties are
> mostly defined and documented by the kafka itself rather than flink. They
> might also follow a different formatting scheme than we have for our
> properties. Moreover how do we decide which properties are put into the
> Properties object and which are not? I would be happy to hear what others
> think about that part, as I am not convinced myself about that part.
>
> Best,
>
> Dawid
> On 13/11/2019 13:22, Jark Wu wrote:
>
> Hi everyone,
>
> Connector properties is a very basic component which is used to
>
> construct a
>
> connector table via YAML, descriptor API, or DDL. It is also the
> serialization representation when persisting into catalog. However, we
>
> have
>
> encountered some problems when using connector properties, especially in
> DDL. This FLIP is aiming to solve following two issues:
>
> - FLINK-14645: Data types defined in DDL loses precision and nullability
> when converting to properties.
> - FLINK-14649: Some properties structure is hard to define in DDL, e.g.
> List and Map structure.
>
> This FLIP proposes new properties to represent data types in schema and
>
> to
>
> flatten properties in existing connectors.
>
> FLIP-86:
>
> https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit#
>
> Thanks for any feedback!
>
> Best,
> Jark
>
>
>
>
> --
>
> Konstantin Knauf | Solutions Architect
>
> +49 160 91394525
>
>
> Follow us @VervericaData Ververica <https://www.ververica.com/> 
> <https://www.ververica.com/>
>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> <https://flink-forward.org/> 
> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Tony) Cheng
>
>
>

Reply via email to