Ryanne, I agree that worker property is named poorly. Maybe
"disable.topic.creation.for.connectors"? I've updated the KIP to use this
and to change the default to be enabled, but suggestions on better names
are welcome.
Ewen, you mentioned this property on the vote thread:
> My only final comment would be that the topic.creation.enable setting
could
> potentially be left out. We're still backwards compatible with what we
> promised before (or at least compatibility is debatable). You could
enforce
> not being able to create topics with ACLs anyway, so avoiding the extra
> config might be worth the not-quite-perfect compatibility story.
I agree that it's possible to leave this setting out, since by default
existing source connector configs that don't use these new
`topic.creation.*` settings would default to the existing behavior. I added
this because of Gwen's earlier comment "but also give ops teams the option
to disable it and avoid surprises." Ewen is right that we could rely upon
ACLs as a means to "disable" or prevent the use of this feature, but that
seems fairly complicated and non-intuitive. Even though I'd love to avoid a
new worker config, it does seem much easier to be able to disable the
feature with a worker config, but default to enable the feature so most
users don't have to worry about it. Thoughts? I've added using ACLs instead
of the new worker config property as a rejected alternative, but happy to
discuss further.
Best regards,
Randall
On Tue, Jan 22, 2019 at 11:04 AM Ryanne Dolan <ryannedo...@gmail.com> wrote:
> Thanks Randall, makes sense to me.
>
> I suggest we change the topic.creation.enabled property name though. Sounds
> like it means topics are not created at all when disabled.
>
> Ryanne
>
> On Sat, Jan 19, 2019, 1:13 PM Randall Hauch <rha...@gmail.com wrote:
>
> > Hi,
> >
> > Thanks again for all of the feedback. Based upon this feedback, I do
> agree
> > that we should indeed just solve the simple problem for the vast majority
> > of use cases that only require a few simple properties. I believe I was
> the
> > only person advocating for a more general and flexible solution, but that
> > flexibility is simply not worth the much greater complexity.
> >
> > So, I've dramatically simplified the KIP as follows:
> >
> > 1. The KIP no longer proposes changes to the Java API. Source
> connector
> > implementations would only be able to use this API if and only if they
> > are
> > willing to constrain the connectors to be deployed to AK 2.2 (or
> > whichever
> > is the first version that includes this feature) or later. I would be
> > surprised if very many source connector developers would take
> advantage
> > of
> > this feature. Plus, removing this from the proposal eliminated a lot
> of
> > complexity.
> > 2. A new `topic.creation.enable` Connect worker configuration property
> > allows a Connect cluster operator to control whether or not connectors
> > can
> > use this feature. It defaults to not enabling the feature, which makes
> > this
> > fully backward compatible.
> > 3. The new properties specified in the source connector configurations
> > are no longer rule based and are straightforward: the default
> > replication
> > factor, number of partitions, and other topic-specific settings for
> any
> > and
> > all new topics created by Connect. This also eliminates a lot of
> > complexity
> > of the design and should make using this feature much easier.
> >
> > I would love for this to get into AK 2.2, which means voting needs to
> start
> > in the next few days. Short notice, but hopefully this smaller and
> simpler
> > proposal is something we can all agree to. Let's start simple and learn
> > from our users whether or not they need more flexibility and control.
> >
> > Please respond with your thoughts. Thanks!
> >
> > Best regards,
> >
> > Randall
> >
> > On Tue, Nov 27, 2018 at 7:36 PM Ryanne Dolan <ryannedo...@gmail.com>
> > wrote:
> >
> > > Randall, have you considered something like:
> > >
> > > - introduce TopicCreationPolicy interface, with methods like
> > > partitionsForTopic(topic).
> > > - provide a DefaultTopicCreationPolicy implementation that implements
> the
> > > current behavior.
> > > - provide a SimpleTopicCreationPolicy that honors
> > topic.creation.partitions
> > > property, etc.
> > > - perhaps also provide a RegexTopicCreationPolicy.
> > > - users can provide their own TopicCreationPolicy implementations when
> > > necessary.
> > > - support topic.creation.policy.class property in worker configs, with
> > > default = org.apache.kafka.connect.DefaultTopicCreationPolicy.
> > >
> > > This way, the only new configuration property is
> > > topic.creation.policy.class. The default behavior doesn't change from
> > what
> > > we have today. If a user wants to change from the default, they can
> > opt-in
> > > to one of the other policies or implement their own.
> > >
> > > Ryanne
> > >
> > > On Tue, Nov 27, 2018 at 6:31 PM Randall Hauch <rha...@gmail.com>
> wrote:
> > >
> > > > Thanks for the feedback. Some thoughts inline.
> > > >
> > > > On Tue, Nov 27, 2018 at 5:47 PM Ewen Cheslack-Postava <
> > e...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > re: AdminClient vs this proposal, one consideration is that
> > AdminClient
> > > > > exposes a lot more surface area and probably a bunch of stuff we
> > > actually
> > > > > don't want Connectors to be able to do, such as deleting topics.
> You
> > > can
> > > > > always lock down by ACLs, but what the framework enables directly
> vs
> > > > > requiring the user to opt in via connector-specific config is an
> > > > important
> > > > > distinction.
> > > > >
> > > > > I'm not a fan of how complex the config is (same deal with
> > > > > transformations), and agree with Ryanne that any case requiring
> > > multiple
> > > > > rules is probably an outlier. A cleaner option for the common case
> > > might
> > > > be
> > > > > worth it. One option that's still aligned with the current state of
> > the
> > > > KIP
> > > > > would be to change the default for topic.creation to a fixed
> default
> > > > value
> > > > > (e.g. 'default'), if that's the case turn the
> > > > topic.creation.default.regex
> > > > > default to .*, and then 99% use case would just be specifying the #
> > of
> > > > > partitions with a single config and relying on cluster defaults for
> > the
> > > > > rest. (I would suggest the same thing for transformations if we
> > added a
> > > > > simple scripting transformation such that most use cases wouldn't
> > need
> > > to
> > > > > compose multiple transformations.)
> > > > >
> > > >
> > > > I agree that any case requiring multiple rules is probably an
> outlier,
> > > and
> > > > I've been trying to think about how to start simple with a single
> case
> > > but
> > > > leave room if we really do need multiple rules in the future. I like
> > > Ewen's
> > > > suggestion a lot. IIUC, it would change the following common case:
> > > >
> > > > topic.creation=default
> > > > topic.creation.default.regex=.*
> > > > topic.creation.default.partitions=5
> > > >
> > > > into the following:
> > > >
> > > > topic.creation.default.partitions=5
> > > >
> > > > where the replication defaults to 3 and all others default to the
> > > brokers'
> > > > default topic settings. This is significantly simpler, yet it still
> > > allows
> > > > us to handle multiple rules if they're needed.
> > > >
> > > > Another common case is to use compacted topics, so that might
> required
> > > > adding:
> > > >
> > > > topic.creation.default.cleanup.policy=compact
> > > >
> > > > Also, I like the idea of defaulting the regex to '.*', but I wonder
> if
> > > it'd
> > > > be easier to explain if that default value applied to the *last* rule
> > in
> > > > the list of rules, rather than to apply only to the rule named
> > "default"
> > > > when that's the only named rule. WDYT?
> > > >
> > > >
> > > > >
> > > > > Along related lines, is there actually a need for TopicSettings
> > class?
> > > We
> > > > > already have NewTopic in the AdminClient APIs. Does that not
> suffice?
> > > > >
> > > >
> > > > There are three reasons I added TopicSettings and didn't simply use
> > > > NewTopic. First, NewTopic is immutable, which makes it impractical
> for
> > > > Connect to pass to a connector and to allow the connector to change
> it.
> > > > Second, TopicSettings is essentially a builder with easy to use and
> > > > chainable methods, whereas NewTopic relies upon Map<String, String>,
> > > String
> > > > constants (in another class), and essentially untyped values. Third,
> > > > NewTopic is not an interface, and I think it's better to expose an
> > > > interface in the connector API when we don't want/expect connectors
> to
> > > > instantiate the instance and to instead only use what we provide
> them.
> > > >
> > > > Now, another option is to move TopicSettings into the
> > > > `org.apache.kafka.clients.admin` package and turn it into
> > > `NewTopicBuilder`
> > > > instead. Then it'd be useful outside of Connect, but is it strange
> that
> > > > it's not in the Connect API packages?
> > > >
> > > > Randall
> > > >
> > > >
> > > > >
> > > > > -Ewen
> > > > >
> > > > > On Mon, Sep 24, 2018 at 11:56 AM Andrew Otto <o...@wikimedia.org>
> > > wrote:
> > > > >
> > > > > > FWIW, I’d find this feature useful.
> > > > > >
> > > > > > On Mon, Sep 24, 2018 at 2:42 PM Randall Hauch <rha...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Ryanne,
> > > > > > >
> > > > > > > If your connector is already using the AdminClient, then you as
> > the
> > > > > > > developer have a choice of switching to the new Connect-based
> > > > > > functionality
> > > > > > > or keeping the existing use of the AdminClient. If the
> connector
> > > uses
> > > > > > both
> > > > > > > mechanisms (which I wouldn't recommend, simply because of the
> > > > > complexity
> > > > > > of
> > > > > > > it for a user), then the topic will be created by the first
> > > mechanism
> > > > > to
> > > > > > > actually attempt and successfully create the topic(s) in the
> > Kafka
> > > > > > cluster
> > > > > > > that the Connect worker uses. As mentioned in the KIP, "This
> > > feature
> > > > > ...
> > > > > > > does not change the topic-specific settings on any existing
> > > topics."
> > > > > IOW,
> > > > > > > if the topic already exists, it can't be created again and
> > > therefore
> > > > > the
> > > > > > > `topic.creation.*` properties will not apply for that existing
> > > topic.
> > > > > > >
> > > > > > > > Do these settings apply to internal topics created by the
> > > framework
> > > > > on
> > > > > > > > bahalf of a connector, e.g. via KafkaConfigBackingStore?
> > > > > > >
> > > > > > > No, they don't, and I'm happy to add a clarification to the KIP
> > if
> > > > you
> > > > > > feel
> > > > > > > it is necessary.
> > > > > > >
> > > > > > > > I'd have the same questions if e.g. transformations could be
> > > > ignored
> > > > > or
> > > > > > > > overridden by connectors or if there were multiple places to
> > > > specify
> > > > > > what
> > > > > > > > serde to use.
> > > > > > >
> > > > > > > There are multiple places that converters can be defined: the
> > > worker
> > > > > > config
> > > > > > > defines the key and value converters that will be used for all
> > > > > > connectors,
> > > > > > > except when a connector defines its own key and value
> converters.
> > > > > > >
> > > > > > > > I don't see how controlling topic creation based on topic
> name
> > is
> > > > > > > something
> > > > > > > > we should support across all connectors, as if it is some
> > > > established
> > > > > > > > pattern or universally useful.
> > > > > > >
> > > > > > > Topics are identified by name, and when you create a topic with
> > > > > specific
> > > > > > > settings or change a topic's settings you identify the topic by
> > > name.
> > > > > The
> > > > > > > fact that this KIP uses regular expressions to match topic
> names
> > > > > doesn't
> > > > > > > seem surprising, since we use regexes elsewhere.
> > > > > > >
> > > > > > > Best regards
> > > > > > >
> > > > > > > On Mon, Sep 24, 2018 at 1:24 PM Ryanne Dolan <
> > > ryannedo...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Randall,
> > > > > > > >
> > > > > > > > Say I've got a connector that needs to control topic
> creation.
> > > > What I
> > > > > > > need
> > > > > > > > is an AdminClient s.t. my connector can do what it knows it
> > needs
> > > > to
> > > > > > do.
> > > > > > > > This KIP doesn't address the issues that have been brought up
> > wrt
> > > > > > > > configuration, principals, ACL etc, since I'll still need to
> > > > > construct
> > > > > > my
> > > > > > > > own AdminClient.
> > > > > > > >
> > > > > > > > Should such a connector ignore your proposed configuration
> > > > settings?
> > > > > > > Should
> > > > > > > > it use it's own principal and it's own configuration
> > properties?
> > > > How
> > > > > > does
> > > > > > > > my AdminClient's settings interact with your proposed
> settings
> > > and
> > > > > the
> > > > > > > > existing cluster settings?
> > > > > > > >
> > > > > > > > What happens when a user specifies topic creation settings
> in a
> > > > > > connector
> > > > > > > > config, but the connector then applies it's own topic
> creation
> > > > logic?
> > > > > > Are
> > > > > > > > the configurations silently ignored? If not, how can a
> > connector
> > > > > honor
> > > > > > > your
> > > > > > > > proposed settings?
> > > > > > > >
> > > > > > > > Do these settings apply to internal topics created by the
> > > framework
> > > > > on
> > > > > > > > bahalf of a connector, e.g. via KafkaConfigBackingStore?
> > > > > > > >
> > > > > > > > When do the cluster settings get applied? Only after 3 layers
> > of
> > > > > > > > fall-through?
> > > > > > > >
> > > > > > > > I'd have the same questions if e.g. transformations could be
> > > > ignored
> > > > > or
> > > > > > > > overridden by connectors or if there were multiple places to
> > > > specify
> > > > > > what
> > > > > > > > serde to use.
> > > > > > > >
> > > > > > > > I don't see how controlling topic creation based on topic
> name
> > is
> > > > > > > something
> > > > > > > > we should support across all connectors, as if it is some
> > > > established
> > > > > > > > pattern or universally useful.
> > > > > > > >
> > > > > > > > Ryanne
> > > > > > > >
> > > > > > > > On Mon, Sep 24, 2018, 10:14 AM Randall Hauch <
> rha...@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Ryanne. My apologies for not responding earlier, as I
> was
> > > on
> > > > a
> > > > > > long
> > > > > > > > > holiday.
> > > > > > > > >
> > > > > > > > > Thanks for your feedback and questions about this KIP.
> You've
> > > > > raised
> > > > > > > > > several points in the discussion so far, so let me try to
> > > address
> > > > > > most
> > > > > > > of
> > > > > > > > > them.
> > > > > > > > >
> > > > > > > > > IIUC, one of your major concerns is that this KIP
> introduces
> > a
> > > > new
> > > > > > way
> > > > > > > to
> > > > > > > > > define configurations for topics. That is true, and the
> whole
> > > > > reason
> > > > > > is
> > > > > > > > to
> > > > > > > > > simply the user experience for people using source
> > connectors.
> > > > You
> > > > > > > still
> > > > > > > > > have the freedom to manually pre-create topics before
> > running a
> > > > > > > > connector,
> > > > > > > > > or to rely upon the broker automatically creating topics
> for
> > > the
> > > > > > > > connectors
> > > > > > > > > when those topics don't yet exist -- in both cases, don't
> > > include
> > > > > > > > anything
> > > > > > > > > about topic creation in your connector configurations. In
> > fact,
> > > > > when
> > > > > > > you
> > > > > > > > do
> > > > > > > > > this, Connect uses the current behavior by assuming the
> > topics
> > > > > exist
> > > > > > or
> > > > > > > > > will be autocreated with the proper configurations.
> > > > > > > > >
> > > > > > > > > But for many environments, this existing approach is not
> > > enough.
> > > > > > First,
> > > > > > > > if
> > > > > > > > > you're relying upon the broker to autocreate topics, then
> the
> > > > > brokers
> > > > > > > > > single set of default topic settings must match the
> > > requirements
> > > > of
> > > > > > > your
> > > > > > > > > new topics. This can be difficult when your running
> multiple
> > > > kinds
> > > > > of
> > > > > > > > > connectors with differing expectations. Consider using a
> CDC
> > > > > > connector
> > > > > > > > that
> > > > > > > > > expects compaction, a high-volume web service connector
> that
> > > > should
> > > > > > not
> > > > > > > > use
> > > > > > > > > compaction but expects deletion after 7 days, and another
> > > > connector
> > > > > > > with
> > > > > > > > > lower volume that uses 30 day retention. Or, consider
> > > connectors
> > > > > that
> > > > > > > are
> > > > > > > > > producing to topics that have very different message
> > > > > characteristics:
> > > > > > > > > different sizes, different throughputs, different
> partitions,
> > > > etc.
> > > > > > The
> > > > > > > > only
> > > > > > > > > way to work around this is to pre-create the topics, but
> this
> > > > adds
> > > > > > more
> > > > > > > > > complexity and room for errors, especially when a single
> > > instance
> > > > > of
> > > > > > > some
> > > > > > > > > source connectors can write to dozens (or even hundreds) of
> > > > topics.
> > > > > > > > >
> > > > > > > > > Second, many operators prefer (or are required) to disable
> > > topic
> > > > > > > > > autocreation, since simple mistakes with command line tools
> > can
> > > > > > result
> > > > > > > in
> > > > > > > > > new topics. In this cases, users have no choice but to
> > manually
> > > > > > > precreate
> > > > > > > > > the topics that complicates the process of running a
> > connector
> > > > and,
> > > > > > as
> > > > > > > > > mentioned above, increases the risk that something goes
> > wrong.
> > > > > > > > >
> > > > > > > > > Third, the reason why this KIP introduces a way for
> connector
> > > > > > > > > implementations to override some topic settings is because
> > some
> > > > > > source
> > > > > > > > > connectors have very specific requirements. When I wrote
> the
> > > > first
> > > > > > > > Debezium
> > > > > > > > > CDC connectors, many first-time users didn't precreate the
> > > topics
> > > > > as
> > > > > > > > > recommended in the documentation, and didn't change their
> > > > brokers'
> > > > > > > > default
> > > > > > > > > topic settings. Only after a few days when they tried
> > > reconsuming
> > > > > the
> > > > > > > > full
> > > > > > > > > streams did they realize that Kafka had deleted messages
> > older
> > > > than
> > > > > > the
> > > > > > > > > default retention period. Debezium expects / requires
> > compacted
> > > > > > topics,
> > > > > > > > so
> > > > > > > > > all kinds of things went wrong. Connect is often one of the
> > > first
> > > > > > ways
> > > > > > > in
> > > > > > > > > which people get introduced to Kafka, and they simply don't
> > yet
> > > > > have
> > > > > > an
> > > > > > > > > understanding of many of the details that you or I don't
> have
> > > to
> > > > > > think
> > > > > > > > > twice about.
> > > > > > > > >
> > > > > > > > > You suggested that maybe Connect should just expose the
> Admin
> > > > API.
> > > > > > > That's
> > > > > > > > > possible, but IMO it's very heavyweight and complex. The
> > whole
> > > > > point
> > > > > > of
> > > > > > > > > Connect's design is to abstract the connector developer
> away
> > > from
> > > > > > most
> > > > > > > of
> > > > > > > > > the details of Kafka -- it doesn't even expose the producer
> > and
> > > > > > > consumer
> > > > > > > > > APIs, which are much simpler. IMO it would be a mistake to
> > > > require
> > > > > > > source
> > > > > > > > > connector developers to deal with the Admin API -- I even
> > have
> > > > > > trouble
> > > > > > > > > writing code that uses it to properly create topics,
> > especially
> > > > > > around
> > > > > > > > > properly handling all of the potential error conditions.
> > > > > > > > >
> > > > > > > > > You also mention that topic settings in a connector
> > > configuration
> > > > > > might
> > > > > > > > not
> > > > > > > > > reflect the actual topic's settings. This is true,
> especially
> > > if
> > > > > the
> > > > > > > > topic
> > > > > > > > > already existed with different settings before the
> connector
> > > was
> > > > > run.
> > > > > > > > > However, this is also very true of the broker's default
> topic
> > > > > > settings,
> > > > > > > > > which very often don't reflect the actual settings for all
> of
> > > the
> > > > > > > topics
> > > > > > > > --
> > > > > > > > > the defaults may have been changed, or topics are created
> > > > manually
> > > > > > with
> > > > > > > > > very different settings. The only way to know the settings
> > of a
> > > > > > > > particular
> > > > > > > > > topic are to get them for that topic.
> > > > > > > > >
> > > > > > > > > The use of naming rules in the topic creation settings is
> > > > > > intentional,
> > > > > > > > and
> > > > > > > > > it allows connector users to define topic settings for
> topics
> > > > based
> > > > > > > upon
> > > > > > > > > the names. In some cases this may require several rules to
> > > handle
> > > > > the
> > > > > > > > > different topics, but most of the time a single rule may be
> > all
> > > > > > that's
> > > > > > > > > required. I also don't agree that users will start naming
> > > topics
> > > > to
> > > > > > > > > simplify their rules -- many source connectors that write
> to
> > > more
> > > > > > than
> > > > > > > > one
> > > > > > > > > topic often don't allow the user to specify the full name
> of
> > > the
> > > > > > topics
> > > > > > > > > anyway, and when they do they often only write to one
> topic.
> > > > > > > > >
> > > > > > > > > I still think that the proposed KIP provides a simple way
> for
> > > > most
> > > > > > > source
> > > > > > > > > connector users to control (via configuration) the settings
> > of
> > > > the
> > > > > > > topics
> > > > > > > > > that will be created by Connect for that connector, which
> > works
> > > > > with
> > > > > > > all
> > > > > > > > > existing source connectors out of the box and does not add
> > > > > additional
> > > > > > > > > complexities for source connector developers.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > Randall
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Sep 12, 2018 at 12:22 PM Ryanne Dolan <
> > > > > ryannedo...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > Rather than go though the connect framework, connectors
> > > > should
> > > > > > just
> > > > > > > > > > create their own AdminClient instance and create their
> own
> > > > > topics?
> > > > > > > > > >
> > > > > > > > > > Rather, can the framework be improved to expose an
> > > AdminClient
> > > > > > ready
> > > > > > > to
> > > > > > > > > > use? Then connectors can use this instance without
> needing
> > > > > separate
> > > > > > > > > > identities/principals and associated configuration
> (which I
> > > > > totally
> > > > > > > > > > understand would be a nightmare). I believe that covers
> all
> > > the
> > > > > > > > > use-cases?
> > > > > > > > > >
> > > > > > > > > > I just don't see how the "terrible config situation" is
> > > > remedied
> > > > > by
> > > > > > > > > adding
> > > > > > > > > > even more configuration.
> > > > > > > > > >
> > > > > > > > > > Also, I'm not sure I can conceive of a use-case in which
> a
> > > > single
> > > > > > > > > connector
> > > > > > > > > > would need multiple default topic settings *based on the
> > > topic
> > > > > > name*.
> > > > > > > > Can
> > > > > > > > > > you give a real-world example? Is this something you've
> > > > > > encountered,
> > > > > > > or
> > > > > > > > > are
> > > > > > > > > > you just trying for a flexible design?
> > > > > > > > > >
> > > > > > > > > > Ryanne
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 11, 2018 at 9:57 PM Gwen Shapira <
> > > > g...@confluent.io>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Ryanne,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback!
> > > > > > > > > > >
> > > > > > > > > > > Can you explain a bit more what you mean by "if we
> allow
> > > > > > connectors
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > this
> > > > > > > > > > > decision, they should have full control of the
> process."?
> > > > > > > > > > >
> > > > > > > > > > > I assume you mean, something like:
> > > > > > > > > > > Rather than go though the connect framework, connectors
> > > > should
> > > > > > just
> > > > > > > > > > create
> > > > > > > > > > > their own AdminClient instance and create their own
> > topics?
> > > > > > > > > > >
> > > > > > > > > > > The problem with this approach is that connectors
> > currently
> > > > > don't
> > > > > > > > have
> > > > > > > > > > > their own identity (in the authentication/authorization
> > > > sense).
> > > > > > All
> > > > > > > > > > > connectors share the framework identity, if the users
> > need
> > > to
> > > > > > start
> > > > > > > > > > > configuring security for both the framework and connect
> > > > itself,
> > > > > > it
> > > > > > > > gets
> > > > > > > > > > > messy rather quickly.
> > > > > > > > > > > We actually already do the thing I'm imagining you
> > > suggested
> > > > in
> > > > > > > some
> > > > > > > > > > > connectors right now (create AdminClient and configure
> > > > topics),
> > > > > > and
> > > > > > > > we
> > > > > > > > > > hope
> > > > > > > > > > > to use the new framework capability to clean-up the
> > > > > configuration
> > > > > > > > mess
> > > > > > > > > > this
> > > > > > > > > > > has caused. I spent 4 days trying to figure out what a
> > > > specific
> > > > > > > > > connector
> > > > > > > > > > > doesn't work, just to find out that you need to give it
> > its
> > > > own
> > > > > > > > > security
> > > > > > > > > > > config because it has an AdminClient so the
> configuration
> > > on
> > > > > the
> > > > > > > > > > framework
> > > > > > > > > > > isn't enough.
> > > > > > > > > > >
> > > > > > > > > > > From my experience with rather large number of
> customers,
> > > > there
> > > > > > are
> > > > > > > > > some
> > > > > > > > > > > companies where the topics are controlled by a central
> > team
> > > > > that
> > > > > > > owns
> > > > > > > > > all
> > > > > > > > > > > the machinery to create and configure topics (sometimes
> > via
> > > > > > gitops,
> > > > > > > > > > > kubernetes custom resources, etc) and they would indeed
> > be
> > > > very
> > > > > > > > > surprised
> > > > > > > > > > > if a connector suddenly had opinions about topics.
> There
> > > are
> > > > > also
> > > > > > > > teams
> > > > > > > > > > > where the application developers feel like they know
> > their
> > > > data
> > > > > > and
> > > > > > > > > > > use-case the best and they are in-charge of making all
> > > > > > topic-level
> > > > > > > > > > > decisions, usually automated by the app itself. Admin
> > > client
> > > > > was
> > > > > > > > > created
> > > > > > > > > > > for those teams and I think they'll appreciate having
> > this
> > > > > > > capability
> > > > > > > > > in
> > > > > > > > > > > connect too. Funny thing is, customers who work with
> one
> > > > model
> > > > > > > > usually
> > > > > > > > > > > can't believe the other model even exists.
> > > > > > > > > > >
> > > > > > > > > > > I'd love to propose a compromise and suggest that we'll
> > > allow
> > > > > > this
> > > > > > > > > > > functionality in Connect but also give ops teams the
> > option
> > > > to
> > > > > > > > disable
> > > > > > > > > it
> > > > > > > > > > > and avoid surprises. But I'm afraid this wont work -
> too
> > > > often
> > > > > > the
> > > > > > > > > > defaults
> > > > > > > > > > > are just terrible for specific connectors (CDC
> connectors
> > > > > > sometimes
> > > > > > > > > need
> > > > > > > > > > a
> > > > > > > > > > > single partition to maintain consistency) and if there
> > is a
> > > > > > chance
> > > > > > > > the
> > > > > > > > > > > connector preference won't be used, connectors will
> have
> > to
> > > > > force
> > > > > > > it
> > > > > > > > > via
> > > > > > > > > > > admin client which brings us back to the terrible
> config
> > > > > > situation
> > > > > > > we
> > > > > > > > > > > currently have with Admin client.
> > > > > > > > > > >
> > > > > > > > > > > Gwen
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 11, 2018 at 7:23 PM, Ryanne Dolan <
> > > > > > > ryannedo...@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Randall,
> > > > > > > > > > > >
> > > > > > > > > > > > I have some concerns with this proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > Firstly, I don't believe it is the job of a connector
> > to
> > > > > > > configure
> > > > > > > > > > > topics,
> > > > > > > > > > > > generally, nor for topic-specific settings to hang
> out
> > in
> > > > > > > connector
> > > > > > > > > > > > configurations. Automatic creation of topics with
> > default
> > > > > > > settings
> > > > > > > > is
> > > > > > > > > > an
> > > > > > > > > > > > established pattern elsewhere, and I don't think
> > > connectors
> > > > > > need
> > > > > > > to
> > > > > > > > > > > diverge
> > > > > > > > > > > > from this.
> > > > > > > > > > > >
> > > > > > > > > > > > I agree there are cases where the default settings
> > don't
> > > > make
> > > > > > > sense
> > > > > > > > > and
> > > > > > > > > > > > it'd be nice to override them. But if we allow
> > connectors
> > > > to
> > > > > > make
> > > > > > > > > this
> > > > > > > > > > > > decision, they should have full control of the
> process.
> > > > > > > > > > > >
> > > > > > > > > > > > Some concerns:
> > > > > > > > > > > > - I'd expect the cluster's default settings to apply
> to
> > > > newly
> > > > > > > > created
> > > > > > > > > > > > topics, regardless of who created them. I wouldn't
> > expect
> > > > > > source
> > > > > > > > > > > connectors
> > > > > > > > > > > > to be a special case. In particular, I'd be surprised
> > if
> > > > > Kafka
> > > > > > > > > Connect
> > > > > > > > > > > were
> > > > > > > > > > > > to ignore my cluster's default settings and apply its
> > own
> > > > > > > defaults.
> > > > > > > > > > > > - It will be possible to add a specific topic to this
> > > > > > > > configuration,
> > > > > > > > > in
> > > > > > > > > > > > which case any reader would expect the topic to have
> > the
> > > > > > > specified
> > > > > > > > > > > > settings. But this will not generally be true. Thus,
> > the
> > > > > > > > > configuration
> > > > > > > > > > > will
> > > > > > > > > > > > end up lying and misleading, and there won't be any
> > > > > indication
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > configuration is lying.
> > > > > > > > > > > > - Connectors that want to control settings will end
> up
> > > > naming
> > > > > > > > topics
> > > > > > > > > > > > accordingly. For example, a connector that wants to
> > > control
> > > > > the
> > > > > > > > > number
> > > > > > > > > > of
> > > > > > > > > > > > partitions would need a bunch of creation rules for 1
> > > > > > partition,
> > > > > > > 2
> > > > > > > > > > > > partitions and so on. This is a bad pattern to
> > > establish. A
> > > > > > > better
> > > > > > > > > > > pattern
> > > > > > > > > > > > is to let the connector control the number of
> > partitions
> > > > > > directly
> > > > > > > > > when
> > > > > > > > > > > that
> > > > > > > > > > > > feature is required.
> > > > > > > > > > > > - The proposal introduces 2 new interfaces to control
> > > topic
> > > > > > > > creation
> > > > > > > > > > > > (configuration rules and TopicSettings), where there
> is
> > > > > > already a
> > > > > > > > > > > perfectly
> > > > > > > > > > > > good one (AdminClient).
> > > > > > > > > > > >
> > > > > > > > > > > > Ryanne
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 4, 2018 at 5:08 PM Randall Hauch <
> > > > > rha...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Okay, I think I cleaned up the formatting issues in
> > the
> > > > KIP
> > > > > > > wiki
> > > > > > > > > > page.
> > > > > > > > > > > > And
> > > > > > > > > > > > > while implementing I realized that it'd be helpful
> to
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > > limit
> > > > > > > > > > > > via
> > > > > > > > > > > > > the connector configuration and the rules which
> > topics
> > > > are
> > > > > > > > > created. I
> > > > > > > > > > > > added
> > > > > > > > > > > > > the `topic.creation.${ruleName}.policy` behavior,
> > with
> > > > > > possible
> > > > > > > > > > values
> > > > > > > > > > > > of
> > > > > > > > > > > > > "create" (the default), "autocreate" (to specify
> that
> > > > > Connect
> > > > > > > > > should
> > > > > > > > > > > let
> > > > > > > > > > > > > the broker autocreate any matching topics), and
> > "fail"
> > > > (to
> > > > > > > > specify
> > > > > > > > > > that
> > > > > > > > > > > > > Connect should not allow the creation of topics
> whose
> > > > names
> > > > > > > match
> > > > > > > > > the
> > > > > > > > > > > > > rule's regular expression).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Let me know what you think. I'd like to start
> voting
> > > > soon,
> > > > > > but
> > > > > > > > > > because
> > > > > > > > > > > I
> > > > > > > > > > > > > made the above change I'll wait a few days.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Randall
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Aug 29, 2018 at 9:41 PM Randall Hauch <
> > > > > > > rha...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks, Magesh.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > All, I've made a few very minor changes to some
> > > > JavaDocs
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > > signatures of the name-value pair methods in
> > > > > TopicSettings
> > > > > > > > > > > interface. I
> > > > > > > > > > > > > > also described as a fifth rejected alternative
> why
> > > this
> > > > > KIP
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > modify
> > > > > > > > > > > > > > any topic settings for existing topics. All of
> > these
> > > > are
> > > > > > > pretty
> > > > > > > > > > > minor,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > I'm happy to hear about issues or suggestions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since the above changes were very minor, I'll
> kick
> > > off
> > > > a
> > > > > > vote
> > > > > > > > to
> > > > > > > > > > > accept
> > > > > > > > > > > > > > this KIP unless I hear something in the next day
> or
> > > > two.
> > > > > > Note
> > > > > > > > > that
> > > > > > > > > > > this
> > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > has been around in virtually the exact form for
> > over
> > > a
> > > > > > year.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Randall
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Aug 29, 2018 at 9:17 PM Magesh
> Nandakumar <
> > > > > > > > > > > > mage...@confluent.io>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Randall,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I originally thought that this proposal was a
> > config
> > > > > only
> > > > > > > > topic
> > > > > > > > > > > > settings
> > > > > > > > > > > > > >> and hence made the comment about configs being
> > pass
> > > > > > > through. I
> > > > > > > > > > just
> > > > > > > > > > > > > >> realized that the connectors can also override
> and
> > > > > provide
> > > > > > > the
> > > > > > > > > > > > > >> TopicSettings. With that in mind, I think the
> > > proposal
> > > > > > looks
> > > > > > > > > > great.
> > > > > > > > > > > > > >> Looking forward to the feature.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >> Magesh
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Tue, Aug 28, 2018 at 8:53 PM Magesh
> Nandakumar
> > <
> > > > > > > > > > > > mage...@confluent.io
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > I was wondering if it would be much simpler to
> > > just
> > > > > do a
> > > > > > > > > > > > pass-through
> > > > > > > > > > > > > so
> > > > > > > > > > > > > >> > that we can support any topic setting added in
> > > Kafka
> > > > > > > without
> > > > > > > > > any
> > > > > > > > > > > > code
> > > > > > > > > > > > > >> > changes in connect. Since these are for topics
> > > that
> > > > > will
> > > > > > > > have
> > > > > > > > > > the
> > > > > > > > > > > > > actual
> > > > > > > > > > > > > >> > data stream, users might possibly need more
> > > > > flexibility
> > > > > > in
> > > > > > > > > terms
> > > > > > > > > > > of
> > > > > > > > > > > > > how
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > topics get created.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Thanks
> > > > > > > > > > > > > >> > Magesh
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch
> <
> > > > > > > > > rha...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >> Do you think we should support name-value
> > pairs,
> > > > too?
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh
> > > Nandakumar <
> > > > > > > > > > > > > >> mage...@confluent.io>
> > > > > > > > > > > > > >> >> wrote:
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> > Randall,
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > Thanks a lot for the KIP. I think this
> would
> > > be a
> > > > > > great
> > > > > > > > > > > addition
> > > > > > > > > > > > > for
> > > > > > > > > > > > > >> >> many
> > > > > > > > > > > > > >> >> > source connectors.
> > > > > > > > > > > > > >> >> > One clarification I had was regarding the
> > topic
> > > > > > > settings
> > > > > > > > > that
> > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > >> >> > configured. Is it limited to the setting
> > > exposed
> > > > in
> > > > > > the
> > > > > > > > > > > > > TopicSettings
> > > > > > > > > > > > > >> >> > interface?
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > Thanks
> > > > > > > > > > > > > >> >> > Magesh
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > On Tue, Aug 21, 2018 at 7:59 PM Randall
> > Hauch <
> > > > > > > > > > > rha...@gmail.com>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > > Okay, after much delay let's try this
> again
> > > for
> > > > > AK
> > > > > > > 2.1.
> > > > > > > > > Has
> > > > > > > > > > > > > anyone
> > > > > > > > > > > > > >> >> found
> > > > > > > > > > > > > >> >> > > any concerns? Stephane suggested that we
> > > allow
> > > > > > > updating
> > > > > > > > > > topic
> > > > > > > > > > > > > >> >> > > configurations (everything but partition
> > > > count).
> > > > > > I'm
> > > > > > > > > > > > unconvinced
> > > > > > > > > > > > > >> that
> > > > > > > > > > > > > >> >> > it's
> > > > > > > > > > > > > >> >> > > worth the additional complexity in the
> > > > > > implementation
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > >> >> > documentation
> > > > > > > > > > > > > >> >> > > to explain the behavior. Changing several
> > of
> > > > the
> > > > > > > > > > > topic-specific
> > > > > > > > > > > > > >> >> > > configurations have significant impact on
> > > > broker
> > > > > > > > > behavior /
> > > > > > > > > > > > > >> >> > functionality,
> > > > > > > > > > > > > >> >> > > so IMO we need to proceed more
> cautiously.
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> > > Stephane, do you have a particular use
> case
> > > in
> > > > > mind
> > > > > > > for
> > > > > > > > > > > > updating
> > > > > > > > > > > > > >> topic
> > > > > > > > > > > > > >> >> > > configurations on an existing topic?
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> > > Randall
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall
> > > Hauch <
> > > > > > > > > > > > rha...@gmail.com>
> > > > > > > > > > > > > >> >> wrote:
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> > > > The KIP deadline for 1.1 has already
> > > passed,
> > > > > but
> > > > > > > I'd
> > > > > > > > > like
> > > > > > > > > > > to
> > > > > > > > > > > > > >> restart
> > > > > > > > > > > > > >> >> > this
> > > > > > > > > > > > > >> >> > > > discussion so that we make the next
> > > release.
> > > > > I've
> > > > > > > not
> > > > > > > > > yet
> > > > > > > > > > > > > >> addressed
> > > > > > > > > > > > > >> >> the
> > > > > > > > > > > > > >> >> > > > previous comment about *existing*
> topics,
> > > but
> > > > > > I'll
> > > > > > > > try
> > > > > > > > > to
> > > > > > > > > > > do
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> >> over
> > > > > > > > > > > > > >> >> > > the
> > > > > > > > > > > > > >> >> > > > next few weeks. Any other
> > > > > > > > > comments/suggestions/questions?
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > > > Best regards,
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > > > Randall
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > > > On Thu, Oct 5, 2017 at 12:13 AM,
> Randall
> > > > Hauch
> > > > > <
> > > > > > > > > > > > > rha...@gmail.com
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > > >> Oops. Yes, I meant “replication
> factor”.
> > > > > > > > > > > > > >> >> > > >>
> > > > > > > > > > > > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <
> > > > > > > > > > yuzhih...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >> >> > > >> >
> > > > > > > > > > > > > >> >> > > >> > Randall:
> > > > > > > > > > > > > >> >> > > >> > bq. AdminClient currently allows
> > > changing
> > > > > the
> > > > > > > > > > > replication
> > > > > > > > > > > > > >> >> factory.
> > > > > > > > > > > > > >> >> > > >> >
> > > > > > > > > > > > > >> >> > > >> > By 'replication factory' did you
> mean
> > > > > > > 'replication
> > > > > > > > > > > > factor' ?
> > > > > > > > > > > > > >> >> > > >> >
> > > > > > > > > > > > > >> >> > > >> > Cheers
> > > > > > > > > > > > > >> >> > > >> >
> > > > > > > > > > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM,
> > Randall
> > > > > > Hauch <
> > > > > > > > > > > > > >> rha...@gmail.com
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > > >> wrote:
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >> Currently the KIP's scope is only
> > > topics
> > > > > that
> > > > > > > > don't
> > > > > > > > > > yet
> > > > > > > > > > > > > >> exist,
> > > > > > > > > > > > > >> >> and
> > > > > > > > > > > > > >> >> > we
> > > > > > > > > > > > > >> >> > > >> have
> > > > > > > > > > > > > >> >> > > >> >> to cognizant of race conditions
> > between
> > > > > tasks
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > >> >> > > connector.
> > > > > > > > > > > > > >> >> > > >> I
> > > > > > > > > > > > > >> >> > > >> >> think it is worthwhile to consider
> > > > whether
> > > > > > the
> > > > > > > > > KIP's
> > > > > > > > > > > > scope
> > > > > > > > > > > > > >> >> should
> > > > > > > > > > > > > >> >> > > >> expand to
> > > > > > > > > > > > > >> >> > > >> >> also address *existing* partitions,
> > > > though
> > > > > it
> > > > > > > may
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > >> >> > appropriate
> > > > > > > > > > > > > >> >> > > to
> > > > > > > > > > > > > >> >> > > >> >> have as much control when changing
> > the
> > > > > topic
> > > > > > > > > settings
> > > > > > > > > > > for
> > > > > > > > > > > > > an
> > > > > > > > > > > > > >> >> > existing
> > > > > > > > > > > > > >> >> > > >> >> topic. For example, changing the
> > number
> > > > of
> > > > > > > > > partitions
> > > > > > > > > > > > > (which
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> >> > KIP
> > > > > > > > > > > > > >> >> > > >> >> considers a "topic-specific
> setting"
> > > even
> > > > > > > though
> > > > > > > > > > > > > technically
> > > > > > > > > > > > > >> it
> > > > > > > > > > > > > >> >> is
> > > > > > > > > > > > > >> >> > > not)
> > > > > > > > > > > > > >> >> > > >> >> shouldn't be done blindly due to
> the
> > > > > > > partitioning
> > > > > > > > > > > > impacts,
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >> >> IIRC
> > > > > > > > > > > > > >> >> > > you
> > > > > > > > > > > > > >> >> > > >> >> can't reduce them (which we could
> > > verify
> > > > > > before
> > > > > > > > > > > > applying).
> > > > > > > > > > > > > >> >> Also, I
> > > > > > > > > > > > > >> >> > > >> don't
> > > > > > > > > > > > > >> >> > > >> >> think the AdminClient currently
> > allows
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > > > > >> replication
> > > > > > > > > > > > > >> >> > > >> factory. I
> > > > > > > > > > > > > >> >> > > >> >> think changing the topic configs is
> > > less
> > > > > > > > > problematic
> > > > > > > > > > > both
> > > > > > > > > > > > > >> from
> > > > > > > > > > > > > >> >> what
> > > > > > > > > > > > > >> >> > > >> makes
> > > > > > > > > > > > > >> >> > > >> >> sense for connectors to
> verify/change
> > > and
> > > > > > from
> > > > > > > > what
> > > > > > > > > > the
> > > > > > > > > > > > > >> >> AdminClient
> > > > > > > > > > > > > >> >> > > >> >> supports.
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >> Even if we decide that it's not
> > > > appropriate
> > > > > > to
> > > > > > > > > change
> > > > > > > > > > > the
> > > > > > > > > > > > > >> >> settings
> > > > > > > > > > > > > >> >> > on
> > > > > > > > > > > > > >> >> > > >> an
> > > > > > > > > > > > > >> >> > > >> >> existing topic, I do think it's
> > > > > advantageous
> > > > > > to
> > > > > > > > at
> > > > > > > > > > > least
> > > > > > > > > > > > > >> notify
> > > > > > > > > > > > > >> >> the
> > > > > > > > > > > > > >> >> > > >> >> connector (or task) prior to the
> > first
> > > > > record
> > > > > > > > sent
> > > > > > > > > > to a
> > > > > > > > > > > > > given
> > > > > > > > > > > > > >> >> topic
> > > > > > > > > > > > > >> >> > > so
> > > > > > > > > > > > > >> >> > > >> that
> > > > > > > > > > > > > >> >> > > >> >> the connector can fail or issue a
> > > warning
> > > > > if
> > > > > > it
> > > > > > > > > > doesn't
> > > > > > > > > > > > > meet
> > > > > > > > > > > > > >> its
> > > > > > > > > > > > > >> >> > > >> >> requirements.
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >> Best regards,
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >> Randall
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM,
> > > Stephane
> > > > > > > Maarek
> > > > > > > > <
> > > > > > > > > > > > > >> >> > > >> >> steph...@simplemachines.com.au>
> > wrote:
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >> >>> Hi Randall,
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> Thanks for the KIP. I like it
> > > > > > > > > > > > > >> >> > > >> >>> What happens when the target topic
> > is
> > > > > > already
> > > > > > > > > > created
> > > > > > > > > > > > but
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> >> > > configs
> > > > > > > > > > > > > >> >> > > >> do
> > > > > > > > > > > > > >> >> > > >> >>> not match?
> > > > > > > > > > > > > >> >> > > >> >>> i.e. wrong RF, num partitions, or
> > > > missing
> > > > > /
> > > > > > > > > > additional
> > > > > > > > > > > > > >> configs?
> > > > > > > > > > > > > >> >> > Will
> > > > > > > > > > > > > >> >> > > >> you
> > > > > > > > > > > > > >> >> > > >> >>> attempt to apply the necessary
> > changes
> > > > or
> > > > > > > throw
> > > > > > > > an
> > > > > > > > > > > > error?
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> Thanks!
> > > > > > > > > > > > > >> >> > > >> >>> Stephane
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu
> > > Fenniak"
> > > > <
> > > > > > > > > > > > > >> >> > > mathieu.fenn...@replicon.com
> > > > > > > > > > > > > >> >> > > >> >
> > > > > > > > > > > > > >> >> > > >> >>> wrote:
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> Ah, yes, I see you a
> highlighted
> > > part
> > > > > > that
> > > > > > > > > > > should've
> > > > > > > > > > > > > made
> > > > > > > > > > > > > >> >> this
> > > > > > > > > > > > > >> >> > > >> clear
> > > > > > > > > > > > > >> >> > > >> >>> to me the first read. :-) Much
> > > > clearer
> > > > > > > now!
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> By the way, enjoyed your
> Debezium
> > > > talk
> > > > > in
> > > > > > > > NYC.
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> Looking forward to this Kafka
> > > Connect
> > > > > > > change;
> > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > > >> allow
> > > > > > > > > > > > > >> >> me
> > > > > > > > > > > > > >> >> > to
> > > > > > > > > > > > > >> >> > > >> >>> remove a post-deployment tool
> > that
> > > I
> > > > > > hacked
> > > > > > > > > > > together
> > > > > > > > > > > > > for
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> >> > > >> purpose
> > > > > > > > > > > > > >> >> > > >> >>> of ensuring auto-created topics
> > > have
> > > > > the
> > > > > > > > right
> > > > > > > > > > > > config.
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> Mathieu
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>> On Tue, May 23, 2017 at 11:38
> AM,
> > > > > Randall
> > > > > > > > > Hauch <
> > > > > > > > > > > > > >> >> > > rha...@gmail.com>
> > > > > > > > > > > > > >> >> > > >> >>> wrote:
> > > > > > > > > > > > > >> >> > > >> >>>> Thanks for the quick feedback,
> > > Mathieu.
> > > > > > Yes,
> > > > > > > > the
> > > > > > > > > > > first
> > > > > > > > > > > > > >> >> > > >> >> configuration
> > > > > > > > > > > > > >> >> > > >> >>> rule
> > > > > > > > > > > > > >> >> > > >> >>>> whose regex matches will be
> > applied,
> > > > and
> > > > > no
> > > > > > > > other
> > > > > > > > > > > rules
> > > > > > > > > > > > > >> will
> > > > > > > > > > > > > >> >> be
> > > > > > > > > > > > > >> >> > > >> >>> used. I've
> > > > > > > > > > > > > >> >> > > >> >>>> updated the KIP to try to make
> this
> > > > more
> > > > > > > clear,
> > > > > > > > > but
> > > > > > > > > > > let
> > > > > > > > > > > > > me
> > > > > > > > > > > > > >> >> know
> > > > > > > > > > > > > >> >> > if
> > > > > > > > > > > > > >> >> > > >> >>> it's
> > > > > > > > > > > > > >> >> > > >> >>>> still not clear.
> > > > > > > > > > > > > >> >> > > >> >>>>
> > > > > > > > > > > > > >> >> > > >> >>>> Best regards,
> > > > > > > > > > > > > >> >> > > >> >>>>
> > > > > > > > > > > > > >> >> > > >> >>>> Randall
> > > > > > > > > > > > > >> >> > > >> >>>>
> > > > > > > > > > > > > >> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM,
> > > > Mathieu
> > > > > > > > > Fenniak <
> > > > > > > > > > > > > >> >> > > >> >>>> mathieu.fenn...@replicon.com>
> > wrote:
> > > > > > > > > > > > > >> >> > > >> >>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> Hi Randall,
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> Awesome, very much looking
> forward
> > > to
> > > > > > this.
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> It isn't 100% clear from the KIP
> > how
> > > > > > > multiple
> > > > > > > > > > > > > config-based
> > > > > > > > > > > > > >> >> rules
> > > > > > > > > > > > > >> >> > > >> >>> would
> > > > > > > > > > > > > >> >> > > >> >>>>> be applied; it looks like the
> > first
> > > > > > > > > configuration
> > > > > > > > > > > rule
> > > > > > > > > > > > > >> whose
> > > > > > > > > > > > > >> >> > regex
> > > > > > > > > > > > > >> >> > > >> >>>>> matches the topic name will be
> > used,
> > > > and
> > > > > > no
> > > > > > > > > other
> > > > > > > > > > > > rules
> > > > > > > > > > > > > >> will
> > > > > > > > > > > > > >> >> be
> > > > > > > > > > > > > >> >> > > >> >>>>> applied. Is that correct? (I
> > > wasn't
> > > > > sure
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > > might
> > > > > > > > > > > > > >> >> cascade
> > > > > > > > > > > > > >> >> > > >> >>>>> together multiple matching
> > rules...)
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> Looks great,
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> Mathieu
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM,
> > > > Randall
> > > > > > > > Hauch <
> > > > > > > > > > > > > >> >> > rha...@gmail.com>
> > > > > > > > > > > > > >> >> > > >> >>> wrote:
> > > > > > > > > > > > > >> >> > > >> >>>>>> Hi, all.
> > > > > > > > > > > > > >> >> > > >> >>>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>> We recently added the ability
> for
> > > > Kafka
> > > > > > > > Connect
> > > > > > > > > > to
> > > > > > > > > > > > > create
> > > > > > > > > > > > > >> >> > > >> >>> *internal*
> > > > > > > > > > > > > >> >> > > >> >>>>> topics
> > > > > > > > > > > > > >> >> > > >> >>>>>> using the new AdminClient, but
> it
> > > > still
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > great
> > > > > > > > > > > > > if
> > > > > > > > > > > > > >> >> Kafka
> > > > > > > > > > > > > >> >> > > >> >>> Connect
> > > > > > > > > > > > > >> >> > > >> >>>>>> could do this for new topics
> that
> > > > > result
> > > > > > > from
> > > > > > > > > > > source
> > > > > > > > > > > > > >> >> connector
> > > > > > > > > > > > > >> >> > > >> >>> records.
> > > > > > > > > > > > > >> >> > > >> >>>>>> I've outlined an approach to do
> > > this
> > > > in
> > > > > > > > > "KIP-158
> > > > > > > > > > > > Kafka
> > > > > > > > > > > > > >> >> Connect
> > > > > > > > > > > > > >> >> > > >> >>> should
> > > > > > > > > > > > > >> >> > > >> >>>>> allow
> > > > > > > > > > > > > >> >> > > >> >>>>>> source connectors to set
> > > > topic-specific
> > > > > > > > > settings
> > > > > > > > > > > for
> > > > > > > > > > > > > new
> > > > > > > > > > > > > >> >> > > >> >> topics".
> > > > > > > > > > > > > >> >> > > >> >>>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>> *
> > > > > > > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > > > > > > > > > > > allow+source+connectors+to+
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > set+topic-specific+settings+for+new+topics
> > > > > > > > > > > > > >> >> > > >> >>>>>> <
> > > > > > > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > > > > > > > > > > > allow+source+connectors+to+
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > set+topic-specific+settings+for+new+topics>*
> > > > > > > > > > > > > >> >> > > >> >>>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>> Please take a look and provide
> > > > > feedback.
> > > > > > > > > Thanks!
> > > > > > > > > > > > > >> >> > > >> >>>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>> Best regards,
> > > > > > > > > > > > > >> >> > > >> >>>>>>
> > > > > > > > > > > > > >> >> > > >> >>>>>> Randall
> > > > > > > > > > > > > >> >> > > >> >>>>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>>
> > > > > > > > > > > > > >> >> > > >> >>
> > > > > > > > > > > > > >> >> > > >>
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > > >
> > > > > > > > > > > > > >> >> > >
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Gwen Shapira*
> > > > > > > > > > > Product Manager | Confluent
> > > > > > > > > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > > > > > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc>
> |
> > > blog
> > > > > > > > > > > <http://www.confluent.io/blog>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
