The KIP deadline for 1.1 has already passed, but I'd like to restart this discussion so that we make the next release. I've not yet addressed the previous comment about *existing* topics, but I'll try to do that over the next few weeks. Any other comments/suggestions/questions?
Best regards, Randall On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> wrote: > Oops. Yes, I meant “replication factor”. > > > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > Randall: > > bq. AdminClient currently allows changing the replication factory. > > > > By 'replication factory' did you mean 'replication factor' ? > > > > Cheers > > > >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> wrote: > >> > >> Currently the KIP's scope is only topics that don't yet exist, and we > have > >> to cognizant of race conditions between tasks with the same connector. I > >> think it is worthwhile to consider whether the KIP's scope should > expand to > >> also address *existing* partitions, though it may not be appropriate to > >> have as much control when changing the topic settings for an existing > >> topic. For example, changing the number of partitions (which the KIP > >> considers a "topic-specific setting" even though technically it is not) > >> shouldn't be done blindly due to the partitioning impacts, and IIRC you > >> can't reduce them (which we could verify before applying). Also, I don't > >> think the AdminClient currently allows changing the replication > factory. I > >> think changing the topic configs is less problematic both from what > makes > >> sense for connectors to verify/change and from what the AdminClient > >> supports. > >> > >> Even if we decide that it's not appropriate to change the settings on an > >> existing topic, I do think it's advantageous to at least notify the > >> connector (or task) prior to the first record sent to a given topic so > that > >> the connector can fail or issue a warning if it doesn't meet its > >> requirements. > >> > >> Best regards, > >> > >> Randall > >> > >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > >> steph...@simplemachines.com.au> wrote: > >> > >>> Hi Randall, > >>> > >>> Thanks for the KIP. I like it > >>> What happens when the target topic is already created but the configs > do > >>> not match? > >>> i.e. wrong RF, num partitions, or missing / additional configs? Will > you > >>> attempt to apply the necessary changes or throw an error? > >>> > >>> Thanks! > >>> Stephane > >>> > >>> > >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <mathieu.fenn...@replicon.com> > >>> wrote: > >>> > >>> Ah, yes, I see you a highlighted part that should've made this clear > >>> to me the first read. :-) Much clearer now! > >>> > >>> By the way, enjoyed your Debezium talk in NYC. > >>> > >>> Looking forward to this Kafka Connect change; it will allow me to > >>> remove a post-deployment tool that I hacked together for the purpose > >>> of ensuring auto-created topics have the right config. > >>> > >>> Mathieu > >>> > >>> > >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> > >>> wrote: > >>>> Thanks for the quick feedback, Mathieu. Yes, the first > >> configuration > >>> rule > >>>> whose regex matches will be applied, and no other rules will be > >>> used. I've > >>>> updated the KIP to try to make this more clear, but let me know if > >>> it's > >>>> still not clear. > >>>> > >>>> Best regards, > >>>> > >>>> Randall > >>>> > >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > >>>> mathieu.fenn...@replicon.com> wrote: > >>>> > >>>>> Hi Randall, > >>>>> > >>>>> Awesome, very much looking forward to this. > >>>>> > >>>>> It isn't 100% clear from the KIP how multiple config-based rules > >>> would > >>>>> be applied; it looks like the first configuration rule whose regex > >>>>> matches the topic name will be used, and no other rules will be > >>>>> applied. Is that correct? (I wasn't sure if it might cascade > >>>>> together multiple matching rules...) > >>>>> > >>>>> Looks great, > >>>>> > >>>>> Mathieu > >>>>> > >>>>> > >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> > >>> wrote: > >>>>>> Hi, all. > >>>>>> > >>>>>> We recently added the ability for Kafka Connect to create > >>> *internal* > >>>>> topics > >>>>>> using the new AdminClient, but it still would be great if Kafka > >>> Connect > >>>>>> could do this for new topics that result from source connector > >>> records. > >>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect > >>> should > >>>>> allow > >>>>>> source connectors to set topic-specific settings for new > >> topics". > >>>>>> > >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >>>>> set+topic-specific+settings+for+new+topics > >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >>>>> set+topic-specific+settings+for+new+topics>* > >>>>>> > >>>>>> Please take a look and provide feedback. Thanks! > >>>>>> > >>>>>> Best regards, > >>>>>> > >>>>>> Randall > >>>>> > >>> > >>> > >>> > >>> > >> >
