Randall, This KIP looks great to me. As for _updating_ topic configs - It’s a nice to have but certainly something that I could live without in order to get this KIP implemented. (Its not something I would use in my current setup but I can see some cases where it could be part of the workflow for mirrored topics). If it were to be included, I’d be happier to see it hidden behind a config flag - (if topic already exists, can be an option to WARN/FAIL or change the topic, where the default would be warn?)
Cheers, Rhys > On Aug 21, 2018, at 10:58 PM, Randall Hauch <rha...@gmail.com> wrote: > > Okay, after much delay let's try this again for AK 2.1. Has anyone found > any concerns? Stephane suggested that we allow updating topic > configurations (everything but partition count). I'm unconvinced that it's > worth the additional complexity in the implementation and the documentation > to explain the behavior. Changing several of the topic-specific > configurations have significant impact on broker behavior / functionality, > so IMO we need to proceed more cautiously. > > Stephane, do you have a particular use case in mind for updating topic > configurations on an existing topic? > > Randall > > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> wrote: > >> The KIP deadline for 1.1 has already passed, but I'd like to restart this >> discussion so that we make the next release. I've not yet addressed the >> previous comment about *existing* topics, but I'll try to do that over the >> next few weeks. Any other comments/suggestions/questions? >> >> Best regards, >> >> Randall >> >> On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> wrote: >> >>> Oops. Yes, I meant “replication factor”. >>> >>>> On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>> Randall: >>>> bq. AdminClient currently allows changing the replication factory. >>>> >>>> By 'replication factory' did you mean 'replication factor' ? >>>> >>>> Cheers >>>> >>>>> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> >>> wrote: >>>>> >>>>> Currently the KIP's scope is only topics that don't yet exist, and we >>> have >>>>> to cognizant of race conditions between tasks with the same connector. >>> I >>>>> think it is worthwhile to consider whether the KIP's scope should >>> expand to >>>>> also address *existing* partitions, though it may not be appropriate to >>>>> have as much control when changing the topic settings for an existing >>>>> topic. For example, changing the number of partitions (which the KIP >>>>> considers a "topic-specific setting" even though technically it is not) >>>>> shouldn't be done blindly due to the partitioning impacts, and IIRC you >>>>> can't reduce them (which we could verify before applying). Also, I >>> don't >>>>> think the AdminClient currently allows changing the replication >>> factory. I >>>>> think changing the topic configs is less problematic both from what >>> makes >>>>> sense for connectors to verify/change and from what the AdminClient >>>>> supports. >>>>> >>>>> Even if we decide that it's not appropriate to change the settings on >>> an >>>>> existing topic, I do think it's advantageous to at least notify the >>>>> connector (or task) prior to the first record sent to a given topic so >>> that >>>>> the connector can fail or issue a warning if it doesn't meet its >>>>> requirements. >>>>> >>>>> Best regards, >>>>> >>>>> Randall >>>>> >>>>> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < >>>>> steph...@simplemachines.com.au> wrote: >>>>> >>>>>> Hi Randall, >>>>>> >>>>>> Thanks for the KIP. I like it >>>>>> What happens when the target topic is already created but the configs >>> do >>>>>> not match? >>>>>> i.e. wrong RF, num partitions, or missing / additional configs? Will >>> you >>>>>> attempt to apply the necessary changes or throw an error? >>>>>> >>>>>> Thanks! >>>>>> Stephane >>>>>> >>>>>> >>>>>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <mathieu.fenn...@replicon.com >>>> >>>>>> wrote: >>>>>> >>>>>> Ah, yes, I see you a highlighted part that should've made this >>> clear >>>>>> to me the first read. :-) Much clearer now! >>>>>> >>>>>> By the way, enjoyed your Debezium talk in NYC. >>>>>> >>>>>> Looking forward to this Kafka Connect change; it will allow me to >>>>>> remove a post-deployment tool that I hacked together for the >>> purpose >>>>>> of ensuring auto-created topics have the right config. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> >>>>>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> >>>>>> wrote: >>>>>>> Thanks for the quick feedback, Mathieu. Yes, the first >>>>> configuration >>>>>> rule >>>>>>> whose regex matches will be applied, and no other rules will be >>>>>> used. I've >>>>>>> updated the KIP to try to make this more clear, but let me know if >>>>>> it's >>>>>>> still not clear. >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Randall >>>>>>> >>>>>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < >>>>>>> mathieu.fenn...@replicon.com> wrote: >>>>>>> >>>>>>>> Hi Randall, >>>>>>>> >>>>>>>> Awesome, very much looking forward to this. >>>>>>>> >>>>>>>> It isn't 100% clear from the KIP how multiple config-based rules >>>>>> would >>>>>>>> be applied; it looks like the first configuration rule whose regex >>>>>>>> matches the topic name will be used, and no other rules will be >>>>>>>> applied. Is that correct? (I wasn't sure if it might cascade >>>>>>>> together multiple matching rules...) >>>>>>>> >>>>>>>> Looks great, >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> >>>>>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> >>>>>> wrote: >>>>>>>>> Hi, all. >>>>>>>>> >>>>>>>>> We recently added the ability for Kafka Connect to create >>>>>> *internal* >>>>>>>> topics >>>>>>>>> using the new AdminClient, but it still would be great if Kafka >>>>>> Connect >>>>>>>>> could do this for new topics that result from source connector >>>>>> records. >>>>>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect >>>>>> should >>>>>>>> allow >>>>>>>>> source connectors to set topic-specific settings for new >>>>> topics". >>>>>>>>> >>>>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >>>>>>>> set+topic-specific+settings+for+new+topics >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >>>>>>>> set+topic-specific+settings+for+new+topics>* >>>>>>>>> >>>>>>>>> Please take a look and provide feedback. Thanks! >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Randall >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> >> >>
