Thanks Matthias. I was also thinking about whether in the future we'd want to enable optimizations at different levels that may or may not impact compatibility. That's why I asked if we have thought about "allowing part of the optimizations in the future".
With that in mind, I'd change my preference and take string typed config. Even if we ended up with no finer grained optimizations in the future we can still have the string typed parameter with only two allowed values, like what we did for EOS. But I think in 2.0 allowing any not-null string values as enabled is still a bit odd, so how about we make two string values, like `none` (default value) and `all`? Guozhang On Tue, May 8, 2018 at 3:35 PM, Matthias J. Sax <matth...@confluent.io> wrote: > One thought I want to bring up about switching optimization on/off: > > While for the initial release, a boolean flag seems to be sufficient, I > could imagine that we apply different and potentially > upgrade-incompatible optimizations in future releases. Thus, to me it > would make sense to use a String type, to indicate what optimizations > are possible based on the release. For example, in next release we > accept `null` for disabled and "2.0". If there are any incompatible > changes, people can stay with "2.0" optimizations level when upgrading > to "2.1" while new applications can use "2.1" optimization level. Old > applications would need to do an offline upgrade to get "2.1" > optimizations. > > I agree with Bill, that switching individual optimizations on/off is too > fine grained and hard to maintain. However, for compatibility, it might > make sense, to have certain "levels of optimizations" (based on the > release) that allow users to stay with on an older level for upgrade > purpose. Using the release numbers to encode those "levels" is easy to > understand for users and should minimize the mental effort to get the > config right. > > Let me know what you think about this. > > > -Matthias > > On 5/8/18 3:08 PM, Ted Yu wrote: > > Bill:That makes sense. > > Using boolean should suffice. > > -------- Original message --------From: Bill Bejeck <bbej...@gmail.com> > Date: 5/8/18 2:48 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: > [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology > Optimization > > Thanks for the comments Guozhang and Ted. > > > > Guozhang: > > > > 1) I'll update the KIP in the "Compatibility, Deprecation and Migration > > Plan" with the expected impact of turning on optimization. But at this > > point, I have not identified a migration plan that doesn't involve having > > to stop all instances and restart. > > > > 2) Setting the type to String was just so we could have the default of > > null, indicating run no optimizations. As for partially enabling > > optimizations, I'm not sure I had that in mind, at least at this point. > > To me having the topology optimized should be an "all or nothing" > > proposition. For now, I'll change the type to boolean (with a default > > value of false) to better reflect the intent of the configuration. > > > > Ted, thanks again for the comments. > > > > The intent of the new configuration, as I mentioned above, is whether to > > turn optimization on or off in aggregate. The main reason for having the > > configuration is for backward compatibility. As optimization may result > in > > a slightly different topology from the original one, we need to allow > users > > to turn it off until they are ready for migrating to the new topology. > > > > I don't think having to select each optimization is a feasible > solution. I > > say this for few reasons: > > > > 1. Maintenance will be an issue. While the initial target is only a > > small number of optimizations, but as the number grows, having to > maintain > > the number of options in the code will difficult at best. > > 2. Users probably don't want to reason about which combination of > > optimizations to have. Thinking about which optimizations to turn on > or > > off raises the complexity of deploying an application. > > 3. When using a relational database or other another framework that > may > > optimize queries, as far as I know, it's not a choice of which > > optimizations to have, they are performed automatically. > > > > Does this make sense? > > > > > > On Mon, May 7, 2018 at 7:49 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > >> There are 4 subtasks for KAFKA-6034 > >> > >> If each optimization can be switched on/off, there should be 4 enums for > >> the switch. > >> > >> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> > >>> Hello Bill, > >>> > >>> Thanks for the KIP. My comments are the following: > >>> > >>> 1) We should state clearly what are the expected impact in > >> "Compatibility, > >>> Deprecation, and Migration Plan" when optimization is turned on. > >>> > >>> 2) I'm wondering why we define this config as a String typed, rather > than > >>> boolean typed? Or are you expecting to extend it to allow partially > >>> enabling part of the optimizations in the future? > >>> > >>> > >>> Hi Ted, > >>> > >>> The cover JIRA for topology optimization can be found here: > >>> https://issues.apache.org/jira/browse/KAFKA-6034 > >>> > >>> > >>> Guozhang > >>> > >>> > >>> On Mon, May 7, 2018 at 11:04 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >>> > >>>> Which JIRA is for the Topology Optimization itself ? > >>>> > >>>> Thanks > >>>> > >>>> On Mon, May 7, 2018 at 10:26 AM, Bill Bejeck <bbej...@gmail.com> > >> wrote: > >>>> > >>>>> All, > >>>>> I'd like to start a discussion about adding a configuration parameter > >>>>> allowing for the forthcoming topology optimization to be optional via > >>>>> configuration. > >>>>> > >>>>> > >>>>> The KIP can be found here: > >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>> 295%3A+Add+Streams+ > >>>>> Configuration+Allowing+for+Optional+Topology+Optimization > >>>>> > >>>>> Thanks, > >>>>> Bill > >>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> -- Guozhang > >>> > >> > > -- -- Guozhang
