Yup :) On Mon, Mar 19, 2018 at 10:01 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> bq. some snippet like ProduceRequest / ProduceRequest > > Did you mean ProduceRequest / Response ? > > Cheers > > On Mon, Mar 19, 2018 at 9:51 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Matthias, > > > > About 2: yeah I guess this is a subjective preference. My main concern > > about keeping the config / handling code beyond 1.2 release is that it > will > > become a non-cleanable tech debt forever, as fewer and fewer users would > > need to upgrade from 0.10.x and 1.1.x, and eventually we will need to > > maintain this for nearly no one. On the other hand, I agree that this > tech > > debt is not too large. So if more people feel this is a good tradeoff to > > pay for not enforcing users from older versions to upgrade twice I'm > happen > > to change my opinion. > > > > A few more minor comments: > > > > 4. For the values of "upgrade.from", could we simply to only major.minor? > > I.e. "0.10.0" than "0.10.0.x" ? Since we never changed compatibility > > behavior in bug fix releases we would not need to specify a bug-fix > version > > to distinguish ever. > > > > 5. Could you also present the encoding format in subscription / > assignment > > metadata bytes in version 2, and in future versions (i.e. which first > bytes > > would be kept moving forward), for readers to better understand the > > proposal? some snippet like ProduceRequest / ProduceRequest in > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > > would be very helpful. > > > > > > > > Guozhang > > > > > > On Fri, Mar 16, 2018 at 2:58 PM, Matthias J. Sax <matth...@confluent.io> > > wrote: > > > > > Thanks for your comments. > > > > > > 1. Because the old leader cannot decode the new Subscription it can > only > > > send an empty assignment back. The idea to send empty assignments to > all > > > members is interesting. I will try this out in an PR to see how it > > behaves. > > > > > > 2. I don't see an issue with keeping config `upgrade.from` for future > > > releases. Personally, I would prefer to not force users to do two > > > upgrades if they want to go from pre-1.2 to post-1.2 version. Is there > a > > > technical argument why you want to get rid of the config? What > > > disadvantages do you see keeping `upgrade.from` beyond 1.2 release? > > > > > > Keeping the config is just a few lines of code in `StreamsConfig` as > > > well we a single `if` statement in `StreamsPartitionAssignor` to force > a > > > downgrade (cf > > > https://github.com/apache/kafka/pull/4636/files#diff- > > > 392371c29384e33bb09ed342e7696c68R201) > > > > > > > > > 3. I updated the KIP accordingly. > > > > > > > > > -Matthias > > > > > > On 3/15/18 3:19 PM, Guozhang Wang wrote: > > > > Hello Matthias, thanks for the KIP. Here are some comments: > > > > > > > > 1. "For all other instances the leader sends a regular Assignment in > > > > version X back." Does that mean the leader will exclude any member of > > the > > > > group whose protocol version that it does not understand? For > example, > > if > > > > we have A, B, C with A the leader, and B bounced with the newer > > version. > > > In > > > > the first rebalance, A will only consider {A, C} for assignment while > > > > sending empty assignment to B. And then later when B downgrades will > it > > > > re-assign the tasks to it again? I felt this is unnecessarily > > increasing > > > > the num. rebalances and the total latency. Could the leader just > sends > > > > empty assignment to everyone, and since upon receiving the empty > > > assignment > > > > each thread will not create / restore any tasks and will not clean up > > its > > > > local state (so that the prevCachedTasks are not lost in future > > > rebalances) > > > > and re-joins immediately, if users choose to bounce an instance once > it > > > is > > > > in RUNNING state the total time of rolling upgrades will be reduced. > > > > > > > > 2. If we want to allow upgrading from 1.1- versions to any of the > > future > > > > versions beyond 1.2, then we'd always need to keep the special > handling > > > > logic for this two rolling-bounce mechanism plus a config that we > would > > > > never be able to deprecate; on the other hand, if the version probing > > > > procedure is fast, I think the extra operational cost from upgrading > > from > > > > 1.1- to a future version, to upgrading from 1.1- to 1.2, and then > > another > > > > upgrade from 1.2 to a future version could be small. So depending on > > the > > > > experimental result of the upgrade latency, I'd suggest considering > the > > > > trade-off of the extra code/config needed maintaining for the special > > > > handling. > > > > > > > > 3. Testing plan: could you elaborate a bit more on the actual > > > upgrade-paths > > > > we should test? For example, I'm thinking the following: > > > > > > > > a. 0.10.0 -> 1.2 > > > > b. 1.1 -> 1.2 > > > > c. 1.2 -> 1.3 (simulated v4) > > > > d. 0.10.0 -> 1.3 (simulated v4) > > > > e. 1.1 -> 1.3 (simulated v4) > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 11:17 PM, Matthias J. Sax < > > matth...@confluent.io > > > > > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> I want to propose KIP-268 to allow rebalance metadata version > upgrades > > > >> in Kafka Streams: > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > >> 268%3A+Simplify+Kafka+Streams+Rebalance+Metadata+Upgrade > > > >> > > > >> Looking forward to your feedback. > > > >> > > > >> > > > >> -Matthias > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
