Re: [DISCUSS] Deprecating the old consumers in trunk

Looks good to me too. On Fri, Dec 9, 2016 at 1:02 PM, Jason Gustafson wrote: > Hey Ismael, that sounds fair to me. I'm +1. > > -Jason > > On Thu, Dec 8, 2016 at 8:01 AM, Ismael Juma wrote: > > > Thanks Onur and Jason. I filed a JIRA to track this: > > > > https://issues.apache.org/jira/browse/K

Re: [DISCUSS] Deprecating the old consumers in trunk

Hey Ismael, that sounds fair to me. I'm +1. -Jason On Thu, Dec 8, 2016 at 8:01 AM, Ismael Juma wrote: > Thanks Onur and Jason. I filed a JIRA to track this: > > https://issues.apache.org/jira/browse/KAFKA-4513 > > My take is that this would be good to have and one could argue that we > should n

Re: [DISCUSS] Deprecating the old consumers in trunk

Thanks Onur and Jason. I filed a JIRA to track this: https://issues.apache.org/jira/browse/KAFKA-4513 My take is that this would be good to have and one could argue that we should not remove the old consumers until we have it. However, I think we should still go ahead with the deprecation of the

Re: [DISCUSS] Deprecating the old consumers in trunk

I like the idea too. Once all the old consumers have upgraded to this new embedded KafkaConsumer and they switch over to the new consumer’s group assignment protocol, does anything change when a real KafkaConsumer joins the group? At this point, if we have say N consumers in the group, we will

Re: [DISCUSS] Deprecating the old consumers in trunk

Onur's suggestion or something like it sounds like it could work. Suppose we add some metadata in Zookeeper for consumers which support the embedded KafkaConsumer. Until all members in the group have declared support, the consumer will continue use Zk for their partition assignments. But once all m

Re: [DISCUSS] Deprecating the old consumers in trunk

Some coworkers may have had issues seeing my earlier post so reposting under a different email: So my earlier stated suboptimal migration plans and Joel's idea all suffer from either downtime or dual partition ownership and consumption. But I think there's a bigger problem: they assume users are

Re: [DISCUSS] Deprecating the old consumers in trunk

So my earlier stated suboptimal migration plans and Joel's idea all suffer from either downtime or dual partition ownership and consumption. But I think there's a bigger problem: they assume users are willing to do the full migration immediately. I'm not convinced that this is realistic. Some team

Re: [DISCUSS] Deprecating the old consumers in trunk

Not sure it is worth doing, but a simple migration approach that avoids *service* downtime could be as follows: - Add a “migration mode” to the old consumer that disables its fetchers and disables offset commits. i.e., the consumers rebalance and own partitions but do basically nothing.

Re: [DISCUSS] Deprecating the old consumers in trunk

Thanks James. I had read your post and was planning to find it in order to share it here so you saved me some work. :) Ismael On Fri, Nov 18, 2016 at 3:21 AM, James Cheng wrote: > Sorry to self-plug, but I wrote a blog post that talks about this, with > respect to mirrormaker. I came to the sam

Re: [DISCUSS] Deprecating the old consumers in trunk

Sorry to self-plug, but I wrote a blog post that talks about this, with respect to mirrormaker. I came to the same 3 solutions that Onur described. https://logallthethings.com/2016/10/07/mirrormaker-gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/

Re: [DISCUSS] Deprecating the old consumers in trunk

Hi Onur, It is a good point that there is currently no out of the box solution for migrating from the old consumer to the new consumer where neither downtime or duplicate consumption are acceptable. As I understand, this is important for some of the usages at LinkedIn. Do you have any plans to tac

Re: [DISCUSS] Deprecating the old consumers in trunk

Does this make sense given that we still don't have a graceful migration plan from the old to new consumer? Different suboptimal migration plans that I can think of are: 1. shutdown all the old consumers of a group first and start them back up with the new consumer, causing downtime. 2. have a mix

Re: [DISCUSS] Deprecating the old consumers in trunk

Starting to deprecate the old consumer in the next release seems like a good idea. Thanks, Jun On Tue, Oct 25, 2016 at 2:45 AM, Ismael Juma wrote: > Hi all, > > In 0.10.1.0, we removed the beta label from the new Java consumer > documentation and updated the various tools so that they can use

Re: [DISCUSS] Deprecating the old consumers in trunk

I am in favor of the deprecation in 0.10.2.0, and generally speaking I'd propose we make up a timeline guidance for removing deprecated clients / protocols. I remember we talked about this and someone proposed three major releases (i.e. a year based on our current timeline release plan), which sou

[DISCUSS] Deprecating the old consumers in trunk

Hi all, In 0.10.1.0, we removed the beta label from the new Java consumer documentation and updated the various tools so that they can use the new consumer without having to pass the `--new-consumer` flag (more specifically the new consumer is used if `bootstrap-server` is set). More details of th