Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

I am ok with #3 as well. On Mon, May 9, 2016 at 12:12 PM, Ismael Juma wrote: > +1 for option 3 (unfortunately all options have pros and cons, but I think > option 3 does a better job with regards to trade-offs). > > Ismael > > On Mon, May 9, 2016 at 7:56 PM, Gwen Shapira wrote: > > > We need to

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 for option 3 (unfortunately all options have pros and cons, but I think option 3 does a better job with regards to trade-offs). Ismael On Mon, May 9, 2016 at 7:56 PM, Gwen Shapira wrote: > We need to conclude this discussion in order to unblock the release. > > If I understand correctly, we

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

We need to conclude this discussion in order to unblock the release. If I understand correctly, we have 3 options on the table: 1. Add the missing APIs as "deprecated". Pros: Conceptually, the right way to get rid of an API without breaking Storm Cons: Users of the new API will get ugly warnings,

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Harsha, See inline. On Sun, May 8, 2016 at 3:02 AM, Harsha wrote: > Ismael, >Do we need to add old assign and subscribe that accepts List. Yes, to maintain binary compatibility. If you compile your jar with kafka-clients 0.9.0.1 and then run it with kafka-clients 0.10.0.0 you will

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Ismael, Do we need to add old assign and subscribe that accepts List. It will get implicitly cast to collection with the new methods. The only problem comes from the methods that accepts varargs. -Harsha On Sat, May 7, 2016, at 05:53 PM, Mark Grover wrote: > Thanks Ismael, I agree w

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Thanks Ismael, I agree with you, it makes sense to leave things the way they are in Kafka 0.10. On Fri, May 6, 2016 at 5:27 PM, Ismael Juma wrote: > Hi Mark, > > Thanks for the email. First of all, I'd like to mention that the `Unstable` > annotation has been removed from the new Java consumer i

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Mark, Thanks for the email. First of all, I'd like to mention that the `Unstable` annotation has been removed from the new Java consumer in 0.10, so you can expect compatibility from now on. We definitely understand that compatibility is important for widespread adoption. The current PR for KA

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

I understand and empathize with both sides of the story here. I spend some of my time on Spark and Kafka integration and I have cc'ed Cody who's been working on new Kafka consumer API with Spark Streaming. Spark hasn't merged the new Kafka consumer API integration, the PR is up and we, as a communi

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Just my two cents here: I agree with Ewen and Grant on the indication of the "unstable" annotations of being possible for backward incompatible. That means, users can make a call themselves of whether to start trying out the new APIs / libraries with the risk or changing code when it changes in a

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Jason, Yes I am in favor removing them 0.11 and it definitely gives everyone one major version to update their clients to remove the deprecated commands. Thanks, Harsha On Fri, Apr 29, 2016, at 11:02 PM, Ewen Cheslack-Postava wrote: > I agree with Grant that

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

I agree with Grant that we really need to indicate to consumers of APIs that when we mark it as unstable, it *really* means unstable. This is a more general problem of needing to define our APIs and stability -- but I's say that while we probably were too hasty in adding APIs, it was probably bette

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hey Harsha, Just to clarify, are you ok with removing the methods in a later release (say 0.11)? As I mentioned above, the only weird ones are subscribe() and assign(), which will have a deprecated version which accepts List. Users will have to change their code to use another collection type or a

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Grant, I am sure this is discussed and voted. I've seen the discussion. Given that there is an opportunity to make it less painful for the users who shipped consumers using the 0.9.x we should consider that. ". However, for now the documentation of > > the Unstab

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

If anyone wants to review the KIP call discussion we had on this just before the vote, here is a link to the relevant session (6 minutes in): https://youtu.be/Hcjur17TjBE?t=6m On Fri, Apr 29, 2016 at 12:21 PM, Grant Henke wrote: > I think you are right Jason. People were definitely on the fence

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

I think you are right Jason. People were definitely on the fence about this and we went back and forth for quite some time. I think the main point in the KIP discussion that made this decision, is that the Consumer was annotated with the Unstable annotation. Given how new the Consumer is, we wante

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hey Harsha, One issue with adding back subscribe(List), but marking it deprecated is that it may confuse some users if they use the typical Arrays.asList() pattern. You'd have to cast to a Collection to avoid the deprecation warning, which is awkward. Maybe it would be better in that case to keep

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Jason, "t. I think what you're saying is that the KafkaSpout has been compiled against the 0.9 client, but it may need to be to run against 0.10 (if the user depends on that version instead). Is that correct?" Yes thats true. " Is that correct? In general, are you expecting that KafkaSpout

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

FYI. I have attached a sample clients API change/compatibility report to KAFKA-1880 . The report shows changes in the public apis between the 0.9 and trunk branches. Some of them are expected per KIP-45 obviously. Thanks, Grant On Thu, Apr 28, 20

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hey Harsha, We're just trying to understand the problem first. I think what you're saying is that the KafkaSpout has been compiled against the 0.9 client, but it may need to be to run against 0.10 (if the user depends on that version instead). Is that correct? In general, are you expecting that Ka

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Ismael, This will solve both binary and source compatibility. Storm has new KafkaSpout that used 0.9.x new KafkaSpout API. As part of that spout we used KafkaConsumer.seekToBeginning and other methods. Since the method signatu

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi Harsha, What is the aim of the PR, is it to fix binary compatibility, source compatibility or both? I think it only fixes source compatibility, so I am interested in what testing has been done to ensure that this fix solves the Storm issue. Thanks, Ismael On Thu, Apr 28, 2016 at 12:58 PM, Har

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Hi, We missed this vote earlier and realized thats its breaking the 0.9.x client api compatibility. I opened a JIRA here https://issues.apache.org/jira/browse/KAFKA-3633 . Can we keep the old methods with deprecated tag in 0.10 release. Thanks, Harsha On Fri, Mar 18,

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 I want to clarify that the compatibility issue is only for unshaded dependencies. Code and wire protocol will be compatible. Gwen On Tue, Mar 15, 2016 at 2:18 PM, Jason Gustafson wrote: > I'd like to open the vote for KIP-45. We've discussed several alternatives > on the mailing list and in

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 (non-binding) On Wed, Mar 16, 2016 at 12:24 PM, Gwen Shapira wrote: > +1 > > I want to clarify that the compatibility issue is only for unshaded > dependencies. Code and wire protocol will be compatible. > > Gwen > > On Tue, Mar 15, 2016 at 2:18 PM, Jason Gustafson > wrote: > > > I'd like to

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1. Normally I'd be more of a stickler for compatibility, but this is new, I think it's worth emphasizing that unstable actually means unstable & might require recompile (and maybe even adapting code when we think the change warrants it), and I think the impact is relatively low since those adopti

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

Looks like the KIP has passed. The finally tally is +5 among committers and +9 overall. Thanks to Pierre-Yves Ritschard for all of the hard work and persistence with this! -Jason On Wed, Mar 16, 2016 at 9:01 PM, Ewen Cheslack-Postava wrote: > +1. > > Normally I'd be more of a stickler for comp

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

correcting my mistake - the proposal is wire-compatible, but not source compatible for some calls. My vote remains though. On Wed, Mar 16, 2016 at 10:24 AM, Gwen Shapira wrote: > +1 > > I want to clarify that the compatibility issue is only for unshaded > dependencies. Code and wire protocol wil

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 I think we should just do it and apologize and mark it stable. Also, the javadocs do show the unstable annotation but it isn't easy to figure out exactly what that means. -Jay On Wed, Mar 16, 2016 at 10:24 AM, Gwen Shapira wrote: > +1 > > I want to clarify that the compatibility issue is on

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 (non-binding) Ismael On Tue, Mar 15, 2016 at 9:18 PM, Jason Gustafson wrote: > I'd like to open the vote for KIP-45. We've discussed several alternatives > on the mailing list and in the KIP call, but this vote is only on the > documented KIP: > https://cwiki.apache.org/confluence/pages/view

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 (non-binding) On Wednesday, March 16, 2016, Gwen Shapira wrote: > correcting my mistake - the proposal is wire-compatible, but not source > compatible for some calls. My vote remains though. > > On Wed, Mar 16, 2016 at 10:24 AM, Gwen Shapira > wrote: > > > +1 > > > > I want to clarify that t

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 On Tue, Mar 15, 2016 at 2:18 PM, Jason Gustafson wrote: > I'd like to open the vote for KIP-45. We've discussed several alternatives > on the mailing list and in the KIP call, but this vote is only on the > documented KIP: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=613

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1 (non-binding) On Wed, Mar 16, 2016 at 1:40 PM, Ismael Juma wrote: > +1 (non-binding) > > Ismael > > On Tue, Mar 15, 2016 at 9:18 PM, Jason Gustafson > wrote: > > > I'd like to open the vote for KIP-45. We've discussed several > alternatives > > on the mailing list and in the KIP call, but th

Re: [VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

+1. And we also need to do a better job educating users about our annotations, since such things are likely to happen in KC / KS in the future. On Wed, Mar 16, 2016 at 2:20 PM, Ashish Singh wrote: > +1 (non-binding) > > On Wed, Mar 16, 2016 at 1:40 PM, Ismael Juma wrote: > > > +1 (non-binding)

[VOTE] KIP-45: Standardize KafkaConsumer API to use Collection

I'd like to open the vote for KIP-45. We've discussed several alternatives on the mailing list and in the KIP call, but this vote is only on the documented KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61337336. This change will not be compatible with 0.9, but it will provid