Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Pierre, Thanks for your persistence on this issue. I've gone back and forth on this a few times. The current API can definitely be annoying in some cases, but breaking compatibility still sucks. We do have the @Unstable annotation on the API, but it's unclear what exactly it means and I'm guess

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Most of the use cases of pause/resume that I've seen work only on single partitions (e.g in Kafka Streams), so the current varargs method is kind of nice. It would also be nice to be able to do the following: consumer.pause(consumer.assignment()); Both variants seem convenient in different situat

Re: KafkaConsumer.poll(long) pauses indefinitely

See https://issues.apache.org/jira/browse/KAFKA-1894. -Jason On Wed, Feb 10, 2016 at 2:08 PM, William Grim wrote: > Hi, > > I'm noticing that when we have a thread that comes up and calls > KafkaConsumer.poll(long), it will pause for a much longer time than our 1s > timeout. It looks like this

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) I agree with Grant on getting KIP-4 into this release if it's possible. There are a lot of tools which depend on it. On Fri, Feb 19, 2016 at 12:52 PM, Ismael Juma wrote: > +1 (non-binding) > > Ismael > > On Fri, Feb 19, 2016 at 7:00 PM, Becket Qin wrote: > > > Hi All, > > > >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I think Dana's suggestion to include the Kafka release version makes a lot of sense. I'm actually wondering why you would need the individual API versions if you have that? It sounds like keeping track of all the api version information would add a lot of complexity to clients since they'll have to

Re: [DISCUSS] KIP-35 - Retrieve protocol version

downgrade? > > On Wed, Mar 2, 2016 at 3:16 PM, Jason Gustafson > wrote: > > > I think Dana's suggestion to include the Kafka release version makes a > lot > > of sense. I'm actually wondering why you would need the individual API > > versions if you have th

Re: [DISCUSS] KIP-35 - Retrieve protocol version

x27;t > know a priori if you will work with the next release until you know if the > protocol changed. > > -Jay > > On Wed, Mar 2, 2016 at 5:28 PM, Jason Gustafson > wrote: > > > Hey Jay, > > > > Yeah, I wasn't suggesting that we eliminate request API

Re: [DISCUSS] KIP-35 - Retrieve protocol version

o an older version of the topic metadata request. What do you think? -Jason On Wed, Mar 2, 2016 at 5:28 PM, Jason Gustafson wrote: > Hey Jay, > > Yeah, I wasn't suggesting that we eliminate request API versions. They're > definitely needed on the broker to support compati

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> > > > Do cool new stuff > > > > elif version >= (0, 8, 2): > > > > Do some older stuff > > > > > > > > else: > > > > raise UnsupportedVersionError > > > > > > > > This will break if / when t

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

back to this, see below. > > On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson > wrote: > > > > > 1. For subscribe() and assign(), change the parameter type to collection > as > > planned in the KIP. This is at least source-compatible, so as long as > use

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

+users On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson wrote: > Hey Ismael, > > Thanks for bringing this up again. Just a quick question: if we do #1, > then there's no way that a user binary could work against both 0.9 and 0.10 > of kafka-clients, right? I'm not

Re: Review Request 34789: Patch for KAFKA-2168

hod and cannot be used to interrupt the other calls. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review87257 --- On J

Re: Review Request 34789: Patch for KAFKA-2168

/SubscriptionState.java, > > line 196 > > <https://reviews.apache.org/r/34789/diff/8/?file=980084#file980084line196> > > > > To be consistent with the naming convention with the rest of the > > methods, should we just name it offsetRestNeeded()? Haha, I ac

Re: [Discussion] New Consumer API / Protocol

Just a minor correction, but #2 is KAFKA-2168 and #3 is KAFKA-2123. For #1, I think there should be some minimal effort to making the poll respect the timeout (with the understanding that spurious wakeups can happen). I think this really just means calling NetworkClient.poll() in a loop and checki

Re: Review Request 34789: Patch for KAFKA-2168

lock forever. So, we don't get a chance to do > > the wakeup check. > > Jason Gustafson wrote: > I might be wrong, but I think we can still use NetworkClient.wakeup to > interrupt a poll call which is waiting forever. > > Jun Rao wrote: > Yes, you are rig

Re: Review Request 34789: Patch for KAFKA-2168

no real guarantee on the poll time, so it > > seems that we could just return when wakeup is called. > > Jason Gustafson wrote: > You might be waking up from a synchronous commit, for example. In that > case, all we can do is raise an exception. We could alternatively

Re: Review Request 34789: Patch for KAFKA-2168

e000cf8e10ebfacd6c9ee68d7b88ff8c157f73c6 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 34789: Patch for KAFKA-2168

ing() is supposed to be a blocking call. Basically, at > > the end of the call, we expect the fetch offset to be set to the beginning. > > This is now changed to async, which doesn't match the intended behavior. We > > need to think through if this matters or not. >

Re: Review Request 34789: Patch for KAFKA-2168

he classes named XResponse may be a bit confusing because the protocol > > responses use that terminology. Future? Result? > > Jason Gustafson wrote: > Agreed. In fact, they were XResult initially. I changed them because > BrokerResult and CoordinatorResult didn't seems

Review Request 35371: Patch for KAFKA-2266

/apache/kafka/common/network/SelectorTest.java d23b4b6a7060eeefa9f47f292fd818c881d789c1 Diff: https://reviews.apache.org/r/35371/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 35371: Patch for KAFKA-2266

effb1e63081ed2c1fff6d08d4ecdf8a3cb43e40a clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java d23b4b6a7060eeefa9f47f292fd818c881d789c1 Diff: https://reviews.apache.org/r/35371/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 35371: Patch for KAFKA-2266

This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35371/#review87667 --- On June 12, 2015, 4:19 p.m., Jason Gustafson wrote: > > -

Re: Review Request 34789: Patch for KAFKA-2168

ldn't allow wakeups in those cases? Maybe the only case we actually wakeup from is when we are polling for data? - Jason ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review87880

Re: Review Request 35231: Fix KAFKA-1740

rCoordinator.scala (line 285) <https://reviews.apache.org/r/35231/#comment140756> Same as above, wrong ticket. - Jason Gustafson On June 8, 2015, 11:12 p.m., Guozhang Wang wrote: > > --- > This is an automatically

Re: Review Request 34789: Patch for KAFKA-2168

: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 34789: Patch for KAFKA-2168

e significant. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review88441 --- On June 18, 2015, 9:4

Re: Review Request 34789: Patch for KAFKA-2168

x27;s worth pulling this functionality into a separate class? - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review88451 --------

Re: Review Request 34789: Patch for KAFKA-2168

2ebe3c21f611dc133a2dbb8c7dfb0845f8c21498 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Review Request 35655: Patch for KAFKA-2271

, Jason Gustafson

Re: Review Request 34789: Patch for KAFKA-2168

/test/java/org/apache/kafka/common/utils/UtilsTest.java 2ebe3c21f611dc133a2dbb8c7dfb0845f8c21498 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 35655: Patch for KAFKA-2271

st bugs > > > Diffs > - > > core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala > 98a5b042a710d3c1064b0379db1d152efc9eabee > > Diff: https://reviews.apache.org/r/35655/diff/ > > > Testing > --- > > > Thanks, > > Jason Gustafson > >

Re: Review Request 35655: Patch for KAFKA-2271

omatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35655/#review88690 --- On June 19, 2015, 4:48 p.m., Jason Gustafson wrote: > > --- > This is an automatically gener

Re: Review Request 34789: Patch for KAFKA-2168

9be8fbc648369ad9db1a7eea94bc1b9edbfdbfd7 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 34789: Patch for KAFKA-2168

ixed in latest patch. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review88914 --- On June 23, 2015, 4:39 p.m., Jason Gustafson wro

Re: Review Request 35791: Patch for KAFKA-2298

/Selector.java (line 282) <https://reviews.apache.org/r/35791/#comment141609> Do you think we should just move the disconnected.add() into the close method? - Jason Gustafson On June 23, 2015, 5:41 p.m., Dong Lin wrote: > > ---

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > <https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282> > > > > Do you think we should ju

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > <https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282> > > > > Do you think we should ju

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > <https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282> > > > > Do you think we should ju

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > <https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282> > > > > Do you think we should ju

Re: Review Request 35791: Patch for KAFKA-2298

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/#review89241 --- Ship it! Ship It! - Jason Gustafson On June 24, 2015, 8 p.m

Re: Review Request 35231: Address Onur and Jason's comments

> On June 18, 2015, 12:50 a.m., Jason Gustafson wrote: > > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines > > 229-236 > > <https://reviews.apache.org/r/35231/diff/1/?file=980918#file980918line229> > > > > In manual group man

Re: Review Request 35231: Address Onur and Jason's comments

> On June 18, 2015, 12:50 a.m., Jason Gustafson wrote: > > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines > > 229-236 > > <https://reviews.apache.org/r/35231/diff/1/?file=980918#file980918line229> > > > > In manual group man

Re: Review Request 34789: Patch for KAFKA-2168

/integration/kafka/api/ConsumerBounceTest.scala f56096b826bdbf760411a54ba067a6a83eca8a10 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 34789: Patch for KAFKA-2168

ed on the next interval, but that has disadvantages as well. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/#review89971

Re: Review Request 35231: Address Onur and Jason's comments

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/#review90155 --- Ship it! Ship It! - Jason Gustafson On June 30, 2015, 1:44 a.m

Review Request 36333: Patch for KAFKA-2123

92ffb91b5e039dc0d4cd0e072ca46db32f280cf9 Diff: https://reviews.apache.org/r/36333/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 36333: Patch for KAFKA-2123

lt for allowing poll to block indefinitely in spite of the passed timeout? Perhaps I'm the only one, but I'm still holding out hope that we'll be able to enforce the timeout even if we are in the middle of a join group. The code is actually not that far from

Re: Review Request 36333: Patch for KAFKA-2123

or dead. This we know we can recover from. A better solution would probably be to expose a connection state listener on NetworkClient to ensure that the consumer sees all disconnects. I started down this path for a while, but abandoned it when it started to get a little complex

Review Request 36368: Patch for KAFKA-1740

/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 36368: Patch for KAFKA-1740

core/src/test/scala/unit/kafka/coordinator/ConsumerCoordinatorResponseTest.scala 3cd726d291d0b1f9dd30d499c204e40961eb2c41 Diff: https://reviews.apache.org/r/36368/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 36368: Patch for KAFKA-1740

https://reviews.apache.org/r/36368/#review91187 ------- On July 9, 2015, 9:41 p.m., Jason Gustafson wrote: > > --- > This is an automatically generated e-ma

Re: Review Request 36333: Patch for KAFKA-2123

ggering poll() and fluding the > > network with metadata requests right? > > > > What we want in this case, is that the consumer should wait for > > retryBackoffMs before retry sending the next metadata request. > > Jason Gustafson wrote: > This code w

Re: Review Request 36333: Patch for KAFKA-2123

/RequestFutureTest.java PRE-CREATION core/src/test/scala/integration/kafka/api/ConsumerTest.scala 92ffb91b5e039dc0d4cd0e072ca46db32f280cf9 Diff: https://reviews.apache.org/r/36333/diff/ Testing --- Thanks, Jason Gustafson

Re: Review Request 36244: Patch for KAFKA-2312

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36244/#review91488 --- Ship it! LGTM - Jason Gustafson On July 7, 2015, 5 a.m., Tim

Re: Review Request 36341: Patch for KAFKA-2311

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36341/#review91556 --- Ship it! Ship It! - Jason Gustafson On July 9, 2015, 1:04 a.m

Re: Review Request 36333: Patch for KAFKA-2123

- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36333/#review91565 --- On July 12, 2015, 12:34 a.m., Jason Gustafson wrote: > >

Re: Review Request 36333: Patch for KAFKA-2123

ically generated e-mail. To reply, visit: https://reviews.apache.org/r/36333/#review91564 --- On July 12, 2015, 12:34 a.m., Jason Gustafson wrote: > > --- > This is an automa

Re: Review Request 36333: Patch for KAFKA-2123

/consumer/internals/RequestFutureTest.java PRE-CREATION core/src/test/scala/integration/kafka/api/ConsumerTest.scala 92ffb91b5e039dc0d4cd0e072ca46db32f280cf9 Diff: https://reviews.apache.org/r/36333/diff/ Testing --- Thanks, Jason Gustafson

Re: Build failed in Jenkins: KafkaPreCommit #147

Looks like there is already KAFKA-2055 for the transient ConsumerBounceTest failure. I'll have a look at it. -Jason On Tue, Jul 14, 2015 at 9:02 AM, Guozhang Wang wrote: > I think there is no JIRA ticket created for ConsumerBounceTest yet. We can > create a new one for it. > > On Tue, Jul 14, 2

Re: Review Request 36333: Patch for KAFKA-2123

clients/src/test/java/org/apache/kafka/clients/consumer/internals/RequestFutureTest.java PRE-CREATION core/src/test/scala/integration/kafka/api/ConsumerTest.scala 92ffb91b5e039dc0d4cd0e072ca46db32f280cf9 Diff: https://reviews.apache.org/r/36333/diff/ Testing --- Thanks, Jason

Re: Review Request 36333: Patch for KAFKA-2123

create a new jira. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36333/#review91646 --- On July 14, 2015, 8:21 p.m., Jason Gustafso

Re: Review Request 36333: Patch for KAFKA-2123

--- Thanks, Jason Gustafson

Re: Review Request 36333: Patch for KAFKA-2123

This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36333/#review91810 ------- On July 15, 2015, 1:21 a.m., Jason Gustafson wrote: > > -

Re: Review Request 36590: Patch for KAFKA-2275

KafkaConsumer in this approach. Feels like there ought to be a way to request the metadata we're interested in directly instead. It would involve a change to NetworkClient, but it might be worth looking into, at least to see the level of effort. - Jason Gusta

Re: Review Request 36590: Patch for KAFKA-2275

c/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java (lines 1074 - 1077) <https://reviews.apache.org/r/36590/#comment146361> Is it an actual problem if we return this topic to the user? - Jason Gustafson On July 20, 2015, 5

Re: Review Request 36590: Patch for KAFKA-2275

> On July 19, 2015, 1:11 a.m., Jason Gustafson wrote: > > > > Ashish Singh wrote: > Jason, thanks for your review! I looked into ConsumerNetworkClient/ > NetwrokClient, Metadata and Cluster classes. On receiving metadataUpdate, > cluster instance in metadata is

Re: Review Request 36681: Patch for KAFKA-2275

eviews.apache.org/r/36681/#comment146812> I think convention is to leave off the braces on one-line if statements. - Jason Gustafson On July 22, 2015, 6:32 a.m., Ashish Singh wrote: > > --- > This is an automatically gener

Re: Review Request 36681: Patch for KAFKA-2275

. clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java (line 173) <https://reviews.apache.org/r/36681/#comment146940> Why do we need to do this in a separate thread? - Jason Gustafson On July 22, 2015, 11:09 p.m., Ashish Singh

Re: Review Request 36681: Patch for KAFKA-2275

> On July 23, 2015, 7:19 a.m., Ashish Singh wrote: > > clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java, > > line 173 > > > > > > getAllTopics is a blocking call and we need to send

Re: [DISCUSS] Partitioning in Kafka

For a little background, the difference between this partitioner and the default one is that it breaks the deterministic mapping from key to partition. Instead, messages for a given key can end up in either of two partitions. This means that the consumer generally won't see all messages for a given

Re: Kafka Consumer thoughts

I think if we recommend a longer session timeout, then we should expose the heartbeat frequency in configuration since this generally controls how long normal rebalances will take. I think it's currently hard-coded to 3 heartbeats per session timeout. It could also be nice to have an explicit Leave

Re: Review Request 36858: Patch for KAFKA-2120

"nextDisconnects"? clients/src/test/java/org/apache/kafka/clients/MockClient.java (line 94) <https://reviews.apache.org/r/36858/#comment147498> Do we need to set sendMs in this method for consistency? - Jason Gustafson On July 27, 2015, 10:32 p.m., Mayuresh Gharat wrote: &g

Re: Review Request 36871: Patch for KAFKA-2381

scala/integration/kafka/api/ConsumerTest.scala (line 220) <https://reviews.apache.org/r/36871/#comment147533> Could we catch this issue more directly with a unit test for SubscriptionState? - Jason Gustafson On July 28, 2015, 12:56 a.m., Ashish

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93321 --- Ship it! Ship It! - Jason Gustafson On July 28, 2015, 4:17 p.m

Re: Kafka Consumer thoughts

; wrote: >>>>>>> >>>>>>>> Great discussion everyone! >>>>>>>> >>>>>>>> One general comment on the sync/async API's on the new consumer. I >>>>>>>> think the producer tackles syn

Re: Review Request 36858: Patch for KAFKA-2120

> On July 27, 2015, 10:55 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/clients/ClientRequest.java, line 26 > > <https://reviews.apache.org/r/36858/diff/1/?file=1022752#file1022752line26> > > > > Should ClientResponse.requestL

Re: Kafka Consumer thoughts

; inconsistency wrt the new producer API's use of Future because the two >>>>>>>> APIs >>>>>>>> have a much more fundamental difference that justifies it: they have >>>>>>>> completely different threading and execution models

Re: Kafka Consumer thoughts

that with the new setAssignment() api, we will also be getting > rid of pause() and resume()? > > Thanks, > > Jun > > On Fri, Jul 31, 2015 at 11:29 AM, Jason Gustafson > wrote: > >> I was thinking a little bit this morning about the subscription API and I >>

Re: New Consumer API and Range Consumption with Fail-over

end offset for each > partition. > > > > Please do let us know your preference to support above simple use-case. > > > Thanks, > > > Bhavesh > > On Thu, Jul 30, 2015 at 1:23 PM, Jason Gustafson > wrote: > > > Hi Bhavesh, > > > > I'm no

[DISCUSS] Client-side Assignment for New Consumer

Hi Kafka Devs, One of the nagging issues in the current design of the new consumer has been the need to support a variety of assignment strategies. We've encountered this in particular in the design of copycat and the processing framework (KIP-28). From what I understand, Samza also has a number o

Re: Review Request 36858: Patch for KAFKA-2120

aking out of the box. - Jason Gustafson On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https:

Re: Kafka Indentation

Can the java code be indented without affecting the results of git blame? If not, then I'd vote to leave it as it is. (Also +1 on rewriting Kafka in Java) -Jason On Tue, Aug 11, 2015 at 5:15 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Bump. Anyone else have an opinion? > > Ne

Re: Review Request 36858: Patch for KAFKA-2120

> On Aug. 11, 2015, 8:49 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java, > > line 302 > > <https://reviews.apache.org/r/36858/diff/4/?file=1037078#file1037078line302> > > > > Can we make t

Re: [DISCUSS] Client-side Assignment for New Consumer

cases where there are either a large number of consumers or topics. I wonder if you guys can share your thoughts about your scaling expectations? -Jason On Wed, Aug 12, 2015 at 12:28 PM, Jason Gustafson wrote: > Hey Jiangjie, > > That's a great point. In the worst case (the

Re: [DISCUSS] Client-side Assignment for New Consumer

inGroupRequest and let coordinator pass around, they are responsible for > understanding the risk. > > Thanks, > > Jiangjie (Becket) Qin > > > On Thu, Aug 13, 2015 at 12:41 PM, Jason Gustafson > wrote: > > > Hey Onur and Jiangjie, > > > > I'

Re: [DISCUSS] Client-side Assignment for New Consumer

ases (e.g. wildcard consumers) will almost > certainly > > >> fail for the first rebalance attempt. Even if we do that, since the > > >> consumers are getting metadata from different brokers, the metadata > > might > > >> still be inconsistent if there i

Re: [DISCUSS] Client-side Assignment for New Consumer

can simplify the metadata format to the following: > > > > > > > > > > > > GroupType => "consumer" > > > > > >> > > > > > >> Protocol => AssignmentStrategy > > > > > >> AssignmentS

Re: [DISCUSS] Client-side Assignment for New Consumer

onsumer" > > > > > >> > > > > > >> Protocol => AssignmentStrategy > > > > > >> AssignmentStrategy => String > > > > > >> > > > > > >> ProtocolMetadata => Versio

Re: [DISCUSS] Client-side Assignment for New Consumer

ability to know the metadata for group members, > > which > > > does not work for any non-consumer use case. > > > > > > I wonder if we may be complicating the design of 95% use cases for the > > > remaining 5%. For instance, how many times do pe

Re: [DISCUSS] Client-side Assignment for New Consumer

eing created/deleted. With > the new consumer, the rebalance can probably settle within 200ms when there > is a topic change. So, as long as we are not changing topic more than 5 > times per sec, there shouldn't be constant churns, right? > > Thanks, > > Jun > > >

Re: [DISCUSS] Client-side Assignment for New Consumer

re frequent metadata > change for consumers. Would that be an issue? > > Thanks, > > Jiangjie (Becket) Qin > > On Tue, Aug 18, 2015 at 10:22 AM, Jason Gustafson > wrote: > > > Hi Jun, > > > > Answers below: > > > > 1. When there are multiple

Re: [DISCUSS] KIP-71 Enable log compaction and deletion to co-exist

Hi Damian, Thanks for the KIP. We have a number of use cases in which we maintain a materialized cache of a compacted topic. The consumer coordinator, for example, has a cache of consumer offsets which is populated from the __consumer_offsets topic. Kafka Connect also uses this pattern for its own

Re: [DISCUSS] KIP-71 Enable log compaction and deletion to co-exist

Hey Damian, That's true, but it would avoid the need to write tombstones for the expired offsets I guess. I'm actually not sure it's a great idea anyway. One thing we've talked about is not expiring any offsets as long as a group is alive, which would require some custom expiration logic. There's

Re: [DISCUSS] KIP-62: Allow consumer to send heartbeats from a background thread

e the KIP to reflect these changes. Please let me know if you have any concerns. Thanks, Jason On Mon, Jun 13, 2016 at 5:46 PM, Jason Gustafson wrote: > Hey Becket, > > Sorry for the late response. I agree there's a little more to think > through on the implementation. The off

Re: [VOTE] KIP-75 - Add per-connector Converters

+1 (non-binding) On Mon, Aug 15, 2016 at 2:53 PM, Gwen Shapira wrote: > +1 (binding) > > On Mon, Aug 15, 2016 at 11:21 AM, Ewen Cheslack-Postava > wrote: > > I would like to initiate the voting process for KIP-75: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 75+-+Add+per-connect

Re: [VOTE] KIP-74: Add Fetch Response Size Limit in Bytes

Hi Andrey, Thanks for picking this up and apologies for the late comment. One thing worth mentioning is that the consumer actually sends multiple parallel fetch requests, one for each broker that is hosting some of the assigned partitions. Unless you were planning to modify this behavior, this KI

Re: [VOTE] KIP-74: Add FetchResponse size limit in bytes

+1 (non-binding) Thanks Andrey! On Thu, Aug 18, 2016 at 1:47 PM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: > Hi all! > I’ve modified KIP-74 a little bit (as requested by Jason Gustafson & Jun > Rao): > 1) provided more detailed explanation on memory usage (n

Re: [DISCUSS] Remove beta label from the new Java consumer

I'm +1 also. I feel a lot more confident about this with all of the system testing we now have in place (including the tests covering Streams and Connect). -Jason On Thu, Aug 25, 2016 at 9:57 AM, Gwen Shapira wrote: > Makes sense :) > > On Thu, Aug 25, 2016 at 9:40 AM, Neha Narkhede wrote: > >

Re: Kafka consumers unable to process message

Hi Achintya, We have a JIRA for this problem: https://issues. apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an exception in this case or do you just want to keep it from blocking indefinitely? If the latter, you could escape the poll from another thread using wakeup(). Than

Re: Kafka consumers unable to process message

, let's figure out the "best" action to take... > > Looks like something I'd like to get a handle on. > > > On Aug 31, 2016, at 4:05 PM, Jason Gustafson wrote: > > > > Hi Achintya, > > > > We have a JIRA for this problem: https://issues. >

<    2   3   4   5   6   7   8   9   10   11   >