Hi All
The PMC for Apache Kafka has invited Deng Ziming to become a committer,
and we are excited to announce that he has accepted!
Ziming has been contributing to Kafka for about three years. He has authored
more than 100 patches and helped to review nearly as many. In particular,
he made signif
Hi Bill,
I found a potential blocker here:
https://issues.apache.org/jira/browse/KAFKA-10799. A patch should be along
shortly.
Best,
Jason
On Mon, Nov 30, 2020 at 7:27 AM Bill Bejeck wrote:
> Thanks for the vote, Gwen.
>
> Here's an update for Jenkins build
>
> * Successful Jenkins builds fo
Hey Randall,
Thanks for putting this together. I think it would be great if the blog
included the names of the release contributors. It's an easy way to give
some recognition. I know we have done that in the past.
Thanks,
Jason
On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch wrote:
> I've prepare
The PMC for Apache Kafka has invited Konstantine Karantasis as a committer
and we
are pleased to announce that he has accepted!
Konstantine has contributed 56 patches and helped to review even more. His
recent work includes a major overhaul of the Connect task management system
in order to support
+1
I ran the basic quickstart on the 2.12 artifact and verified
signatures/checksums.
I also looked over the release notes. I see that KAFKA-8950 is included, so
maybe they just need to be refreshed.
Thanks for running the release!
-Jason
On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote:
>
Hi David,
Thanks for running the release. I think we should consider getting this bug
fixed: https://issues.apache.org/jira/browse/KAFKA-8896. The impact of this
bug is that consumer groups cannot commit offsets or rebalance. The patch
should be ready shortly.
Thanks,
Jason
On Fri, Sep 13, 201
ACL validation. Only authenticated clients
with Write permission on the respective topics are able to exploit this
vulnerability.
Mitigation: Apache Kafka users should upgrade to 2.1.1 or later where this
vulnerability has been fixed.
Acknowledgements: This issue was reported by Jason Gustafson
Ran the quickstart against the 2.11 artifact and checked the release notes.
For some reason, KAFKA-7897 is not included in the notes, though I
definitely see it in the tagged version. The RC was probably created before
the JIRA was resolved. I think we can regenerate without another RC, so +1
from
Hi All,
The PMC for Apache Kafka has invited Vahid Hashemian as a project committer and
we are
pleased to announce that he has accepted!
Vahid has made numerous contributions to the Kafka community over the past
few years. He has authored 13 KIPs with core improvements to the consumer
and the too
Hi all,
The PMC for Apache Kafka has invited Manikumar Reddy as a committer and we
are
pleased to announce that he has accepted!
Manikumar has contributed 134 commits including significant work to add
support for delegation tokens in Kafka:
KIP-48:
https://cwiki.apache.org/confluence/display/KAF
on:
> > >
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to th
+1 (binding). I checked release notes, documentation, and went through the
quickstart.
Thanks Matthias!
On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax
wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 0.10.2.2.
>
> Note,
Congrats Dong!
On Wed, Mar 28, 2018 at 12:04 PM, Mickael Maison
wrote:
> Congratulations Dong!
>
> On Wed, Mar 28, 2018 at 7:31 PM, Ismael Juma wrote:
> > Congratulations Dong! Thanks for your contributions so far and looking
> > forward to future ones.
> >
> > Ismael
> >
> > On Wed, 28 Mar 201
+1 Went through the quickstart, checked upgrade documentation. Thanks
Rajini!
On Tue, Mar 27, 2018 at 6:28 AM, Manikumar
wrote:
> +1 (non-binding)
>
> - Verified src, binary artifacts and basic quick start
> - Verified delegation token operations and docs
> - Verified dynamic broker configuratio
The fix has been merged to 1.1.
Thanks,
Jason
On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy wrote:
> Hi Jason,
>
> Ok - thanks. Let me know how you get on.
>
> Cheers,
> Damian
>
> On Wed, 28 Feb 2018 at 19:23 Jason Gustafson wrote:
>
> > Hey Damian,
&
Hey Damian,
I think we should consider https://issues.apache.org/jira/browse/KAFKA-6593
for the release. I have a patch available, but still working on validating
both the bug and the fix.
-Jason
On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax
wrote:
> No. Both will be released.
>
> -Matthias
+1. Verified artifacts and ran quickstart. Thanks Ewen!
-Jason
On Thu, Feb 15, 2018 at 1:42 AM, Rajini Sivaram
wrote:
> +1
>
> Ran quickstart with binaries, built source and ran tests,
>
> Thank you for running the release, Ewen.
>
> Regards,
>
> Rajini
>
> On Thu, Feb 15, 2018 at 2:31 AM, Guoz
> I believe a lot of users are using the kafka high level consumers, it is
> effectively an **unordered** messaging/streaming pattern. People using high
> level consumers don't actually need any ordering guarantees. In this sense,
> a *shared* subscription in Apache Pulsar seems to be better than c
Hi Khurrum,
Thanks for sharing the article. I think one interesting aspect of Pulsar
that stands out to me is its notion of a subscription and how it impacts
message retention. In Kafka, consumers are more loosely coupled and
retention is enforced independently of consumption. There are some
scena
Sorry for being late to the party, but congratulations Onur!
On Wed, Nov 8, 2017 at 1:47 AM, Sandeep Nemuri wrote:
> Congratulations Onur!!
>
> On Wed, Nov 8, 2017 at 9:19 AM, UMESH CHAUDHARY
> wrote:
>
> > Congratulations Onur!
> >
> > On Tue, 7 Nov 2017 at 21:44 Jun Rao wrote:
> >
> > > Af
+1. Hope this is the last one.
On Fri, Oct 27, 2017 at 10:28 AM, Guozhang Wang wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the fifth candidate for release of Apache Kafka 1.0.0. The main PRs
> that gets merged in after RC3 are the following:
>
> *https://github.com/
27;s worth expanding the API
> to return some additional info (e.g. generation id, group leader, ...)?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Jason Gustafson
> To: Kafka Users
> Cc: d...@kafka.apache.org
> Date: 07/19/2017 01:46 PM
> Subject:
What log level do you have configured? You might bump up to DEBUG or TRACE
and see if anything stands out.
-Jason
On Tue, Jul 18, 2017 at 7:59 PM, 陈江枫 wrote:
> Hi,
>
> I was integrating Kafka with Spark, using DirectStream, when my
> authentication fail, the stream just blocked. No log, no exc
signment too, so we can avoid an additional `--verbose`
> option?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Jason Gustafson
> To: Kafka Users
> Cc: d...@kafka.apache.org
> Date: 07/19/2017 01:46 PM
> Subject:Re: [DISCUSS] KIP-175: Additi
e: 07/17/2017 03:24 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hi Jason,
>
> Thanks for your quick feedback. Your suggestions seem reasonable.
> I'll start updating the KIP accordingly and w
although it's unfortunate that it's likely to
> break code that is parsing the output of the performance tool. Would it
> make sense to only enable this if an option is provided?
>
> Ismael
>
> On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson
> wrote:
>
>
+Users
Thanks for the KIP. I think tracking the rebalance time separately will
help resolve some confusion about the performance results given the
rebalance delay in KIP-134. And it seems generally useful to know how much
overhead is coming from the rebalance in any case.
-Jason
On Thu, Jul 13,
> > wrote:
>
> > I think if we had the opportunity to start from scratch, --describe
> would
> > have been the following:
> > --describe --offsets: shows all offsets committed for the group as well
> as
> > lag
> > --describe --state (or maybe --members)
Hey Vahid,
Thanks for the KIP. Looks like a nice improvement. One minor suggestion:
Since consumers can be subscribed to a large number of topics, I'm
wondering if it might be better to leave out the topic list from the
"describe members" option so that the output remains concise? Perhaps we
could
Hey Frank,
I think I spotted the issue and submitted a patch. Here's a link to the
JIRA: https://issues.apache.org/jira/browse/KAFKA-5456. I expect we'll get
the fix into 0.11.0.0. Thanks for finding this!
-Jason
On Thu, Jun 15, 2017 at 11:53 PM, Frank Lyaruu wrote:
> Yes, compression was on (
Woohoo! Great work, Rajini!
On Mon, Apr 24, 2017 at 3:27 PM, Jun Rao wrote:
> Congratulations, Rajini ! Thanks for all your contributions.
>
> Jun
>
> On Mon, Apr 24, 2017 at 2:06 PM, Gwen Shapira wrote:
>
> > The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
> > are ple
Congrats!
On Wed, Jan 11, 2017 at 11:57 AM, Sriram Subramanian
wrote:
> Congratulations Grant!
>
> On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote:
>
> > The PMC for Apache Kafka has invited Grant Henke to join as a
> > committer and we are pleased to announce that he has accepted!
> >
> >
+1
On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote:
> Looks like there was a good consensus on the discuss thread for KIP-106 so
> lets move to a vote.
>
> Please chime in if you would like to change the default for
> unclean.leader.election.enabled from true to false.
>
> https://cwiki.apa
t this is triggering the bug. We
> have a separate cluster where we don't have short-lived producers, and that
> one has been rock solid.
>
>
> We'll look into applying the patch, which could help reduce the problem.
> The ticket says it's being targeted for the 0.10.2
Hey Marcos,
Thanks for the report. Can you check out
https://issues.apache.org/jira/browse/KAFKA-3994 and see if it matches? At
a glance, it looks like the same problem. We tried pretty hard to get the
fix into the release, but it didn't quite make it. A few questions:
1. Did you not see this in
Great work, Becket!
On Mon, Oct 31, 2016 at 10:54 AM, Onur Karaman <
okara...@linkedin.com.invalid> wrote:
> Congrats Becket!
>
> On Mon, Oct 31, 2016 at 10:35 AM, Joel Koshy wrote:
>
> > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a
> > committer and we are pleased to
Had the wrong address for dev and users (haven't sent from this account
before).
On Thu, Oct 20, 2016 at 11:05 AM, Jason Gustafson wrote:
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.1.0. This is a feature release which includes the complet
+1 from myself too.
The vote passes with 9 +1 votes and no 0 or -1 votes.
+1 votes
PMC Members:
* Gwen Shapira
* Jun Rao
* Neha Narkhede
Committers:
* Ismael Juma
* Jason Gustafson
Community:
* Eno Thereska
* Manikumar Reddy
* Dana Powers
* Magnus Edenhill
0 votes
* No votes
-1 votes
* No
, Jason Gustafson wrote:
> Hello Kafka users, developers and client-developers,
>
> One more RC for 0.10.1.0. We're hoping this is the final one so that we
> can meet the release target date of Oct. 17 (Monday). Please let me know as
> soon as possible if you find any major proble
Hello Kafka users, developers and client-developers,
One more RC for 0.10.1.0. We're hoping this is the final one so that we can
meet the release target date of Oct. 17 (Monday). Please let me know as
soon as possible if you find any major problems.
Release plan: https://cwiki.apache.org/confluen
>
> Thanks,
> Damian
>
> On Wed, 12 Oct 2016 at 20:05 Dana Powers wrote:
>
>> +1; all kafka-python integration tests pass.
>>
>> -Dana
>>
>>
>> On Wed, Oct 12, 2016 at 10:41 AM, Jason Gustafson
>> wrote:
>> > Hello Kafka
Hello Kafka users, developers and client-developers,
One more RC for 0.10.1.0. I think we're getting close!
Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Rele
ase+Plan+0.10.1.
Release notes for the 0.10.1.0 release:
http://home.apache.org/~jgus/kafka-0.10.1.0-rc2/RELEASE_NOTES.
FYI: I'm cutting another RC this morning due to
https://issues.apache.org/jira/browse/KAFKA-4290. Hopefully this is the
last!
-Jason
On Mon, Oct 10, 2016 at 8:20 PM, Jason Gustafson wrote:
> The documentation is mostly fixed now: http://kafka.apache.org/0
> 101/documentation.html
The documentation is mostly fixed now: http://kafka.apache.org/
0101/documentation.html. Thanks to Derrick Or for all the help. Let me know
if anyone notices any additional problems.
-Jason
On Mon, Oct 10, 2016 at 1:10 PM, Jason Gustafson wrote:
> Hello Kafka users, developers and cli
Hello Kafka users, developers and client-developers,
This is the second candidate for release of Apache Kafka 0.10.1.0. This is
a minor release that includes great new features including throttled
replication, secure quotas, time-based log searching, and queryable state
for Kafka Streams. A full l
I'll submit a patch for the trivial changes in the quick start.
> Do you recommend adding Windows version of commands along with the current
> commands?
>
> I'll also open a JIRA for the new consumer issue.
>
> --Vahid
>
>
>
> From: Jason Gustafson
> To
>
> I suggest not having a "Fix version" set for issues that don't fix anything
> (it's not part of any release really).
Yeah, good call.
On Fri, Oct 7, 2016 at 8:59 AM, Ismael Juma wrote:
> On Fri, Oct 7, 2016 at 4:56 PM, Jason Gustafson
> wrote:
>
>
a.lang.OutOfMemoryError: Map failed
> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > ... 29 more
> >
> > This issue seems to break the broker and I have to clear out the logs so
> I
> > can bring the broker back up again.
> >
> >
>
ither from within the website or a google link - it
> always takes me to the top of the Kafka 0.10 documentation page. I can't
> figure out how to get to the javadoc.
>
> Thanks, Jonathan
>
> On Tue, Oct 4, 2016 at 6:46 PM Jason Gustafson wrote:
>
> > Huge improvemen
Huge improvement. Thanks Derrick and Gwen!
On Tue, Oct 4, 2016 at 5:54 PM, Becket Qin wrote:
> Much fancier now :)
>
> On Tue, Oct 4, 2016 at 5:51 PM, Ali Akhtar wrote:
>
> > Just noticed this on pulling up the documentation. Oh yeah! This new look
> > is fantastic.
> >
> > On Wed, Oct 5, 2016
One clarification: this is a minor release, not a major one.
-Jason
On Tue, Oct 4, 2016 at 4:01 PM, Jason Gustafson wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 0.10.1.0. This is
> a major release that in
Hello Kafka users, developers and client-developers,
This is the first candidate for release of Apache Kafka 0.10.1.0. This is a
major release that includes great new features including throttled
replication, secure quotas, time-based log searching, and queryable state
for Kafka Streams. A full li
>
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From: Neha Narkhede
> > > > To: "d...@kafka.apache.org" ,
> > > > "users@kafka.apache.org"
> > > > Cc: "priv...@ka
nding if 3 nodes
> are up and message already replicated why it's trying to fetch the data
> from failed node. Can you please explain bit details how it works. Thanks
> for your response.
>
> -Original Message-
> From: Jason Gustafson [mailto:ja...@confluent.io]
>
roup Metadata Manager on Broker 4]:
> Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.
> GroupMetadataManager)
> [2016-09-01 02:13:04,928] INFO [Group Metadata Manager on Broker 4]:
> Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.
> GroupMetadataManager)
> [
props.put("bootstrap.servers", "psaq1-wc.sys.comcast.net:
> 61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,
> psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3
> -ab.sys.comcast.net:61617");
>
, let's figure out the "best" action to take...
>
> Looks like something I'd like to get a handle on.
>
> > On Aug 31, 2016, at 4:05 PM, Jason Gustafson wrote:
> >
> > Hi Achintya,
> >
> > We have a JIRA for this problem: https://issues.
>
Hi Achintya,
We have a JIRA for this problem: https://issues.
apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an
exception in this case or do you just want to keep it from blocking
indefinitely? If the latter, you could escape the poll from another thread
using wakeup().
Than
Verified checksums and signatures. Ran quickstart. +1 (non-binding)
On Fri, Aug 5, 2016 at 2:51 PM, Jun Rao wrote:
> Thanks for running the release. +1
>
> Jun
>
> On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the t
his wrong
> >
> > [ac...@ekk001.scl ~]$ cat
> > /opt/kafka/kafka_2.11-0.10.0.0/config/log4j.properties
> > ..
> > log4j.rootLogger=DEBUG, kafkaAppender
> > ..
> >
> >
> > See no extra logs when running the consumer-group.sh tool.
> >
>
Hey Allen,
Can you turn on DEBUG logging and see if there's another exception? First
thought that occurs to me is that it might be the topic metadata request
which is actually failing. There was a version bump in 0.10 which would not
be supported by the 0.9 brokers.
Thanks,
Jason
On Tue, May 24,
To pile on a little bit, the API is designed to ensure consumer liveness so
that partitions cannot be held indefinitely by a defunct process. Since
heartbeating and message processing are done in the same thread, the
consumer needs to demonstrate "progress" by calling poll() often enough not
to get
Hi Florian,
It's actually OK if processing takes longer than the heartbeat interval,
but it does need to finish before the session timeout expires or the
consumer will be kicked out of the group (which typically is revealed by
commit failures). If the problem is just that the consumer is handling
Hi Marcos,
We should really update that wiki! We renamed ConsumerMetadataRequest to
GroupCoordinatorRequest in 0.9.0 because of the coordinator's more general
role. You might also want to take a look at consumer-groups.sh, which is
shipped with the release.
-Jason
On Thu, Mar 31, 2016 at 3:32 PM
This could be caused by a bug in our client's network layer which
occasionally prevents multiple requests from being sent at the same time.
Usually for both heartbeats and periodic commits, we expect the next one to
be sent successfully, so it hasn't been a big problem. However, when the
intervals
t; Cheers
> Oleg
>
> > On Mar 30, 2016, at 12:03 PM, Jason Gustafson
> wrote:
> >
> > Hi Prabhakar,
> >
> > We fixed a couple critical bugs in the 0.9.0.1 release, so you should
> > definitely make sure to use that version if you want to try it out. Since
Hi Prabhakar,
We fixed a couple critical bugs in the 0.9.0.1 release, so you should
definitely make sure to use that version if you want to try it out. Since
then, we've mostly been tweaking the behavior for some edge cases and
trying to improve messaging. I'd recommend giving it a shot. The upcom
try to reproduce it in Erlang, and post here a (hopefully
> sensible)
> sequence of timestamped heartbeat and commit requests and responses.
>
> Will ask more questions if we have new findings.
>
> Regards
> -Zaiming
>
>
>
> On Fri, Mar 25, 2016 at 5:43 PM, Jason Gust
Hey Ryan,
Sounds like you might be using the so-called "simple consumer" mode. If you
use assign() to give your consumer a specific partition, you're not
actually using a consumer group, so there won't be any coordination with
other consumers. If you use subscribe() on the other hand, then you sho
:09 AM, Zaiming Shi wrote:
> Hi Jason
>
> thanks for the reply!
>
> Forgot to mention that in we tried to test the simplest scenario in which
> there was only one member in the group. I think that should rule out group
> rebalancing right?
>
> On Thursday, March 24, 2
HI Zaiming,
I think the problem is not that commit requests aren't considered as
effective as heartbeats (they are), but that you can't rejoin the group
using only commits/heartbeats. Every time the group rebalances, all members
must rejoin the group by sending a JoinGroup request. Once a rebalanc
Have you looked at partitionsFor()?
-Jason
On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote:
> Hi,
>
> I'm using the new consumer in assign mode. I would like to assign all
> the partitions of a topic to the consumer.
>
> For that, I need to know the number of partitions available in the topi
...@gmail.com> wrote:
> Jason, Jay, could you please share bug number to track this issue?
>
> On Tue, Mar 15, 2016 at 10:57 PM, Jason Gustafson
> wrote:
>
> > Yeah, I agree the bug is probably more serious than I had thought before
> > (I've gotten too used to exam
e, Mar 15, 2016 at 11:24 AM, Jay Kreps wrote:
>
> > This seems like a bug, no? It should just initiate the request not wait
> for
> > it to be written, there is no way for the user to reason about the state
> of
> > the send buffer.
> >
> > -jay
> >
cases
that we handle internally).
-Jason
On Mon, Mar 14, 2016 at 6:50 PM, tao xiao wrote:
> Thanks Jason. What does consumer 1 would do upon
> receiving UNKNOWN_MEMBER_ID and does it rejoin the group eventually if it
> keeps polling?
>
> On Tue, 15 Mar 2016 at 00:58 Jason Gustafson
possible, even if that means making some things harder
> than they were with the old simple consumer?
>
>
>
>
>
> On Mon, Mar 14, 2016 at 2:35 PM, Jason Gustafson
> wrote:
> > Ah, that makes more sense. I have no idea about the limitations of your
> use
>
honestly don't see what the problem is with
> keeping poll() behavior as is, but also having public methods to e.g.
> fetch metadata, for people that need that functionality without
> actually consuming messages.
>
> On Mon, Mar 14, 2016 at 1:01 PM, Jason Gustafson
> wrot
:
>
> > No I haven't. It's still running the 0.9.0 client. I'll try upgrading if
> > it sounds like an old bug.
> >
> > On Mon, Mar 14, 2016 at 11:24 AM, Jason Gustafson
> > wrote:
> >
> >> Hey Rajiv,
> >>
> >> That
Hey Alexey,
Asynchronous commit handling could probably be improved quite a bit.
Basically what's happening is that the client's send buffer is getting
filled up, which causes the subsequent commits to fail with
SendFailedException. We don't currently implement any retrying for
asynchronous commit
I can take a look? Would the "sticky"
> proposal prefer to keep partitions assigned to consumers who currently have
> them and have not failed?
>
> On Mon, Mar 14, 2016 at 10:16 AM, Jason Gustafson
> wrote:
>
> > Hey Michael,
> >
> > I don't think
Hey Rajiv,
That sounds suspiciously like one of the bugs from 0.9.0.0. Have you
updated kafka-clients to 0.9.0.1?
-Jason
On Mon, Mar 14, 2016 at 11:18 AM, Rajiv Kurian wrote:
> Has any one run into similar problems. I have experienced the same problem
> again. This time when I use kafka-consum
Late arrival to this discussion. I'm not really sure I see the problem with
accessing the consumer in the rebalance listener. Before we passed the
consumer instance as a separate argument, but that was only because the
rebalance listener had to be passed by classname before a reference to the
consu
Hey Michael,
I don't think a policy of retrying indefinitely is generally possible with
the new consumer even if you had a heartbeat API. The problem is that the
consumer itself doesn't control when the group needs to rebalance. If
another consumer joins or leaves the group, then all consumers wil
Hey Tao,
This error indicates that a rebalance completed successfully before the
consumer could rejoin. Basically it works like this:
1. Consumer 1 joins the group and is assigned member id A
2. Consumer 1's session timeout expires before successfully heartbeating.
3. The group is rebalanced with
vice.
> If migrate form high level api to new consumer ,this approach is more
> straightly.
>
> Second approach only use for specific requirement, but it has to control
> more detail information.
> It is suitable for target clear job or web service to get given length
> offsets
This is actually a really good question. If you change the retention policy
of the offsets topic, then in the worst case, consumer groups could lose
their last committed positions and fall back to the auto reset behavior.
However, if your consumers are not down for a long time and you set the
reten
+users
On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson wrote:
> Hey Ismael,
>
> Thanks for bringing this up again. Just a quick question: if we do #1,
> then there's no way that a user binary could work against both 0.9 and 0.10
> of kafka-clients, right? I'm not
Hi Ken,
It looks like what happened is this:
1. First thread joins the group and is assigned partitions 0 and 1
2. First thread races through a bunch of messages from these partitions.
3. Second thread joins and is assigned partition 1 (leaving partition 0 for
the first thread)
4. Both threads re
Hi Robin,
Sorry for the late reply. I'm a little puzzled with your consumer code.
Once the "end" flag is set to true, you won't ever hit the poll() call
again, or am I missing something? Do you even need that inner loop?
-Jason
On Tue, Mar 1, 2016 at 6:06 AM, Péricé Robin wrote:
> Producer : h
g kafka-console-consumer.sh,
> console-consumer also hangs.
>
>
>
> Regards
> Muthu
>
>
> -Original Message-
> From: Jason Gustafson [mailto:ja...@confluent.io]
> Sent: Friday, March 04, 2016 5:33 AM
> To: users@kafka.apache.org
> Subject: Re: Consume
lient
from accessing the paths used by the brokers directly. See here for one
JIRA: https://issues.apache.org/jira/browse/KAFKA-1793.
-Jason
On Thu, Mar 3, 2016 at 4:51 PM, Jason Gustafson wrote:
> Hi Tian,
>
> Removing the client dependence on Zookeeper has been one of the main goals
Hi Tian,
Removing the client dependence on Zookeeper has been one of the main goals
of the Kafka team for a while now. It simplifies client development since
it's one less dependence and one less remote system they have to manage
interaction with. It also makes a lot of sense with the security fea
Hi there,
Just to clarify, is the broker still on 0.8? Unfortunately, the new
consumer needs 0.9. That probably would explain the hanging.
-Jason
On Thu, Mar 3, 2016 at 2:28 AM, Muthukumaran K
wrote:
> Ewen,
>
> By new Consumer API, you mean KafkaConsumer ? I have an issue with a poll
> in 0.9
that, i will experiment with the pause() api, separate thread
> for the actual message processing and poll()'ing with all partitions paused
>
> guven
>
>
> > On 25 Feb 2016, at 20:19, Jason Gustafson wrote:
> >
> > Hey Guven,
> >
> > This problem is
Hi there,
I think what you're asking is how the group protocol can guarantee that
each partition is assigned to one and only consumer in the group at any
point in time. Is that right? The short answer is that it can't. Because of
unexpected pauses on the client (e.g. for garbage collection), there
Hey Guven,
This problem is what KIP-41 was created for:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
.
The patch for this was committed yesterday and will be included in 0.10. If
you need something in the shorter term, you could probably use the client
fro
Hey Luke,
I took a look at the code and it does look like the whitelist argument is
handled differently between the old and new consumers. For the new
consumer, we just treat it as a raw regular expression, but the old
consumer does some preprocessing. We should probably do the preprocessing
in bo
ssigned the same partitions as
> before.
>
> On Wed, Feb 24, 2016 at 1:44 PM, Jason Gustafson
> wrote:
>
> > Sure, but in that case, the commits are still being stored in Kafka, so
> > resetting to the last committed position seems like what you want.
> >
> > -
invoke "commitSync" passing it the map of commits to sync.
>
> On Wed, Feb 24, 2016 at 1:35 PM, Jason Gustafson
> wrote:
>
> > I think the problem is the call to position() from within the callback.
> > When onAssigned() gets invoked, we don't have a posit
I think the problem is the call to position() from within the callback.
When onAssigned() gets invoked, we don't have a position yet, so calling
position() forces the consumer to request the last committed offset which
evidently causes an infinite loop. It might be worth opening a JIRA for
this sin
1 - 100 of 204 matches
Mail list logo
