tening,
Cliff Rhyne
--
Cliff Rhyne
Software Engineering Manager
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged
information. Any
e we don't directly control the processing time. I consider it very
> likely in the next iteration that we will either 1) add a background thread
> to the consumer for asynchronous heartbeating or 2) expose an API to make
> it easy for users to do the same thing.
>
>
> Thanks,
ffsets
Is there a special way to read the consumer offsets topic?
Thanks,
Cliff
--
Cliff Rhyne
Software Engineering Manager
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(
.coordinator.GroupMetadataManager\$OffsetsMessageFormatter
>
>
> If for some reason that doesn’t work, you can try
> "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead.
>
> -Todd
>
>
>
>
> On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote:
>
> > I&#
g I can offer now
> is
> > did you put quotes around the arg to --formatter so you don’t get weird
> > shell interference?
> >
> > -Todd
> >
> >
> > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote:
> >
> > > Thanks, Todd. It'
Hi Tao,
Sorry for the delay. Thanks for pointing out that property. That was the
fix.
On Mon, May 9, 2016 at 6:00 PM, tao xiao wrote:
> You need to enable internal topic in the consumer.properties
>
> exclude.internal.topics=false
>
> On Mon, 9 May 2016 at 12:42 Cli
> > > > if
> > > > > > I choose to lower the session timeout for some reason, I will
> have
> > to
> > > > > > modify the number of records returning too.
> > > > > >
> > > > > > If those assumptions are correct,
gt; > > > Given the background, it sounds like you'll generally want
> each
> > > > call
> > > > > to
> > > > > > > > poll() to return the same number of events (which is the
> number
> > > you
> > > > > > > planned
> &g
d the
alway-on consumer eventually doesn't succeed in rejoining (I see the
attempt in the logs to rejoin but nothing happens after that). I only have
logs on the client side to work with; there's nothing showing up in the
kafka logs to show why the group's state isn't stable.
u using the 0.9.0.0 release? We've fixed a few problems in the 0.9.0
> branch, some of which might explain the behavior you're seeing. There was
> one bug in particular which resulted in the consumer not fetching data for
> a set of partitions after a rebalance.
>
> -Jason
>
imeline. I haven't seen
> any major consumer-related bugs in the past couple weeks, so my feeling is
> that it's starting to stabilize. It would be nice to get KIP-41 into the
> next release though.
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:18 PM, Cliff Rhyne wrote:
>
>
; Not sure I understand the issue. Rebalances are triggered when either 1)
> group membership changes, 2) a consumer's subscription changes, or 3) when
> the number of partitions for a topic changes.
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:38 PM, Cliff Rhyne wrote:
>
>
arate groups.
> Would that not make sense in this case?
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:54 PM, Cliff Rhyne wrote:
>
> > I'll explain.
> >
> > Say there are two topics, foo and bar. Foo has two partitions (foo-0 and
> > foo-1). Bar has
er.enable=false
>
> default.replication.factor=3
>
> auto.create.topics.enable=true
>
> controlled.shutdown.enable=true
>
> delete.topic.enable=true
>
> # Zookeeper #
>
> # Zookeeper connection string (see zookeeper docs for detai
s there a constraint that I'm not aware of or is this something that could
be changed?
Thanks,
Cliff
--
Cliff Rhyne
Software Engineering Lead
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
Just following up on this concern. Is there a constraint that prevents
ConsumerGroupCommand from reporting offsets on a group if no members are
connected, or is this just the current implementation?
Thanks,
Cliff
On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote:
> I'm running in
he log compaction policy).
>
> There is a ticket open for improving on this scenario (
> https://issues.apache.org/jira/browse/KAFKA-2720) which lets the
> coordinator to only "purge" dead groups periodically instead of
> immediately, and that may partially resolve your
kept in the log.
> > When
> > > > we
> > > > > delete the group, we remove the corresponding entry from memory map
> > and
> > > > put
> > > > > a tombstone into log as well so that the old offsets will be
> > compacted
> > > &g
her the latency will
> start degrading over time (in 1 week's time for instance). Hopefully, with
> Kafka 0.9.0.0 that won't happen.
>
> If you have other ideas what may be causing that, please, let me know. I
> appreciate it.
>
> Cheers,
> Leo
>
> On 21 January
Hi,
My team and I are looking into a problem where the Java high level consumer
provides duplicate messages if we turn auto commit off (using version
0.8.2.1 of the server and Java client). The expected sequence of events
are:
1. Start high-level consumer and initialize a KafkaStream to get a
Co
hey start consuming again, that they will consume from the last
> committed offset.
>
> You should get more verification on this, tho. I might be remembering
> wrong.
>
> -James
>
> > On Oct 21, 2015, at 8:40 AM, Cliff Rhyne wrote:
> >
> > Hi,
> >
iff,
>
> One other case I observed in my environment is - when there were gc pauses
> on one of our high level consumer in the group.
>
> Thanks,
> Kris
>
> On Wed, Oct 21, 2015 at 10:12 AM, Cliff Rhyne wrote:
>
> > Hi James,
> >
> > There are two scenari
hold
on a partition and shutdown is called before starting a high level consumer
but shutdown is done asynchronously?
- What are the various things that can cause a consumer rebalance other
than adding / removing high level consumers?
Thanks,
Cliff
On Wed, Oct 21, 2015 at 4:20 PM, Cliff Rhyne wr
dropped out of the group and rejoins. This typically happens when
> you have a ZK session timeout. In that case, you should see "ZK expired" in
> your log. You can search for that and see if that is the problem.
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Oct 22, 2015 at 1:
ng logged) is very dangerous in our situation.
One or both of these might need to be behind a consumer config parameter,
I'm not familiar enough with when these get created and when the behavior
is simply changed for the better for the kafka project.
What are your thoughts?
Thanks,
Cliff
On Fr
> >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
&
balances
> and can address the problem in KAFKA-2725 better.
>
> Thanks,
>
> Jun
>
>> On Mon, Nov 2, 2015 at 11:16 AM, Cliff Rhyne wrote:
>>
>> Hi Jun,
>>
>> I openned KAFKA-2725 based on my experience with duplicate message
>> processing
t; the documentation
> > >>
> > >> Now, I would like to develop my application with kafka. I'm
> comfortable
> > >> using scala/java
> > >>
> > >> Please let me know which producer and consumer (both high level and
> > >> simple) c
> https://github.com/apache/kafka/blob/ef5d168cc8f10ad4f0efe9df4cbe849a4b35496e/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
>
> Thanks
>
> Jeff
>
>
>
> On Thu, Nov 5, 2015 at 12:51 PM, Cliff Rhyne wrote:
>
> > Hi Jeff,
> >
> > Is there a
(with the committed() method), but it doesn't replace
the OffsetRequest.
Is this a value we could either create another request type for or add it
to the OffsetAndMetadata response for the committed() method?
Thanks,
Cliff
On Thu, Nov 5, 2015 at 1:12 PM, Cliff Rhyne wrote:
> Hi Jeff,
>
Cliff Rhyne created KAFKA-3161:
--
Summary: Refactor Java client's use of the Properties class
Key: KAFKA-3161
URL: https://issues.apache.org/jira/browse/KAFKA-3161
Project: Kafka
Issue
[
https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cliff Rhyne updated KAFKA-3161:
---
Description:
The KafkaConsumer takes a Properties class for the config, but then instead of
using
Cliff Rhyne created KAFKA-2725:
--
Summary: high level consumer rebalances with auto-commit disabled
should throw an exception
Key: KAFKA-2725
URL: https://issues.apache.org/jira/browse/KAFKA-2725
Project
33 matches
Mail list logo
