Hi Tao,
Sorry for the delay. Thanks for pointing out that property. That was the
fix.
On Mon, May 9, 2016 at 6:00 PM, tao xiao wrote:
> You need to enable internal topic in the consumer.properties
>
> exclude.internal.topics=false
>
> On Mon, 9 May 2016 at 12:42 Cli
g I can offer now
> is
> > did you put quotes around the arg to --formatter so you don’t get weird
> > shell interference?
> >
> > -Todd
> >
> >
> > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote:
> >
> > > Thanks, Todd. It'
.coordinator.GroupMetadataManager\$OffsetsMessageFormatter
>
>
> If for some reason that doesn’t work, you can try
> "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead.
>
> -Todd
>
>
>
>
> On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote:
>
> > I&#
ffsets
Is there a special way to read the consumer offsets topic?
Thanks,
Cliff
--
Cliff Rhyne
Software Engineering Manager
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(
e we don't directly control the processing time. I consider it very
> likely in the next iteration that we will either 1) add a background thread
> to the consumer for asynchronous heartbeating or 2) expose an API to make
> it easy for users to do the same thing.
>
>
> Thanks,
tening,
Cliff Rhyne
--
Cliff Rhyne
Software Engineering Manager
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged
information. Any
her the latency will
> start degrading over time (in 1 week's time for instance). Hopefully, with
> Kafka 0.9.0.0 that won't happen.
>
> If you have other ideas what may be causing that, please, let me know. I
> appreciate it.
>
> Cheers,
> Leo
>
> On 21 January
kept in the log.
> > When
> > > > we
> > > > > delete the group, we remove the corresponding entry from memory map
> > and
> > > > put
> > > > > a tombstone into log as well so that the old offsets will be
> > compacted
> > > &g
he log compaction policy).
>
> There is a ticket open for improving on this scenario (
> https://issues.apache.org/jira/browse/KAFKA-2720) which lets the
> coordinator to only "purge" dead groups periodically instead of
> immediately, and that may partially resolve your
Just following up on this concern. Is there a constraint that prevents
ConsumerGroupCommand from reporting offsets on a group if no members are
connected, or is this just the current implementation?
Thanks,
Cliff
On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote:
> I'm running in
[
https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cliff Rhyne updated KAFKA-3161:
---
Description:
The KafkaConsumer takes a Properties class for the config, but then instead of
using
Cliff Rhyne created KAFKA-3161:
--
Summary: Refactor Java client's use of the Properties class
Key: KAFKA-3161
URL: https://issues.apache.org/jira/browse/KAFKA-3161
Project: Kafka
Issue
s there a constraint that I'm not aware of or is this something that could
be changed?
Thanks,
Cliff
--
Cliff Rhyne
Software Engineering Lead
e: crh...@signal.co
signal.co
Cut Through the Noise
This e-mail and any files transmitted with it are for the sole use of the
er.enable=false
>
> default.replication.factor=3
>
> auto.create.topics.enable=true
>
> controlled.shutdown.enable=true
>
> delete.topic.enable=true
>
> # Zookeeper #
>
> # Zookeeper connection string (see zookeeper docs for detai
arate groups.
> Would that not make sense in this case?
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:54 PM, Cliff Rhyne wrote:
>
> > I'll explain.
> >
> > Say there are two topics, foo and bar. Foo has two partitions (foo-0 and
> > foo-1). Bar has
; Not sure I understand the issue. Rebalances are triggered when either 1)
> group membership changes, 2) a consumer's subscription changes, or 3) when
> the number of partitions for a topic changes.
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:38 PM, Cliff Rhyne wrote:
>
>
imeline. I haven't seen
> any major consumer-related bugs in the past couple weeks, so my feeling is
> that it's starting to stabilize. It would be nice to get KIP-41 into the
> next release though.
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:18 PM, Cliff Rhyne wrote:
>
>
u using the 0.9.0.0 release? We've fixed a few problems in the 0.9.0
> branch, some of which might explain the behavior you're seeing. There was
> one bug in particular which resulted in the consumer not fetching data for
> a set of partitions after a rebalance.
>
> -Jason
>
d the
alway-on consumer eventually doesn't succeed in rejoining (I see the
attempt in the logs to rejoin but nothing happens after that). I only have
logs on the client side to work with; there's nothing showing up in the
kafka logs to show why the group's state isn't stable.
gt; > > > Given the background, it sounds like you'll generally want
> each
> > > > call
> > > > > to
> > > > > > > > poll() to return the same number of events (which is the
> number
> > > you
> > > > > > > planned
> &g
> > > > if
> > > > > > I choose to lower the session timeout for some reason, I will
> have
> > to
> > > > > > modify the number of records returning too.
> > > > > >
> > > > > > If those assumptions are correct,
(with the committed() method), but it doesn't replace
the OffsetRequest.
Is this a value we could either create another request type for or add it
to the OffsetAndMetadata response for the committed() method?
Thanks,
Cliff
On Thu, Nov 5, 2015 at 1:12 PM, Cliff Rhyne wrote:
> Hi Jeff,
>
> https://github.com/apache/kafka/blob/ef5d168cc8f10ad4f0efe9df4cbe849a4b35496e/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
>
> Thanks
>
> Jeff
>
>
>
> On Thu, Nov 5, 2015 at 12:51 PM, Cliff Rhyne wrote:
>
> > Hi Jeff,
> >
> > Is there a
t; the documentation
> > >>
> > >> Now, I would like to develop my application with kafka. I'm
> comfortable
> > >> using scala/java
> > >>
> > >> Please let me know which producer and consumer (both high level and
> > >> simple) c
balances
> and can address the problem in KAFKA-2725 better.
>
> Thanks,
>
> Jun
>
>> On Mon, Nov 2, 2015 at 11:16 AM, Cliff Rhyne wrote:
>>
>> Hi Jun,
>>
>> I openned KAFKA-2725 based on my experience with duplicate message
>> processing
> >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
&
Cliff Rhyne created KAFKA-2725:
--
Summary: high level consumer rebalances with auto-commit disabled
should throw an exception
Key: KAFKA-2725
URL: https://issues.apache.org/jira/browse/KAFKA-2725
Project
ng logged) is very dangerous in our situation.
One or both of these might need to be behind a consumer config parameter,
I'm not familiar enough with when these get created and when the behavior
is simply changed for the better for the kafka project.
What are your thoughts?
Thanks,
Cliff
On Fr
dropped out of the group and rejoins. This typically happens when
> you have a ZK session timeout. In that case, you should see "ZK expired" in
> your log. You can search for that and see if that is the problem.
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Oct 22, 2015 at 1:
hold
on a partition and shutdown is called before starting a high level consumer
but shutdown is done asynchronously?
- What are the various things that can cause a consumer rebalance other
than adding / removing high level consumers?
Thanks,
Cliff
On Wed, Oct 21, 2015 at 4:20 PM, Cliff Rhyne wr
iff,
>
> One other case I observed in my environment is - when there were gc pauses
> on one of our high level consumer in the group.
>
> Thanks,
> Kris
>
> On Wed, Oct 21, 2015 at 10:12 AM, Cliff Rhyne wrote:
>
> > Hi James,
> >
> > There are two scenari
hey start consuming again, that they will consume from the last
> committed offset.
>
> You should get more verification on this, tho. I might be remembering
> wrong.
>
> -James
>
> > On Oct 21, 2015, at 8:40 AM, Cliff Rhyne wrote:
> >
> > Hi,
> >
Hi,
My team and I are looking into a problem where the Java high level consumer
provides duplicate messages if we turn auto commit off (using version
0.8.2.1 of the server and Java client). The expected sequence of events
are:
1. Start high-level consumer and initialize a KafkaStream to get a
Co
33 matches
Mail list logo
