Thanks, Jun. That is exactly what I want to know.
Libo
> Date: Thu, 23 Oct 2014 17:53:24 -0700
> Subject: Re: a questions about 0.8.1 async publishing
> From: jun...@gmail.com
> To: users@kafka.apache.org
>
> The new java producer to be released in 0.8.2 supports a callback on each
> message sen
Rajiv,
The new producer does maintain a buffer per partition, but you need to
consider synchronizing the access to the buffer since it can take data from
multiple caller threads. I think Jay's suggestion 1) does the same thing
for your purpose if you already have the data buffer storing your data:
Thanks I'll take a look at both. Just to be sure we are talking about
client version 0.82 right?
On Fri, Oct 24, 2014 at 8:39 AM, Guozhang Wang wrote:
> Rajiv,
>
> The new producer does maintain a buffer per partition, but you need to
> consider synchronizing the access to the buffer since it
I think 0.8.2 already used the new producer as the standard client.
Guozhang
On Fri, Oct 24, 2014 at 8:51 AM, Rajiv Kurian wrote:
> Thanks I'll take a look at both. Just to be sure we are talking about
> client version 0.82 right?
>
>
>
> On Fri, Oct 24, 2014 at 8:39 AM, Guozhang Wang wrote:
>
Hi all!
One of our broker went out of sync for one partition in one topic and we
were wondering how this can happen. The broker is still in the list of
replica and registered in zookeeper with the right ip address.
I noticed that the offset in replication-offset-checkpoint is stuck to a
certain v
Thanks!
On Fri, Oct 24, 2014 at 9:03 AM, Guozhang Wang wrote:
> I think 0.8.2 already used the new producer as the standard client.
>
> Guozhang
>
> On Fri, Oct 24, 2014 at 8:51 AM, Rajiv Kurian
> wrote:
>
> > Thanks I'll take a look at both. Just to be sure we are talking about
> > client vers
Hmm, I haven't read the design doc lately, but I'm surprised that there's
even a discussion of sequential disk access. I suppose for small subsets of
the writes you can write larger blocks of sequential data, but that's about
the extent of it. Maybe one of the developers can speak more to that aspe
Todd,
Did you load-test using SSDs?
Got numbers to share?
On Fri, Oct 24, 2014 at 10:40 AM, Todd Palino wrote:
> Hmm, I haven't read the design doc lately, but I'm surprised that there's
> even a discussion of sequential disk access. I suppose for small subsets of
> the writes you can write larg
We haven't done any testing of Kafka on SSDs, mostly because our storage
density needs are too high. Since our IO load has been fine on the current
model, we haven't pushed in that direction yet. Additionally, I haven't
done any real load testing since I got here, which is part of why we're
going t
At this moment, the encoder is called from a single thread per producer
instance.
Thanks,
Jun
On Thu, Oct 23, 2014 at 8:20 PM, Rajiv Kurian wrote:
> Are encoders only ever called from a single thread?
>
> I have a stateful utility class that I use to encode my objects. Is it safe
> to only cre
Yeah this is the org.apache.kafka.producer client in the clients/ directory.
-Jay
On Fri, Oct 24, 2014 at 9:54 AM, Rajiv Kurian wrote:
> Thanks!
>
> On Fri, Oct 24, 2014 at 9:03 AM, Guozhang Wang wrote:
>
> > I think 0.8.2 already used the new producer as the standard client.
> >
> > Guozhang
I am seeing following exception, don't understand the issue here. Is there
a way to resolve this error?
client consumer logs:
Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
groupB_ip-10-38-19-230-1414174925481-97fa3f2a can't rebalance after 4
retries
at
kafka.c
Hi,
We have a Kafka / Zookeeper test setup on a single EC2 machine. The
host.name is the private ip address and the advertised.host.name is the
public address in server.properties.
We are trying to publish from outside EC2 using the public ip of the EC2
machine but get this error on the client si
hi kafka'ers
I couldn't find anything useful where I could add partitions through an API
using python or simple calls to kafka/zookeeper without writing my own
java/scala code.
In another way, is there a python/REST way to do what create-topics alter
functionality does to change partitions ?
Tha
Hey David, that would be something I think we should support with moving
administrative functions into the broker and accessed over new Admin
Request/Response wire protocol calls. There are a few steps though I think
first in doing that. One is
https://issues.apache.org/jira/browse/KAFKA-1694 so
Thanks joe.
Wondering if there are any hacks for time being until that gets folded ?
like creating a child zk under partitions, etc ?
On Fri, Oct 24, 2014 at 12:59 PM, Joe Stein wrote:
> Hey David, that would be something I think we should support with moving
> administrative functions into the
Looks great, Gwen. I've added a few comments to the ticket.
On Mon, Oct 20, 2014 at 2:32 PM, Gwen Shapira wrote:
> Hi Kyle,
>
> I added new documentation, which will hopefully help. Please take a look
> here:
> https://issues.apache.org/jira/browse/KAFKA-1555
>
> I've heard rumors that you are v
Could you see if this is the case:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
?
On Fri, Oct 24, 2014 at 11:35 AM, Sameer Yami wrote:
> Hi,
>
> We have a Kafka / Zookeeper test setup on a single EC2 machine. The
> host.name is the
In python you can (easily) wrap the shell commands for that function
https://kafka.apache.org/documentation.html#basic_ops_modify_topic
import subprocess
subprocess.Popen(["/opt/apache/kafka/bin/kafka-topcs.sh","--alter",
and parse the response from it
/**
I saw the example at
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
and it answered most of my questions. I am still trying to figure out the
pattern to be used when I want to use a single simple consumer to fetch
messages from different partitions (possibly from dif
That is not the case.
This detailed error is below. Due to this error , the producer is not able
to send messages
2014-10-24 14:31:47,209 main DEBUG kafka.producer.BrokerPartitionInfo-52:
Getting broker partition info for topic Test
2014-10-24 14:31:47,209 main INFO kafka.client.ClientUtils$-68
hi kafka'ers
I couldn't find anything useful where I could add partitions through an API
using python or simple calls to kafka/zookeeper without writing my own
java/scala code.
In another way, is there a python/REST way to do what create-topics alter
functionality does to change partitions ?
Tha
HI Kafka Community ,
I am using kafka trunk source code and I get following exception. What
could cause the iterator to have FAILED state. Please let me know how I
can fix this issue.
*java.lang.IllegalStateException: Iterator is in failed stateat
kafka.utils.IteratorTemplate.hasNext(It
Which version of Kafka are you using on the consumer?
On Fri, Oct 24, 2014 at 4:14 PM, Bhavesh Mistry
wrote:
> HI Kafka Community ,
>
> I am using kafka trunk source code and I get following exception. What
> could cause the iterator to have FAILED state. Please let me know how I
> can fix thi
Is there a better way to do this where I can build one big requests with
broker -> partition mappings and call a consumer.fetch() with one giant
request?
Unfortunately, not until the 0.9 consumer is released. Until then, you can
take a look at ZookeeperConsumerConnector on how it manages multiple
I am using one from the Kafka Trunk branch.
Thanks,
Bhavesh
On Fri, Oct 24, 2014 at 5:24 PM, Neha Narkhede
wrote:
> Which version of Kafka are you using on the consumer?
>
> On Fri, Oct 24, 2014 at 4:14 PM, Bhavesh Mistry <
> mistry.p.bhav...@gmail.com>
> wrote:
>
> > HI Kafka Community ,
> >
26 matches
Mail list logo
