Hi Navneet,
Your payload size is larger than the maximum allowed size of a message on
the Producer/Broker, 100 is the current default value. Currently the
check for max allowed message size is present on both the producer and the
broker. So you can override the property "max.message.size" in
P
Hi Samir,
In 0.8, we don't have SyncProducer as a public api. You can use
kafka.javaapi.producer.Producer to send the data synchronously or
asynchronously.
The quick start page is outdated, and we plan to update a few pages soon
(before the 0.8 BETA release). In fact, we would love to have contr
Could you give some pointers (or test case perhaps) how to detect that a
message was sent successfully to the broker, when sending the data
asynchronously?
-Original Message-
From: Swapnil Ghike [mailto:sgh...@linkedin.com]
Sent: Friday, April 19, 2013 10:24 AM
To: users@kafka.apache.or
Hi Jun,
It seems I failed to respond to this post. My apologies. I did test a week
ago, with the latest 0.8 and it still failed. The reason I did wait so long is
to get my home machine setup, which I finished today. At work we are tearing
it up and leaving behind issues so we can get messag
The reason is the console consumer deletes the previous state information
from zookeeper when given the from-beginning option. So effectively only
the 2nd consumer instance is alive. You can run ConsumerShell instead.
Thanks,
Neha
On Thursday, April 18, 2013, Eric Sites wrote:
> Jun,
>
> I have
Neha,
That makes perfect sense now... :)
The console consumer is working just how I expect it to work now with
that option removed from the command line.
Thanks,
Eric
On 4/19/13 10:31 AM, "Neha Narkhede" wrote:
>The reason is the console consumer deletes the previous state information
>from
I'm pretty sure the issue is that those 3 zookeepers are not part of
the same quorum. Try following the quickstart for starting up a local
zookeeper instance or follow
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
for a zookeeper cluster setup.
Thanks,
Neha
On T
In 0.8, the producer API does not have callbacks to notify you of the
status of the messages. However, I think we want to rethink the client
APIs for 0.9. Let us know if you have feedback.
Thanks,
Neha
On Fri, Apr 19, 2013 at 4:15 AM, Itai Frenkel wrote:
> Could you give some pointers (or test c
I would second this request. We would like to gather metrics on the
SyncProducer (bytes in/out, messages in/out, time spent in preparing/sending
messages, etc.) when we are sending asynchronously. Or would you recommend
something like Scala actors as a better solution?
-drew
_
I would like to write a simple REST GW for event producers outside the
firewall, and that the REST would acknowledge that the items were persisted in
Kafka (so the REST client could decide to retry if an error occured).
Without any callback, I need to revert to the Producer sync API which means I
What is the relationship between topic partitions and consumers? From what I've
read and experimented with, the number of partitions limits the number of
active consumers in a given consumer group on a given topic. So if the number
of partitions is 10, and the number of consumers is 12 (all in t
That is correct, you cannot have more active consumers than the total
# of partitions.
Thanks,
Neha
On Fri, Apr 19, 2013 at 2:30 PM, Michael Golowka
wrote:
> What is the relationship between topic partitions and consumers? From what
> I've read and experimented with, the number of partitions li
Is the number of partitions calculated based on an individual broker, or a sum
of all partitions across multiple brokers? For instance, if I have 2 brokers
with 10 partitions each, is the maximum active consumers 10 or 20?
Michael Golowka
From: Neha Narkh
sum of all partitions across multiple brokers, so 20.
Thanks,
Neha
On Fri, Apr 19, 2013 at 2:56 PM, Michael Golowka
wrote:
> Is the number of partitions calculated based on an individual broker, or a
> sum of all partitions across multiple brokers? For instance, if I have 2
> brokers with 10 p
Great, thanks for the info!
Michael Golowka
From: Neha Narkhede [neha.narkh...@gmail.com]
Sent: Friday, April 19, 2013 4:04 PM
To: users@kafka.apache.org
Subject: Re: Consumers vs Partitions
sum of all partitions across multiple brokers, so 20.
Thanks,
Ne
Hi,
Actually I followed the above link to setup my zookeeper1 to zookeeper3.
They are in the same quorum, as you can see in my above example that when I
created the /testkafka path in zookeeper1, I can see list it in zookeeper2
Thanks
On Sat, Apr 20, 2013 at 12:08 AM, Neha Narkhede wrote:
> I
Neha,
Does that hold true for both .7.x and .8?
Joe
On Fri, Apr 19, 2013 at 3:53 PM, Neha Narkhede wrote:
> That is correct, you cannot have more active consumers than the total
> # of partitions.
>
> Thanks,
> Neha
>
> On Fri, Apr 19, 2013 at 2:30 PM, Michael Golowka
> wrote:
> > What is the
Hmm, so if you use all 3 zookeeper servers will creating and reading
the node, do you still see the problem ?
zkCli.sh -server zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
create /testkafka
ls /
Thanks
Neha
On Fri, Apr 19, 2013 at 8:55 PM, Ryan Chan wrote:
> Hi,
>
> Actually I followed the a
Yes, same for 0.8.
Thanks,
Neha
On Fri, Apr 19, 2013 at 9:21 PM, Joseph Crotty wrote:
> Neha,
>
> Does that hold true for both .7.x and .8?
>
> Joe
>
>
> On Fri, Apr 19, 2013 at 3:53 PM, Neha Narkhede wrote:
>
>> That is correct, you cannot have more active consumers than the total
>> # of parti
>> There must be some way I can extend/hack/fork into the Producer in the async
>> background thread in version 0.8 (and not wait until 0.9 is released). For
>> example in 0.7, it looked like extending DefaultEventHandler could do the
>> trick.
0.8 has the same DefaultEventHandler logic as 0.7,
20 matches
Mail list logo