Hi, sorry, what do you mean by 'container'? I use bare EC2 instances...
Shlomi
On Wed, Sep 10, 2014 at 1:41 AM, Jun Rao wrote:
> Are you starting the broker in some container? You want to make sure that
> the container doesn't overwrite the open file handler limit.
>
> Thanks,
>
> Jun
>
> On Tue
Hi folks,
In order to meet latency requirements for a system we are building, we
tested with different values of the above two parameters and found that
settings as low as 100 work best for us, balancing the required throughput
and latencies.
I just wanted to check if 100 is a sane value, notwith
Is it possible for the high level consumer to use a different zk cluster
than the cluster that manages broker leader electivity?
The high level consumer adds a lot of value but I don't like the idea that
bad user code could pound the core zk and effectively hurt the kafka
brokers
mike marzo
908 209
When 0.8.2 arrives in the near future, consumer offsets will be stored by the
brokers, and thus that workload will not be impacting ZK.
Best Regards,
-Jonathan
On Sep 10, 2014, at 8:20 AM, Mike Marzo wrote:
> Is it possible for the high level consumer to use a different zk cluster
> than the
But that could potentially impact the brokers directly. Which is my core
concern.
Is there any idea of the perf impact of having brokers be client aware.
This moves kafka more toward the traditional mom style messaging broker...
mike marzo
908 209-4484
On Sep 10, 2014 11:24 AM, "Jonathan Weeks"
Actually, with the new producer, you can configure the batch size in bytes.
If you set the batch size to be smaller than the max message size, messages
exceeding the max message limit will be in its own batch. Then, only one
message will be rejected by the broker.
Thanks,
Jun
On Tue, Sep 9, 2014
https://botbot.me/freenode/apache-kafka/
Just FYI, wasn't sure if we had any logging in place
Cheers,
David
We are planning for upgrading to 0.8.1.1 from 0.8-beta.
Can you please let us know the impacts of doing it ?.
I understand we have fixes for deadlocks in Consumer, maybe we should upgrade
the consumer.
But if we run with default offset storage we can still run old producers right.
Thanks,
Bal
this topic foo is created by auto creation at Sep 3. I looked at the old
log files on controller broker 5. it has some errors regarding partition 1
and 2 when the topic is auto created. who is creating/updating IsrLeader
record?
errors regarding partition 1
Hey Guys,
The current transactionality proposal
(https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka)
and implementation use control messages to handle transactions in Kafka. Kafka
traditionally hasn't had control messages in its topics. Transactionality (as
it's
Hi Jun,
Thank for highlighting. Please correct me my understanding. If the max
request size <= message.max.bytes then batch size will be optimally decided
( batch size will be determine by either max.request.size limit or
batch.size which ever is less).
Here is configuration parameter I was re
We actually don't allow "." in the topic name. Topic name can be
alpha-numeric plus "-" and "_".
Thanks,
Jun
On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
wrote:
> Thanks, I was using without JMX. I will go through doc. But how about
> Topic or Metric name Topic Name Convention or Metric Na
You just need to set batch.size to be less than 2MB. max.request.size in
the producer has to be less than socket.request.max.bytes in the broker.
Thanks,
Jun
On Wed, Sep 10, 2014 at 2:22 PM, Bhavesh Mistry
wrote:
> Hi Jun,
>
> Thank for highlighting. Please correct me my understanding. If th
I meant whether you start the broker in service containers like jetty or
tomcat.
Thanks,
Jun
On Wed, Sep 10, 2014 at 12:28 AM, Shlomi Hazan wrote:
> Hi, sorry, what do you mean by 'container'? I use bare EC2 instances...
> Shlomi
>
> On Wed, Sep 10, 2014 at 1:41 AM, Jun Rao wrote:
>
> > Are y
As long as the I/O load is reasonable, this is probably ok.
Thanks,
Jun
On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala
wrote:
> Hi folks,
>
> In order to meet latency requirements for a system we are building, we
> tested with different values of the above two parameters and found that
> se
Storing consumer offsets in the broker will add a bit overhead to the
broker, but shouldn't be too bad. Compared with traditional messaging
systems, Kafka delivers messages to consumers in order and therefore only
needs to keep track of a single last consumed offset per consumer client
per topic pa
David,
Thanks for the pointer. Added the link to our website.
Jun
On Wed, Sep 10, 2014 at 11:05 AM, David Arthur wrote:
> https://botbot.me/freenode/apache-kafka/
>
> Just FYI, wasn't sure if we had any logging in place
>
> Cheers,
> David
>
>
>
Yes, the older producer should work with the 0.8.1.1 brokers.
Thanks,
Jun
On Wed, Sep 10, 2014 at 11:28 AM, Seshadri, Balaji wrote:
> We are planning for upgrading to 0.8.1.1 from 0.8-beta.
>
> Can you please let us know the impacts of doing it ?.
>
> I understand we have fixes for deadlocks i
That's awesome.
-Jay
On Wed, Sep 10, 2014 at 11:05 AM, David Arthur wrote:
> https://botbot.me/freenode/apache-kafka/
>
> Just FYI, wasn't sure if we had any logging in place
>
> Cheers,
> David
>
>
Thanks Jun.
On Thu, Sep 11, 2014 at 4:13 AM, Jun Rao wrote:
> As long as the I/O load is reasonable, this is probably ok.
>
> Thanks,
>
> Jun
>
> On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala
> wrote:
>
> > Hi folks,
> >
> > In order to meet latency requirements for a system we are building
20 matches
Mail list logo
