There is only one broker that thinks it's the controller right now. The
double controller situation happened earlier this morning. Do the other
brokers have to be bounced after the controller situation is fixed? I did
not do that for all brokers.
Wes
On Apr 21, 2015 8:25 PM, "Jiangjie Qin" wrot
Looking at the output from the jmx stats from our Kafka cluster, I see a
more or less constant leader election rate of around 2.5 from our
controller. Is this expected, or does this mean that leaders are shifting
around constantly?
If they are shifting, how should I go about debugging, and what tr
A call with Amazon confirmed instability for d2 and c4 instances triggered
by lots of network activity. They fixed the problem and have since rolled
it out. We've been running Kafka with d2's for a little while now and so
far so good.
Wes
On Tue, Jun 2, 2015 at 1:39 PM, Wes Chow wrote:
>
> We
Should not matter. We're running 12.04.
Wes
On Jun 16, 2015 12:18 PM, "Henry Cai" wrote:
> Does it still matter whether we are using Ubuntu 14 or 12?
>
> On Tue, Jun 16, 2015 at 8:44 AM, Wesley Chow wrote:
>
> >
> > A call with Amazon confirmed instabil
We deployed a Samza process that effectively doubled our fetch rates in our
cluster and noticed a significant drop fetch-follower requests. What is the
difference between fetch-consumer and fetch-follower?
[image: Inline image 1]
We noticed that some topics moved along undisturbed, sending in the
I have a producer configured to snappy compress data sent to a cluster. Is
there some way to verify that the data indeed is being compressed? If I
peek at the .log files on the broker, I can read some plain text amongst
binary. Similarly, tcpdump shows plain text readable data. I do not know if
thi
sed with snappy.
> >
> > Another try would be to change compression to other values (or remove
> > compression) and compare the tcp dump with the one you already have.
> >
> >
> > Vincent.
> >
> > On Wed, Sep 20, 2017 at 11:58 PM, Wesley Chow wrote:
>
I’ve come up with a couple solutions since we too have a power law
distribution. However, we have not put anything into practice.
Fixed Slicing
One simple thing to do is to take each key and slice it into some fixed number
of partitions. So your function might be:
(hash(key) % num) + (hash(key
Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -Original Message-
> From: Wesley Chow [mailto:w...@chartbeat.com <mailto:w...@chartbeat.com>]
> Sent: Tuesday, May 03, 2016 9:51 AM
> To: users@kafka.apache.org <mailto:users@kafka.apache.org>
> Subject: Re: Ha
>
> Upload to S3 is partitioned by the "key" field. I.e, one folder per key. It
> does offset management to make sure offset commit is in sync with S3 upload.
We do this in several spots and I wish we had built our system in such a way
that we could just open source it. I’m sure many people have
put custom partitioner into production.
> Would you mind sharing how you worked around this currently?
>
> Srikanth
>
> On Tue, May 3, 2016 at 5:43 PM, Wesley Chow wrote:
>
>>>
>>> Upload to S3 is partitioned by the "key" field. I.e, one folder per ke
Are there any thoughts on supporting TTLs on keys in compacted logs? In
other words, some way to set on a per-key basis a time to auto-delete.
Wes
period it
> will be removed. You could run a process that reprocesses the log and sends
> a NULL to keys you want to purge based on some custom logic.
>
> On Thu, May 12, 2016 at 2:01 PM, Wesley Chow wrote:
>
>> Are there any thoughts on supporting TTLs on keys in compacted log
? Is your data
>> source expiring records and hence you want to auto "delete" the
>> corresponding Kafka records as well?
>>
>> Guozhang
>>
>>> On Thu, May 12, 2016 at 2:35 PM, Wesley Chow wrote:
>>>
>>> Right, I’m trying to a
I’ve been investigating some possible network performance issues we’re having
with our Kafka brokers, and noticed that traffic sent between brokers tends to
show frequent bursts of very small packets:
16:09:52.299863 IP stream02.chartbeat.net.9092 > stream03.chartbeat.net.39399:
Flags [P.], seq
Is there any guidance on a maximum number different keys in a compacted
log? Such total numbers, or "keys need to fit in memory, message data does
not", etc. Is it unreasonable to expect tens or hundreds of millions of
keys in a single topic to be handled gracefully?
Thanks,
Wes
16 matches
Mail list logo
