Good day,
I'm wondering a bit about the effects of changing the log.retention
setting, be it in the configuration or on-the-fly via ZooKeeper, will it
apply to already existing log segments, or only for new ones?
For example, we have a default of 12 hours, if I change the value to 24
hours in the
r, I'm not sure how useful it will be with "a large number of topics"
> / turning its output into a script that alerts upon a threshold. Could take
> a look and see what they're doing though.
>
> On Mon, Mar 16, 2015 at 8:31 AM, Mathias Söderberg <
> mathias.so
Good day,
I'm looking into using SimpleConsumer#getOffsetsBefore and offsets
committed in ZooKeeper for monitoring the lag of a consumer group.
Our current use case is that we have a service that is continuously
consuming messages of a large number of topics and persisting the messages
to S3 at s
t; https://issues.apache.org/jira/browse/KAFKA-1952. The new patch may not
> > improve the latency and CPU usage further, but will potentially improve
> > memory consumption. It would be great if you guys can test the new patch
> > out.
> >
> > Thanks,
> >
t; wrote:
> > > > >
> > > > > > I saw a very similar jump in CPU usage when I tried upgrading
> from
> > > > > 0.8.1.1
> > > > > > to 0.8.2.0 today in a test environment. The Kafka cluster there
> is
> > > two
> &g
anks,
>
> Jun
>
> On Tue, Feb 3, 2015 at 5:09 AM, Mathias Söderberg <
> mathias.soederb...@gmail.com> wrote:
>
> > Jun,
> >
> > I re-ran the hprof test, for about 30 minutes again, for 0.8.2.0-rc2 with
> > the same version of snappy that 0.8.1.1 u
prof output, one has to be aware that the thread state isn't shown and
> the thread need not necessarily be doing any CPU activity.
>
> -Jaikiran
>
>
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 10:05 AM, Mathias Söderberg <
&g
0.05% of the CPU. In 0.8.2.0, it uses about 0.10% of the CPU. We did
> upgrade snappy from 1.0.5 in 0.8.1.1 to 1.1.1.6 in 0.8.2.0. Could you try
> to use the same version of snappy in 0.8.1.1 and 0.8.2.0 and rerun your
> tests?
>
> Jun
>
> On Mon, Feb 2, 2015 at 1:15
nges in the socket server code btw 0.8.1 and 0.8.2. Could
> >> you run the same hprof test on 0.8.1 so that we can see the difference?
> >>
> >> Jaikiran,
> >>
> >> The fix you provided in probably unnecessary. The channel that we use in
> >> Si
observe the problem.
>
> On Tue, Dec 9, 2014 at 3:47 AM, Mathias Söderberg <
> mathias.soederb...@gmail.com> wrote:
>
> > Hi Neha,
> >
> > Yeah sure. I'm not familiar with hprof, so any particular options I
> should
> > include or just run with defau
put?
>
> On Mon, Dec 8, 2014 at 1:25 AM, Mathias Söderberg <
> mathias.soederb...@gmail.com> wrote:
>
> > Good day,
> >
> > I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that
> > the CPU usage on the broker machines went up by roughly 4
Good day,
I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that
the CPU usage on the broker machines went up by roughly 40%, from ~60% to
~100% and am wondering if anyone else has experienced something similar?
The load average also went up by 2x-3x.
We're running on EC2 and th
Does the log cleaner in 0.8.2 support topics with compressed messages? IIRC
that wasn't supported in 0.8.1.1.
On 29 November 2014 at 17:23, Jun Rao wrote:
> Yes, log cleaner is in 0.8.2. You just need to set the retention policy of
> a topic to "compact".
>
> Thanks,
>
> Jun
>
> On Thu, Nov 27,
y time the leader is moved as part of partition
> reassignment.
>
> I've played around with controlled shutdown and noticed that the broker
> that's shutting down is truncating logs prior to actually shutting down,
> which I don't really understand?
>
> can you
Good day,
We're currently running a Kafka cluster in our staging environment, testing
everything out and so far it runs fairly smoothly.
One thing that we're currently looking into is scaling a cluster that is
already up 'n serving data, and I have some questions regarding moving
partitions betwe
t designed for my use case.
>
> thanks,
>
> Jason
>
>
> On Mon, Dec 17, 2012 at 1:40 PM, Mathias Söderberg
> wrote:
> > The SimpleConsumer API [1] has a method called getOffsetsBefore which
> takes
> > a topic, partition, timestamp (UNIX I assume since it'
The SimpleConsumer API [1] has a method called getOffsetsBefore which takes
a topic, partition, timestamp (UNIX I assume since it's a long) and integer
limit on how many offsets to get.
Might not solve your problem *exactly*, but could be useful, unless you're
using the ConsumerConnector?
[1]: ht
17 matches
Mail list logo
