RE: Details of segment deletion

Thanks, that's answered all my questions! Simon -Original Message- From: Gwen Shapira Sent: 13 June 2018 02:42 To: Users Subject: Re: Details of segment deletion See below: On Mon, Jun 11, 2018 at 3:36 AM, Simon Cooper < simon.coo...@featurespace.co.uk> wrote: > Hi

Details of segment deletion

tion by the log cleaner? Thanks for the help! Simon Cooper

RE: Getting the consumer to operate deterministically with small messages

Forgot to mention - this is on 0.9. We can't upgrade to 0.10 yet, as we haven't upgraded our brokers. -Original Message----- From: Simon Cooper [mailto:simon.coo...@featurespace.co.uk] Sent: 22 May 2017 16:05 To: users@kafka.apache.org Subject: Getting the consumer

Getting the consumer to operate deterministically with small messages

Hi, I'm having significant problems getting the kafka consumer to operate deterministically with small message numbers & sizes (this is for local testing). I'm controlling the offset manually, and using manual partition/topic assignment. I've set auto commit off, and set fetch.min.bytes to 1.

0.9 client persistently high CPU usage

Hi all, We've got a problem with high CPU usage on a 0.9 client. We've got a monitoring system that polls kafka topics for metadata (to get the last message offset) every so often, and this has started using very high CPU continuously. We're seeing the following being spammed in the logs every

RE: FW: 0.9 consumer log spam - Marking the coordinator dead

g the coordinator dead Hi Simon, It may be worth trying the 0.9.0 branch as it includes a number of important fixes to the new consumer. Ismael On Fri, Feb 5, 2016 at 12:33 PM, Simon Cooper < simon.coo...@featurespace.co.uk> wrote: > Actually, this is incorrect - it looks like the con

FW: 0.9 consumer log spam - Marking the coordinator dead

Actually, this is incorrect - it looks like the consumer does not receive any messages until the *first* 'coordinator dead' message is logged! Is anyone able to offer insight into what's going on here? Simon -Original Message----- From: Simon Cooper [mailto:simon.coo...@fea

0.9 consumer log spam - Marking the coordinator dead

Hi, We've just updated to the 0.9 client & broker, and we're suddenly seeing a lot of log spam in the consumers: 2016-02-05T03:31:34,182 | INFO | o.a.k.c.c.i.AbstractCoordinator [kafkaspout-thread-0] | Response-events:7 | Marking the coordinator 2147483646 dead. 2016-02-05T03:31:34,182 | INFO

RE: Does MirrorMaker delete topics?

a. Kind regards, Stevo Slavic. On Tue, Jan 5, 2016 at 2:53 PM, Simon Cooper < simon.coo...@featurespace.co.uk> wrote: > Hi all, > > In the kafka docs, it mentions that MirrorMaker will automatically > create topics on the mirror cluster if the right options are set. Does &g

Does MirrorMaker delete topics?

Hi all, In the kafka docs, it mentions that MirrorMaker will automatically create topics on the mirror cluster if the right options are set. Does it automatically delete topics on the mirror that are deleted on the main cluster as well? Thanks, SimonC

API to query cluster metadata on-demand

Is there a basic interface in the new client APIs to get the list of topics on a cluster, and get information on the topics (offsets, sizes, etc), without having to deal with a producer or consumer? I just want a basic synchronous API to query the metadata as-is. Does this exist in some form? T

RE: Topic partitions randomly failed on live system

auses! Thanks, SimonC -Original Message- From: Simon Cooper [mailto:simon.coo...@featurespace.co.uk] Sent: 17 August 2015 12:31 To: users@kafka.apache.org Subject: Topic partitions randomly failed on live system Hi, We've had an issue on a live system (3 brokers, ~10 topics, some

Topic partitions randomly failed on live system

Hi, We've had an issue on a live system (3 brokers, ~10 topics, some replicated, some partitioned) where a partition wasn't properly reassigned, causing several other partitions to go down. First, this exception happened on broker 1 (we weren't doing anything particular on the system at the ti

RE: new consumer api?

Reading on the consumer docs, there's no mention of a relatively simple consumer that doesn't need groups, coordinators, commits, anything like that - just read and poll from specified offsets of specific topic partitions - but automatically deals with leadership changes and connection losses (s

Is the new consumer API ready?

I'm updating the kafka APIs we use to the new standalone ones, but it look like the new consumer isn't ready yet (the code has got lots of placeholders etc), and there's only the producer in the Javadoc at http://kafka.apache.org/082/javadoc/index.html. Is there an ETA on when the new consumer

Finding out how underreplicated a broker is

In the output of kafka-topics.sh -describe, it gives the ISR. What I want to find out is why a particular broker is not in ISR, how far behind it is, and when it will get back into the ISR. I can't seem to find this information in the logs, cmdline output, or zookeeper. How can I find out how fa

SimpleConsumer long poll test case randomly times out

I've got a very strange bug doing a long poll using SimpleConsumer on 0.8.1.1. The test app I've got (attached) uses a thread to do a long poll of a topic, and then sends single messages to that queue, one at a time, waiting for the poll to return the message before sending the next one. The pr

Consuming from a replica

Hi, I've been looking at the SimpleConsumer example, and that I've noticed that it always reads from the leader, and reacts to leader changes by reconnecting to the new leader. Is it possible to read from a replica in ISR that's not the leader? If so, how does the consumer get notified the repl

Relationship between socket timeout & consumer wait timeout

I'm trying to use the SimpleConsumer to consume a queue, using long polling. I'm encountering some difficulties when using very long poll times (60s or more). It seems to be conflicting with the socket timeout. I thought the socket timeout was the timeout to actually connect to the broker, then