Log Cleanup/Deletion isn't working correctly[KAFKA-1194]

Hi, I have bothered you guys a lot so far regarding this and the bug ticket is still open in Kafka JIRA (1194). It seems that log.cleanup.policy=DELETE doesn't close the current file graceful and schedule a delete. My stacktrace for this error is given below: [2017-06-14 23:07:34,965] INFO Partit

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

ate Topic t1 on cluster c1 => Topic t1 on Cluster t2 > > > > Is that the best way of doing it ? > > > > > > > > > > On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna > > wrote: > > > >> You should plan this ahead and make sure that your

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

You should plan this ahead and make sure that your topic partition and replication factors are updated correctly using kafka topics admin or a batch job setup for Kafka Mirror Maker. Apache wiki explains how Mirror maker can guarantee a failsafe mirroring. I cannot quite remember how deep the mirro

Re: Kafka old log files couldn't be deleted [JIRA 1194]

Is there any work-around for this? How can we leverage the auto-cleanup without taking the server down? KR, On 1 June 2017 at 15:46, Mohammed Manna wrote: > Sorry for bugging everyone, but does anyone have any workaround which has > been implemented successfully? I am assuming that

Re: Cluster in weird state: no leaders no ISR for all topics, but it works!

om> wrote: > > > I might give it a try tomorrow. The reason for having so large init and > > sync limit times is because in the past our ZK cluster was storing large > > amount of data, and lower values were not enough for the server syncs > when > > restarting zk p

Re: Zookeeper on same server as Kafka

Usually, the overhead comes when you have kafka and zookeeper doing the housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you should keep their logs on totally different physical machine for better performance. Furthermore, if a mechanical failure occurs, you might not want both

Re: Cluster in weird state: no leaders no ISR for all topics, but it works!

data.dir=1 > zookeeper.connect=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181/kafka > zookeeper.connection.timeout.ms=6000 > > In general they're pretty much the default ones. > I can see in Zookeeper the kafka brokers connected to it and exchanging > data... > > Thanks fo

Re: Cluster in weird state: no leaders no ISR for all topics, but it works!

titions, but > now kafka-topics shows *Leader: none* when for all the other topics, it > shows *Leader: -1* > > > On 1 June 2017 at 17:05, Mohammed Manna wrote: > > > I had a similar situation, but only 1 of my ZKs was struggling - but > since > > the ISR synching t

Re: Cluster in weird state: no leaders no ISR for all topics, but it works!

services > which are not reporting problems. > As you suggested, I haven't tried restarting the kafka-server processes > because there's no leader for topic partitions, so I don't know what will > happen. Never been in a similar situation with Kafka after some years using &

Re: Kafka old log files couldn't be deleted [JIRA 1194]

2017 at 16:01, Mohammed Manna wrote: > Hi, > > I can see that this is an existing issue. The latest comments says > "Everything works fine after manual clean-up", but this information is > vague and doesn't really say "What/When to delete"? Has anyone g

Re: Cluster in weird state: no leaders no ISR for all topics, but it works!

Hi Alberto, Usually this means that the leader election/replica syncing couldn't be successful and the zookeeper logs should be able to show this information too. The leader -1 is what worries me. For your case (3 broker cluster), I am assuming you have done the cluster configuration to have 1 bro

Re: Java APIs for ZooKeeper related operations

1) For issue no. 1 I think you might find the AdminUtils useful This link should help you understand. I haven't got around using ACL for Kafka yet (as I am still doing PoC myself) - so probably

Kafka old log files couldn't be deleted [JIRA 1194]

Hi, I can see that this is an existing issue. The latest comments says "Everything works fine after manual clean-up", but this information is vague and doesn't really say "What/When to delete"? Has anyone got any idea whether this has been addressed already? I am using the latest release. FYI - t

log.cleanup.policy (Comma Separated list)

Hi, Since the documentation says that the acceptable values are comma-separated list of policies (compact, delete), does it mean that it will prioritise Left-to-Right when using the policies? For example, if I have set it to "compact,delete", does it mean the following: 1) Will perform log compac

File Transfers between two systems using Kafka

Hello, I currently have the following understanding of KafkaProducer and KafkaConsumer: 1) If I send a file, it's broken down in lines using some default delimiter (LF or \n). 2) Therefore, if 2 producers publish 2 different files to the same topic, that doesn't mean that they are going as two di

Question on message.max.bytes setting and inconsistencies observed

Hello, I have seen some (perhaps) strange behaviour with the brokers when using payloads which are: 1) pure byte array composed of random integers 2) byte array retrieved from a text file (5 MB size). I tried to use the following JUNIT test code to send payload data (random integers, the code is

Differece observed during kafka-producer-perf-test and JUnit producer test

Hi, I ran a JUnit test where i delivered a payload of ~5MB file to a Kafka cluser (3 broker, 3 zookeeper). These are all setup on my laptop. My test config for producer is the following: max.request.size=5242880 > batch.size=8192 > key.serializer=org.apache.kafka.common.serialization.StringSerial

Re: Question regarding buffer.memory, max.request.size and send.buffer.bytes

m. > > > > > > On Tue, May 23, 2017 at 3:28 PM, Mohammed Manna > wrote: > > > This could be for various reasons: > > > > 1) Your consumer.property settings - if you have not been acknowledging > > automatically, you need to provide a sufficient poll

Re: Question regarding buffer.memory, max.request.size and send.buffer.bytes

This could be for various reasons: 1) Your consumer.property settings - if you have not been acknowledging automatically, you need to provide a sufficient polling time and commit in sync/async. 2) You are not consuming the messages how you think. I don't know how you got this buffer.memory prope

Server Property num.partitions and Specification of Topic Paritions on Console

Hello, I wanted to understand the relationship between the number of partitions in default specification (server.properties) and topic specification. If I do the following: kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic myTopic It will actually *o

Request to Join Kafka user group

Request to Join Kafka user group