Re: Delete "kafka-logs"

Hi, Thanks Sachin for your great effort. 😃 The last settings you suggested actually did the job for me. I still do not understand why my previous settings did not work. However I learned a lot about log cleaning. Thank you so much for your quick replies and time. Cheers, Regards, Eranga Hesha

Re: Delete "kafka-logs"

Maybe because of these settings log.retention.check.interval.ms=1 log.retention.ms=1 Try setting check interval at 1000 ms. I just feel these values are too low. Try setting retention to be 1 hr and keep check interval to default which is 300 seconds. Also just check the permission of th

Re: Delete "kafka-logs"

The reason I set retention time to 10 seconds is because I run kafka for about 4 minutes each time. What I am running is a microbenchmark and I take throughput and latency values for 4mins of runtime. However when I run I get these printed on terminal. [2016-11-23 12:21:54,517] INFO Deleting inde

Re: Delete "kafka-logs"

try bin/kafka-server-start.sh ./config/server.properties Also check how many segments have got created in logs dir and size of each. May be set the retention to 1 hour or 15 minutes. 10 sec seems very less. On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan wrote: > As you suggested I did these

Re: Delete "kafka-logs"

As you suggested I did these changes to the server.properties file on all nodes. log.retention.check.interval.ms=1 log.retention.ms=1 log.segment.bytes=1049 How I start each server is by, /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh -daemon /home/ubuntu/

Re: Delete "kafka-logs"

Check these in server.properties # Log Retention Policy # # The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segme

Re: Delete "kafka-logs"

This is the exact parameter I added to server.config file on all the nodes. log.retention.ms=1000 Thanks, Regards, Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com

Re: Delete "kafka-logs"

I have 4 node cluster and node1 has zookeeper running. I added those parameters on each of my server.properties files in kafka nodes. Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com

Re: Delete "kafka-logs"

Kafka logs are not kept based on data consumption. They are deleted after it reaches a particular size and retention time. Even if they are consumed multiple times in that duration, they are still kept. Where have you added those params and how are you starting your zookeeper and server. On Wed,

Re: Delete "kafka-logs"

Sorry for repeatedly asking the same question, but although I tried all those parameters; my log file still at large about 2.5GB. Is it due to the data inside my topic is flushed to disk? I meant that there might be data inside topic which is not consumed yet or not been deleted even after consumed

Re: Delete "kafka-logs"

Set some value to log.retention.ms which is ms time to retain log and also log.segment.bytes which is size of a single log file. delete.topic.enable is needed if you want to delete the topic itself. Thanks Sachin On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan wrote: > I tried log.retention.byt

Re: Delete "kafka-logs"

I tried log.retention.bytes=1049000 which is about 10MB. But it did not work. Is it because I had not given any value to delete.topic.enable? Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.c

Re: Delete "kafka-logs"

Check http://kafka.apache.org/documentation.html#brokerconfigs log.retention.bytes log.retention.ms log.segment.bytes delete.topic.enable On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan wrote: > Hi all, > > I want to keep the size of kafka-logs reduced as much as possible. To do it > I need to

Delete "kafka-logs"

Hi all, I want to keep the size of kafka-logs reduced as much as possible. To do it I need to set it to delete its unwanted logs frequently. Is there a way to do it? Thanks, Regards, Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B

Delete kafka-logs

Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com