As you suggested I did these changes to the server.properties file on all nodes.
log.retention.check.interval.ms=10000 log.retention.ms=10000 log.segment.bytes=10490000 How I start each server is by, /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh -daemon /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/server.properties However, I still get about 4GB log files. Did I miss anything? Thanks, Regards, Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com <osanda.des...@gmail.com> <https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan> <https://www.linkedin.com/in/erangaheshan> On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > Check these in server.properties > ############################# Log Retention Policy > ############################# > > # The following configurations control the disposal of log segments. The > policy can > # be set to delete segments after a period of time, or after a given size > has accumulated. > # A segment will be deleted whenever *either* of these criteria are met. > Deletion always happens > # from the end of the log. > > # The minimum age of a log file to be eligible for deletion > log.retention.hours=168 > > # A size-based retention policy for logs. Segments are pruned from the log > as long as the remaining > # segments don't drop below log.retention.bytes. > #log.retention.bytes=1073741824 > > # The maximum size of a log segment file. When this size is reached a new > log segment will be created. > log.segment.bytes=1073741824 > > Try by setting hours as 1 and set segment bytes as say 10 MB. You should be > able to see multiple segments created of size 20 MB under you kafka logs > dir > say: tmp\kafka-logs\<topic>\*.log > > As the files age > 1 hour it will be marked for deletion and be deleted > next time delete thread runs. By default the delete thread check is 5 > minutes. > > Also make sure when you start server you specify this property as the > argument. > > bin\windows\kafka-server-start.bat config\server.properties > > > > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan <eranga....@gmail.com> > wrote: > > > This is the exact parameter I added to server.config file on all the > nodes. > > > > log.retention.ms=1000 > > > > Thanks, > > Regards, > > > > > > Eranga Heshan > > *Undergraduate* > > Computer Science & Engineering > > University of Moratuwa > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > Email: era...@wso2.com <osanda.des...@gmail.com> > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > erangaheshan> > > <https://www.linkedin.com/in/erangaheshan> > > > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan <eranga....@gmail.com> > > wrote: > > > > > I have 4 node cluster and node1 has zookeeper running. I added those > > > parameters on each of my server.properties files in kafka nodes. > > > > > > > > > Eranga Heshan > > > *Undergraduate* > > > Computer Science & Engineering > > > University of Moratuwa > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > <https://www.facebook.com/erangaheshan> > > > <https://twitter.com/erangaheshan> > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal <sjmit...@gmail.com> > > wrote: > > > > > >> Kafka logs are not kept based on data consumption. They are deleted > > after > > >> it reaches a particular size and retention time. > > >> Even if they are consumed multiple times in that duration, they are > > still > > >> kept. > > >> > > >> Where have you added those params and how are you starting your > > zookeeper > > >> and server. > > >> > > >> > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan <eranga....@gmail.com> > > >> wrote: > > >> > > >> > Sorry for repeatedly asking the same question, but although I tried > > all > > >> > those parameters; my log file still at large about 2.5GB. Is it due > to > > >> the > > >> > data inside my topic is flushed to disk? I meant that there might be > > >> data > > >> > inside topic which is not consumed yet or not been deleted even > after > > >> > consumed. > > >> > > > >> > Thanks, > > >> > Regards, > > >> > > > >> > > > >> > Eranga Heshan > > >> > *Undergraduate* > > >> > Computer Science & Engineering > > >> > University of Moratuwa > > >> > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > >> > Email: era...@wso2.com <osanda.des...@gmail.com> > > >> > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > >> > erangaheshan> > > >> > <https://www.linkedin.com/in/erangaheshan> > > >> > > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal <sjmit...@gmail.com> > > >> wrote: > > >> > > > >> > > Set some value to log.retention.ms which is ms time to retain log > > and > > >> > also > > >> > > log.segment.bytes which is size of a single log file. > > >> > > delete.topic.enable is needed if you want to delete the topic > > itself. > > >> > > > > >> > > Thanks > > >> > > Sachin > > >> > > > > >> > > > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan < > > eranga....@gmail.com> > > >> > > wrote: > > >> > > > > >> > > > I tried log.retention.bytes=1049000 which is about 10MB. But it > > did > > >> not > > >> > > > work. Is it because I had not given any value to > > >> delete.topic.enable? > > >> > > > > > >> > > > > > >> > > > Eranga Heshan > > >> > > > *Undergraduate* > > >> > > > Computer Science & Engineering > > >> > > > University of Moratuwa > > >> > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > >> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > >> > > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > >> > > > erangaheshan> > > >> > > > <https://www.linkedin.com/in/erangaheshan> > > >> > > > > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal < > > sjmit...@gmail.com> > > >> > > wrote: > > >> > > > > > >> > > > > Check > > >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs > > >> > > > > log.retention.bytes > > >> > > > > log.retention.ms > > >> > > > > log.segment.bytes > > >> > > > > delete.topic.enable > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan < > > >> eranga....@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Hi all, > > >> > > > > > > > >> > > > > > I want to keep the size of kafka-logs reduced as much as > > >> possible. > > >> > To > > >> > > > do > > >> > > > > it > > >> > > > > > I need to set it to delete its unwanted logs frequently. Is > > >> there a > > >> > > way > > >> > > > > to > > >> > > > > > do it? > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Regards, > > >> > > > > > > > >> > > > > > Eranga Heshan > > >> > > > > > *Undergraduate* > > >> > > > > > Computer Science & Engineering > > >> > > > > > University of Moratuwa > > >> > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > >> > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > >> > > > > > <https://www.facebook.com/erangaheshan> < > > https://twitter.com/ > > >> > > > > > erangaheshan> > > >> > > > > > <https://www.linkedin.com/in/erangaheshan> > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
