Broker configs:===========broker.id=36num.network.threads=3num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/var/log/kafkanum.partitions=1num.recovery.threads.per.data.dir=1offsets.topic.replication.factor=1transaction.state.log.replication.factor=1transaction.state.log.min.isr=1log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000zookeeper.connect=myserver1:2181,myserver2:2181,myserver3:2181zookeeper.connection.timeout.ms=6000confluent.support.metrics.enable=trueconfluent.support.customer.id=anonymousgroup.initial.rebalance.delay.ms=0auto.create.topics.enable=false Topic configs:==========--partitions 15--replication-factor 3retention.ms=31449600000retention.bytes=10737418240 As you can see, I have tried to override the retention.bytes for each partition to 10GB to be explicit. 96GB over 10 partitions which 6.4GB. So, I gave myself more than enough buffer. Even then, I am left with no logs. Here's an example: % ls -ltr /var/log/kafka/MyTopic-0total 4-rw-r--r-- 1 root root 14 Jul 17 15:05 leader-epoch-checkpoint-rw-r--r-- 1 root root 10485756 Jul 17 15:05 00000000000005484128.timeindex-rw-r--r-- 1 root root 0 Jul 17 15:05 00000000000005484128.log-rw-r--r-- 1 root root 10485760 Jul 17 15:05 00000000000005484128.index
I kept my eyes on the directory for each partition as the producer was publishing data and I saw periodic .deleted files. Does it mean that Kafka was deleting logs? Any help would be highly appreciated. On Wednesday, July 17, 2019, 01:47:44 PM EDT, Peter Bukowinski <pmb...@gmail.com> wrote: Can you share your broker and topic config here? > On Jul 17, 2019, at 10:09 AM, Sachin Nikumbh <saniku...@yahoo.com.INVALID> > wrote: > > Thanks for the quick response, Tom. > I should have mentioned in my original post that I am always using > --from-beginning with my console consumer. Even then I don't get any data. > And as mentioned, the .log files are of size 0 bytes. > On Wednesday, July 17, 2019, 11:09:22 AM EDT, Thomas Aley ><thomas.a...@ibm.com> wrote: > > Hi Sachin, > > Try adding --from-beginning to your console consumer to view the > historically produced data. By default the console consumer starts from > the last offset. > > Tom Aley > thomas.a...@ibm.com > > > > From: Sachin Nikumbh <saniku...@yahoo.com.INVALID> > To: Kafka Users <users@kafka.apache.org> > Date: 17/07/2019 16:01 > Subject: [EXTERNAL] Kafka logs are getting deleted too soon > > > > Hi all, > I have ~ 96GB of data in files that I am trying to get into a Kafka > cluster. I have ~ 11000 keys for the data and I have created 15 partitions > for my topic. While my producer is dumping data in Kafka, I have a console > consumer that shows me that kafka is getting the data. The producer runs > for a few hours before it is done. However, at this point, when I run the > console consumer, it does not fetch any data. If I look at the logs > directory, .log files for all the partitions are of 0 byte size. > If I am not wrong, the default value for log.retention.bytes is -1 which > means there is no size limit for the logs/partition. I do want to make > sure that the value for this setting is per partition. Given that the > default time based retention is 7 days, I am failing to understand why the > logs are getting deleted. The other thing that confuses me is that when I > use kafka.tools.GetOffsetShell, it shows me large enough values for all > the 15 partitions for offsets. > Can someone please help me understand why I don't see logs and why > is kafka.tools.GetOffsetShell making me believe there is data. > ThanksSachin > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >