Are you running the console consumer with the ‘--from-beginning’ option? It defaults to reading from tail of the log, so if there is nothing being produced it will be idle.
-- Peter (from phone) > On Jul 17, 2019, at 8:00 AM, Sachin Nikumbh <saniku...@yahoo.com.invalid> > wrote: > > Hi all, > I have ~ 96GB of data in files that I am trying to get into a Kafka cluster. > I have ~ 11000 keys for the data and I have created 15 partitions for my > topic. While my producer is dumping data in Kafka, I have a console consumer > that shows me that kafka is getting the data. The producer runs for a few > hours before it is done. However, at this point, when I run the console > consumer, it does not fetch any data. If I look at the logs directory, .log > files for all the partitions are of 0 byte size. > If I am not wrong, the default value for log.retention.bytes is -1 which > means there is no size limit for the logs/partition. I do want to make sure > that the value for this setting is per partition. Given that the default time > based retention is 7 days, I am failing to understand why the logs are > getting deleted. The other thing that confuses me is that when I use > kafka.tools.GetOffsetShell, it shows me large enough values for all the 15 > partitions for offsets. > Can someone please help me understand why I don't see logs and why is > kafka.tools.GetOffsetShell making me believe there is data. > ThanksSachin
