Hello, We have question regarding transactional producer states and disk space usage.
We did a quick and dirty test recently with 3 Simple Java client producers writing to compressed and topics with compression.type set for both producers and topic correctly. We performed two rounds of tests (with Transactional and Non-Transactional Producers). What we see is that the disk space usage quite high when there are transactional producers. It's true that once the cleanup happens, some of these data is expected to be cleaned up and we can reclaim disk space. But until the files are closed, the disk space usage is quite significant compared to non-transactional. For our test, all our producers ran for 24 minutes. For transactional producers, the kafka log directory apparent-size was 2GB. But for transactional, it was 60 MB. Our disk space alert was fired based on the apparent-size. And once we shutdown all our brokers, the directory sizes changed to 2.2 MB (for txn) and 1.5 MB (for non-txn) Is this expected for transactional producer-based writes? If so, what is the recommendation for optimising producers (e.g. any official docs etc.) ? Regards,
