Workaround: as these brokers are only for test environments I have set very small values for index file size, which affects pre-allocation segment.index.bytes=65536 log.index.size.max.bytes=65536
If anyone has some thought it will be very appreciated Cheers Enrico 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > The only fact I have found is that with Java8 Kafka is creating "SPARSE" > files and with Java9 this is not true anymore > > Enrico > > 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > >> Hi, >> This is a very strage case. I have a Kafka broker (part of a cluster of 3 >> brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK >> 9.0.4. >> >> There are a lot of .index and .timeindex files taking 10MB, they are for >> empty partiions. >> >> Running with Java 9 the server seems to rebuild these files and each file >> takes "really" 10MB. >> The sum of all the files (calculated using du -sh) is 22GB and the broker >> crashes during startup, disk becomes full and no log more is written. (I >> can send an extraction of the logs, but the tell only about 'rebuilding >> index', the same as on Java 8) >> >> Reverting the same broker to Java 8 and removing the index files, the >> broker rebuilds such files, each files take 10MB, but the full sum of sizes >> (calculated using du -sh) is 38 MB ! >> >> I am running this broker on CentosOS 7 on EXT4 FS. >> >> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from >> 0.10.2) without any success. >> >> All of the other testing clusters on CentOS7 (same SO settings) did not >> have any problem. >> >> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8 >> it works fine with 1GB and default max direct memory size) >> >> Which are the relevant configuration options ? >> >> Thoughts ? >> >> Thank you >> >> Enrico >> >> >> >> >
