Il ven 20 apr 2018, 20:24 Ismael Juma <ism...@juma.me.uk> ha scritto:
> Hi Enrico, > > Coincidentally, I saw your message to nio-dev and followed up there. > I think this is not a 'real' problem, you will notice the difference only if you have a lot of empty topics/partitions. In fact when you simply upgrade to jdk9/10 immediately you are charged with 10MB of disk space for each partition. In production I did not suffer this change because usually you do not have empty partitions. In my test environments, where I had thousands of test empty partitions, disks filled up immediately and without any reason, it took time to understand the real cause. Broker will crash without much 'log' as disk is out of space. Maybe it would be useful to add some notice about this potential problem during the upgrade of the jdk. Hope that helps Enrico > Ismael > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > > > It is a deliberate change in JDK code > > > > Just for reference see this discussion on nio-dev list on OpenJDK > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html > > > > > > see > > https://bugs.openjdk.java.net/browse/JDK-8168628 > > > > Cheers > > Enrico > > > > > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > > > > > Workaround: > > > as these brokers are only for test environments I have set very small > > > values for index file size, which affects pre-allocation > > > segment.index.bytes=65536 > > > log.index.size.max.bytes=65536 > > > > > > If anyone has some thought it will be very appreciated > > > Cheers > > > > > > Enrico > > > > > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > > > > > >> The only fact I have found is that with Java8 Kafka is creating > "SPARSE" > > >> files and with Java9 this is not true anymore > > >> > > >> Enrico > > >> > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > > >> > > >>> Hi, > > >>> This is a very strage case. I have a Kafka broker (part of a cluster > of > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to > > Oracle JDK > > >>> 9.0.4. > > >>> > > >>> There are a lot of .index and .timeindex files taking 10MB, they are > > for > > >>> empty partiions. > > >>> > > >>> Running with Java 9 the server seems to rebuild these files and each > > >>> file takes "really" 10MB. > > >>> The sum of all the files (calculated using du -sh) is 22GB and the > > >>> broker crashes during startup, disk becomes full and no log more is > > >>> written. (I can send an extraction of the logs, but the tell only > > about > > >>> 'rebuilding index', the same as on Java 8) > > >>> > > >>> Reverting the same broker to Java 8 and removing the index files, the > > >>> broker rebuilds such files, each files take 10MB, but the full sum of > > sizes > > >>> (calculated using du -sh) is 38 MB ! > > >>> > > >>> I am running this broker on CentosOS 7 on EXT4 FS. > > >>> > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from > > >>> 0.10.2) without any success. > > >>> > > >>> All of the other testing clusters on CentOS7 (same SO settings) did > not > > >>> have any problem. > > >>> > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on > > >>> Java8 it works fine with 1GB and default max direct memory size) > > >>> > > >>> Which are the relevant configuration options ? > > >>> > > >>> Thoughts ? > > >>> > > >>> Thank you > > >>> > > >>> Enrico > > >>> > > >>> > > >>> > > >>> > > >> > > > > > > -- -- Enrico Olivelli
