Re: getting an error about: java.lang.OutOfMemoryError: Direct buffer memory

2016-11-16 Thread Wei Li
java.lang.OutOfMemoryError is not really necessary directly related with memory usage. In your config, it requests only 1G. If your system is not stressed, I would suggest you to check ulimit for kafka runtime user, particularly check max number of open file descriptor and max number of processes

kafka 0.10 offset storage on zookeeper

2016-11-13 Thread Wei Li
Hi All, We recently upgraded kafka from 0.8 to 0.10 with security features enabled (ssl/sasl/acls). One thing noticed is the offset is not longer kept on zookeeper any more in 0.10 release. Though this is new feature for new version, we like to still keep the offsets stored on zookeeper in short