I've had similar errors cause by mmap counts; try with vm.max_map_count=262144
On 2022/09/01 23:57:54 Paul Brebner wrote: > Hi all, > > I've been attempting to benchmark Kafka KRaft version for an ApacheCon talk > and have identified 2 problems: > > 1 - it's still impossible to create large number of partitions/topics - I > can create more than the comparable Zookeeper version but still not > "millions" - this is with RF=1 (as anything higher needs huge clusters to > cope with the replication CPU overhead) only, and no load on the clusters > yet (i.e. purely a topic/partition creation experiment). > > 2 - eventually the topic/partition creation command causes the Kafka > process to fail - looks like a memory error - > > java.lang.OutOfMemoryError: Metaspace > OpenJDK 64-Bit Server VM warning: INFO: > os::commit_memory(0x00007f4f554f9000, 65536, 1) failed; error='Not enough > space' (errno=12) > > or similar error > > seems to happen consistently around 30,000+ partitions - this is on a test > EC2 instance with 32GB Ram, 500,000 file descriptors (increased from > default) and 64GB disk (plenty spare). I'm not an OS expert, but the kafka > process and the OS both seem to have plenty of RAM when this error occurs. > > So there's 3 questions really: What's going wrong exactly? How to achieve > more partitions? And should the topic create command (just using the CLI at > present to create topics) really be capable of killing the Kafka instance, > or should it fail and throw an error, and the Kafka instance still continue > working... > > Regards, Paul Brebner >
