RE: OutOfMemory error on broker when rolling logs

2015-05-15 Thread Jeff Field
From: Jeff Field Sent: Thursday, May 14, 2015 11:17 AM To: 'users@kafka.apache.org' Subject: RE: OutOfMemory error on broker when rolling logs Replying to Jay's message, though some mailing list snafu made me not able to see it except through the archive. Apologies if it brea

RE: OutOfMemory error on broker when rolling logs

2015-05-14 Thread Jeff Field
ame options you're running kafka with it should succeed but if it fails with the same error that is the address space limit for 32 bits kicking in. -Jay From: Jeff Field Sent: Wednesday, May 13, 2015 4:25 PM To: users@kafka.apache.org Subject: OutOfMemory error on broker when rolling logs Hell

Re: OutOfMemory error on broker when rolling logs

2015-05-13 Thread Jay Kreps
I think "java.lang.OutOfMemoryError: Map failed" has usually been "out of address space for mmap" if memory serves. If you sum the length of all .index files while the service is running (not after stopped), do they sum to something really close to 2GB? If so it is likely either that the OS/arch i

OutOfMemory error on broker when rolling logs

2015-05-13 Thread Jeff Field
Hello, We are doing a Kafka POC on our CDH cluster. We are running 3 brokers with 24TB (48TB Raw) of available RAID10 storage (XFS filesystem mounted with nobarrier/largeio) (HP Smart Array P420i for the controller, latest firmware) and 48GB of RAM. The broker is running with "-Xmx4G -Xms4G -ser