RE: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-31 Thread Kyle Tinker
what unit tests you would want around this change? - Kyle -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael Juma Sent: Wednesday, January 31, 2018 6:35 PM To: dev Subject: Re: Excessive Memory Usage with Compression enabled and possible resolutions

Re: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-31 Thread Ismael Juma
u need > any additional information. I'm new to this project so hopefully the > format is what you were looking for. > > - Kyle > > -Original Message- > From: Ismael Juma [mailto:isma...@gmail.com] > Sent: Tuesday, January 30, 2018 9:01 PM > To: dev > Subject: Re:

RE: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-31 Thread Kyle Tinker
ssage- From: Ismael Juma [mailto:isma...@gmail.com] Sent: Tuesday, January 30, 2018 9:01 PM To: dev Subject: Re: Excessive Memory Usage with Compression enabled and possible resolutions Thanks for the report. I haven't looked at the code, but it seems like we would want to do both 1 and

Re: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-30 Thread Ismael Juma
Thanks for the report. I haven't looked at the code, but it seems like we would want to do both 1 and 2. Can you please file a JIRA with 1.1.0 as the target version? Ismael On 30 Jan 2018 5:46 pm, "Kyle Tinker" wrote: > I'm using Kafka 1.0.0 and the Java producer. > > I've noticed high memory u

Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-30 Thread Kyle Tinker
I'm using Kafka 1.0.0 and the Java producer. I've noticed high memory usage in the producer when enabling compression (gzip or lz4). I don't observe the behavior with compression off, but with it on I'll run out of heap (2GB). Using a Java profiler, I see the data is in the KafkaLZ4BlockOutpu