FWIW: I went through and removed all the 'custom' serdes from my code and
replaced them with 'string serdes'. The memory leak problem went away.
The code is a bit more cumbersome now as it's constantly flipping back and
forth between Objects and JSON.. but that seems to be what it takes to keep
it
Hello Jon,
It is hard to tell, since I cannot see how is your Aggregate() function is
implemented as well.
Note that the deserializer of transactionSerde is used in both `aggregate`
and `KstreamBuilder.stream`, while the serializer of transactionSerde is
only used in `aggregate`, so if you suspec
I narrowed this problem down to this part of the topology (and yes, it's
100% repro - for me):
KStream transactionKStream =
kStreamBuilder.stream(stringSerde,transactionSerde,TOPIC);
KTable, SumRecordCollector> ktAgg =
transactionKStream.groupByKey().aggregate(
SumRecordCollector::new,
Yes - that's the one. It's 100% reproducible (for me).
On Thu, Dec 22, 2016 at 8:03 AM, Damian Guy wrote:
> Hi Jon,
>
> Is this for the topology where you are doing something like:
>
> topology: kStream -> groupByKey.aggregate(minute) -> foreach
> \-> groupByKey.agg
Hi Jon,
Is this for the topology where you are doing something like:
topology: kStream -> groupByKey.aggregate(minute) -> foreach
\-> groupByKey.aggregate(hour) -> foreach
I'm trying to understand how i could reproduce your problem. I've not seen
any such issues with
Im still hitting this leak with the released version of 0.10.1.1.
Process mem % grows over the course of 10-20 minutes and eventually the OS
kills it.
Messages like this appear in /var/log/messages:
Dec 22 13:31:22 ip-172-16-101-108 kernel: [2989844.793692] java invoked
oom-killer: gfp_mask=0x24
