Hi, https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html
and https://docs.microsoft.com/en-us/azure/architecture/patterns/claim-check are good source of information. end of the day, my experience Apachr Kafka will perform good with max between 6 and 8mb messages. Is important to know if sporadic or regular, etc. My best recommendation implement this pattern. -- Pere On Thu, 22 Aug 2019, 20:01 l vic <lvic4...@gmail.com> wrote: > Thank you, can you elaborate on "ClaimChecks"? Didn't find it with > search... > Thank you again, > > > On Thu, Aug 22, 2019 at 12:49 AM Pere Urbón Bayes <pere.ur...@gmail.com> > wrote: > > > Hi, > > i agree with Liam, the OOM look to happen during produce time. I would > > look into that. > > > > But with your message size, i would recommend investigating to implement > > ClaimChecks. It will be much easier and reduce the avg message size. > > > > -- Pere > > > > On Thu, 22 Aug 2019, 01:12 Liam Clarke <liam.cla...@adscale.co.nz> > wrote: > > > > > Hi I Vic, > > > > > > Your OOM is happening before any compression is applied. It's occurring > > > when the StringSerializer is converting the string to bytes. Looking > > deeper > > > into StringCoding.encode, it's first allocating a byte array to fit > your > > > string, and this is where your OOM is occurring, line 300 of > > > StringCoding.java is byte[] ba = new byte[en]; > > > > > > Compression is applied after the string is serialized to bytes. So > you'll > > > need to increase your heap size to support this. > > > > > > Hope that helps :) > > > > > > Liam Clarke > > > > > > On Thu, Aug 22, 2019 at 1:52 AM l vic <lvic4...@gmail.com> wrote: > > > > > > > I have to deal with large ( 16M) text messages in my Kafka system, > so i > > > > increased several message limit settings on broker/producer/consumer > > site > > > > and now the system is able to get them through....I also tried to > > enable > > > > compression in producer: > > > > "compression.type"= "gzip" > > > > but to my surprise ended up with OOM exceptions on producer side: > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap > space > > at > > > > java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) at > > > > java.lang.StringCoding.encode(StringCoding.java:344) at > > > > java.lang.String.getBytes(String.java:918) at > > > > > > > > > > > > > > org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:43) > > > > at > > > > > > > > > > > > > > org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:326) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248) > > > > Shouldn't I be able to save memory with compression? Why does the > > > > compression have the opposite effect? > > > > > > > > > >
