looking for kafka mockup framework

2018-08-22 Thread l vic
Hello, I need to unit-test java code that should be able to subscribe to kafka topic and asynchronously receive messages as input... Can anybody recommend junit mockup framework for doing that? Thank you,

RecordTooLargeException on 16M messages in Kafka?

2019-08-14 Thread l vic
My kafka (1.0.0) producer errors out on large (16M) messages. ERROR Error when sending message to topic test with key: null, value: 16777239 bytes with error: (org.apache.kafka.clients.producer.internals. ErrorLoggingCallback) org.apache.kafka.common.errors.RecordTooLargeException: The message is

Re: RecordTooLargeException on 16M messages in Kafka?

2019-08-15 Thread l vic
Yes, it's still there On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli < jonathansanti...@gmail.com> wrote: > Hello, try to send and flush just one message of 16777239 bytes, to verify > the error still shows up. > > Cheers! > -- > Jonathan > > > > On

Re: RecordTooLargeException on 16M messages in Kafka?

2019-08-15 Thread l vic
gt; wrote: > Hello, try to send and flush just one message of 16777239 bytes, to verify > the error still shows up. > > Cheers! > -- > Jonathan > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic wrote: > > > My kafka (1.0.0) producer errors out on large (16M) messages. &g

Re: RecordTooLargeException on 16M messages in Kafka?

2019-08-15 Thread l vic
39 bytes, to verify > the error still shows up. > > Cheers! > -- > Jonathan > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic wrote: > > > My kafka (1.0.0) producer errors out on large (16M) messages. > > ERROR Error when sending message to topic tes

Re: RecordTooLargeException on 16M messages in Kafka?

2019-08-15 Thread l vic
Cheers! > -- > Jonathan > > > > > On Thu, Aug 15, 2019 at 1:44 PM l vic wrote: > > > I tested it with kafka-console-consumer and kafka-console-producer > reading > > from large text file (no newlines): > > > > kafka-console-prod

Re: RecordTooLargeException on 16M messages in Kafka?

2019-08-15 Thread l vic
previous email when you > tried the console producer. > > Jonathan. > > On Thu, Aug 15, 2019, 3:07 PM l vic wrote: > > > yes, in producer.properties > > > > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli < > > jonathansanti...@gmail.com> wrote: &g

OOM for large messages with compression?

2019-08-21 Thread l vic
I have to deal with large ( 16M) text messages in my Kafka system, so i increased several message limit settings on broker/producer/consumer site and now the system is able to get them throughI also tried to enable compression in producer: "compression.type"= "gzip" but to my surprise ended up

Re: OOM for large messages with compression?

2019-08-22 Thread l vic
Coding.java is byte[] ba = new byte[en]; > > > > Compression is applied after the string is serialized to bytes. So you'll > > need to increase your heap size to support this. > > > > Hope that helps :) > > > > Liam Clarke > > > > On