Hi Divya, Something that we've found useful in the past is to have a secondary key-value cache that can store large amounts of data under a key, then pass the key through Kafka for the consumer to use for retrieval on the other end. Usually leads to much better performance from Kafka's perspective.
Cheers, Malcolm McFarland Cavulus On Fri, Jul 1, 2022 at 1:13 PM Divya Jain <truedeeds.divya1...@gmail.com> wrote: > Hi, > > It's the max that could be defined. It cannot go beyond it. I am not sure > how to solve this. > > Thanks > Divya Jain > > > On Sat, 2 Jul, 2022, 1:40 am M. Manna, <manme...@gmail.com> wrote: > > > Hi, you need to increase record and message size because your real > message > > payload is bigger than what’s mention in properties file. > > > > Regards, > > > > On Fri, 1 Jul 2022 at 20:24, Divya Jain <truedeeds.divya1...@gmail.com> > > wrote: > > > > > Hi, > > > > > > I am facing this issue: > > > 2022-07-01 19:01:05,548] INFO Topic 'postgres.public.content_history' > > > already exists. (org.apache.kafka.connect.runtime.WorkerSourceTask:423) > > > [2022-07-01 19:01:05,641] INFO > > > WorkerSourceTask{id=smtip-de-content2-source-connector-0} Committing > > > offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:488) > > > [2022-07-01 19:01:05,642] INFO > > > WorkerSourceTask{id=smtip-de-content2-source-connector-0} flushing 155 > > > outstanding messages for offset commit > > > (org.apache.kafka.connect.runtime.WorkerSourceTask:505) > > > [2022-07-01 19:01:06,034] INFO > > > WorkerSourceTask{id=smtip-de-content2-source-connector-0} Finished > > > commitOffsets successfully in 393 ms > > > (org.apache.kafka.connect.runtime.WorkerSourceTask:586) > > > [2022-07-01 19:01:06,575] WARN [Producer > > > clientId=connector-producer-smtip-de-content2-source-connector-0] Got > > error > > > produce response in correlation id 610 on topic-partition > > > postgres.public.content_history-1, splitting and retrying (2147483647 > > > attempts left). Error: MESSAGE_TOO_LARGE > > > (org.apache.kafka.clients.producer.internals.Sender:582) > > > [2022-07-01 19:01:06,843] ERROR > > > WorkerSourceTask{id=smtip-de-content2-source-connector-0} failed to > send > > > record to postgres.public.content_history: > > > (org.apache.kafka.connect.runtime.WorkerSourceTask:372) > > > org.apache.kafka.common.errors.RecordTooLargeException: The request > > > included a message larger than the max message size the server will > > accept. > > > [2022-07-01 19:01:06,927] WARN [Producer > > > clientId=connector-producer-smtip-de-content2-source-connector-0] Got > > error > > > produce response in correlation id 643 on topic-partition > > > postgres.public.content_history-0, splitting and retrying (2147483647 > > > attempts left). Error: MESSAGE_TOO_LARGE > > > (org.apache.kafka.clients.producer.internals.Sender:582) > > > [2022-07-01 19:01:06,941] INFO The task will send records to topic > > > 'postgres.public.content_attributes' for the first time. Checking > whether > > > topic exists (org.apache.kafka.connect.runtime.WorkerSourceTask:419) > > > [2022-07-01 19:01:06,943] INFO Topic > 'postgres.public.content_attributes' > > > already exists. (org.apache.kafka.connect.runtime.WorkerSourceTask:423) > > > > > > > > > The properties I am using in my worker.properties are: I have tried > > > combination of properties I am not sure how to fix it. Please guide on > > the > > > same. > > > > > > offset.flush.timeout.ms=60000 > > > > > > offset.flush.interval.ms=5000 > > > > > > max.partition.fetch.bytes=2147483647 > > > > > > max.request.size=2147483647 > > > > > > max.message.bytes=2147483647 > > > > > > message.max.bytes=2147483647 > > > > > > replica.fetch.max.bytes=2147483647 > > > > > > producer.security.protocol=SSL > > > > > > producer.max.request.size=2147483647 > > > > > > connector.client.config.override.policy=All > > > > > > Thanks & Regards > > > Divya Jain > > > > > >