Re: MONGODB KAFKA CONNECT

2020-07-28 Thread Robin Moffatt
Hi Milka, You've not mentioned in which direction you want to connect MongoDB with Kafka, but either way - Kafka Connect is generally the best approach. You can learn about Kafka Connect here: https://docs.confluent.io/current/connect/ and see a talk about it here: https://rmoff.dev/ljc-kafka-02

Re: Upgrade connectors logging from log4j to log4j2

2020-07-28 Thread Deepak Raghav
Hi Tom Can you please reply to this. Regards and Thanks Deepak Raghav On Mon, Jul 27, 2020 at 10:05 PM Deepak Raghav wrote: > Hi Tom > > I have to change the log level at runtime i.e without restarting the > worker process. > > Can you please share any suggestion on this with log4j. > > Rega

Re: Upgrade connectors logging from log4j to log4j2

2020-07-28 Thread Tom Bentley
Hi Deepak, Sorry, I slightly misunderstood what you were asking. Kafka currently supports log4jv1 out of the box. You've said you've switched to using log4jv2, so I guess you could do this via JMX[1], or having log4j poll for updates to the config file[2]. This is really a log4j question rather th

RecordTooLargeException with old (0.10.0.0) consumer

2020-07-28 Thread Thomas Becker
We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message: org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger

Re: RecordTooLargeException with old (0.10.0.0) consumer

2020-07-28 Thread Manoj.Agrawal2
Hi , You also make to change at producer and consumer side as well server.properties: message.max.bytes=15728640 replica.fetch.max.bytes=15728640 max.request.size=15728640 fetch.message.max.bytes=15728640 and producer.properties: max.request.size=15728640 consumer max.partition.fetch.

Re: RecordTooLargeException with old (0.10.0.0) consumer

2020-07-28 Thread Thomas Becker
Again, we haven't changed the default message size, I believe this exception is a red herring. On Tue, 2020-07-28 at 17:38 +, manoj.agraw...@cognizant.com wrote: [EXTERNAL EMAIL] Attention: This email was sent from outside Xperi. DO NOT CLICK any links or attachments unless you expected the

[VOTE] 2.6.0 RC2

2020-07-28 Thread Randall Hauch
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 2.6.0. This is a major release that includes many new features, including: * TLSv1.3 has been enabled by default for Java 11 or newer. * Smooth scaling out of Kafka Streams applications *

Mirrormaker 2 logs - WARN Catching up to assignment's config offset

2020-07-28 Thread Iftach Ben-Yosef
Hello I'm running a mirrormaker 2 cluster which copies from 3 source clusters into 1 destination. Yesterday I restarted the cluster and it took 1 of the mirrored topics a pretty long time to recover (2~ hours) Since the restart the mm2 cluster has been sending a lot of these warning messages from