Anup, Here's the best workaround I can think of:
I think you can reconfigure the mechanisms which trigger task reconfiguration with: * `refresh.topics.enabled` * `refresh.topics.interval.seconds` * `refresh.groups.enabled` * `refresh.groups.interval.seconds` Disabling these mechanisms will prevent the errors entirely, but will also disable these features operating in the background. As an additional workaround, you can have an external process periodically restart the MirrorSourceConnector and MirrorCheckpointConnector instances to force refresh the topics and groups respectively. This will replace the functionality of the MM2 feature, while still avoiding the errors. A lower effort but generally worse/more dangerous option would be to temporarily disable logging of errors for the DistributedHerder. You can find details on how to do this here: https://rmoff.net/2019/01/29/kafka-connect-change-log-level-and-write-log-to-file/ and here https://rmoff.net/2020/01/16/changing-the-logging-level-for-kafka-connect-dynamically/ This is not a very good solution, as this will mask other more important logs from the DistributedHerder that you will certainly be interested in. But if you're an hour away from a disk filling up, any solution is better than no solution. Good luck! Greg On Wed, Feb 8, 2023 at 3:39 AM Shirolkar, Anup <anup.shirol...@netapp.com.invalid> wrote: > Yes, that makes sense thanks. > But the side effect of this is there is enormous amount of log generated. > Is there a quick solution possible to slow down the logs. > > Cheers. > > From: Greg Harris <greg.har...@aiven.io.INVALID> > Date: Wednesday, 8 February 2023 at 1:08 pm > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: Re: Mirror maker worker can't issue with REST uri > NetApp Security WARNING: This is an external email. Do not click links or > open attachments unless you recognize the sender and know the content is > safe. > > > > > Anup, > > This is the expected behavior of the MirrorMaker2 application when a > connector attempts to reconfigure it's tasks. > It is a limitation of the MirrorMaker2 distributed mode, and has an > improvement in-progress that I don't believe has been released yet. See > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters > for more details. > As a workaround, I believe you can restart the MirrorMaker2 connectors to > force a reconfiguration. > > I hope this helps, > Greg Harris > > On Tue, Feb 7, 2023, 10:47 PM Shirolkar, Anup > <anup.shirol...@netapp.com.invalid> wrote: > > > Hi, > > > > I have deployed a 3-node mirror maker cluster version 3.2.1 > > I have configured the connect-mirror-maker.properties file and started > the > > mirror service using connect-mirror-maker.sh > > > > It runs fine but one of the three workers always gets below exception. > > If I restart the connect worker with the error, another worker gets the > > same error. > > > > > > [2023-02-08 06:11:03,509] ERROR [Worker clientId=connect-2, > > groupId=ruh-mm2] Request to leader to reconfigure connector tasks failed > > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1610) > > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Error > > trying to forward REST request: Invalid URI host: null (authority: null) > > at > > > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:147) > > at > > > org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607) > > at > > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > Caused by: java.lang.IllegalArgumentException: Invalid URI host: null > > (authority: null) > > at > > org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521) > > at > > org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506) > > at > > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464) > > at > > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453) > > at > > > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107) > > ... 6 more > > > > What could be wrong here can you please advise. > > > > > > Thanks, > > Anup Shirolkar. > > > > > > > > >
