Anup, I realized I forgot to mention this in the previous message, sorry about that:
The additional workarounds to restart the connectors or dynamically reconfigure the log level will only work for MirrorMaker 2.0 running on a regular Connect cluster which has the REST API enabled. The MirrorMaker2 nodes started by the entry script connect-mirror-maker.sh will not have the REST API enabled, and cannot perform these operations. Greg On Wed, Feb 8, 2023 at 10:02 AM Greg Harris <greg.har...@aiven.io> wrote: > Anup, > > Here's the best workaround I can think of: > > I think you can reconfigure the mechanisms which trigger task > reconfiguration with: > * `refresh.topics.enabled` > * `refresh.topics.interval.seconds` > * `refresh.groups.enabled` > * `refresh.groups.interval.seconds` > > Disabling these mechanisms will prevent the errors entirely, but will also > disable these features operating in the background. > As an additional workaround, you can have an external process periodically > restart the MirrorSourceConnector and MirrorCheckpointConnector instances > to force refresh the topics and groups respectively. > This will replace the functionality of the MM2 feature, while still > avoiding the errors. > > A lower effort but generally worse/more dangerous option would be to > temporarily disable logging of errors for the DistributedHerder. > > You can find details on how to do this here: > https://rmoff.net/2019/01/29/kafka-connect-change-log-level-and-write-log-to-file/ > and here > https://rmoff.net/2020/01/16/changing-the-logging-level-for-kafka-connect-dynamically/ > This is not a very good solution, as this will mask other more important > logs from the DistributedHerder that you will certainly be interested in. > But if you're an hour away from a disk filling up, any solution is better > than no solution. > > Good luck! > Greg > > On Wed, Feb 8, 2023 at 3:39 AM Shirolkar, Anup > <anup.shirol...@netapp.com.invalid> wrote: > >> Yes, that makes sense thanks. >> But the side effect of this is there is enormous amount of log generated. >> Is there a quick solution possible to slow down the logs. >> >> Cheers. >> >> From: Greg Harris <greg.har...@aiven.io.INVALID> >> Date: Wednesday, 8 February 2023 at 1:08 pm >> To: users@kafka.apache.org <users@kafka.apache.org> >> Subject: Re: Mirror maker worker can't issue with REST uri >> NetApp Security WARNING: This is an external email. Do not click links or >> open attachments unless you recognize the sender and know the content is >> safe. >> >> >> >> >> Anup, >> >> This is the expected behavior of the MirrorMaker2 application when a >> connector attempts to reconfigure it's tasks. >> It is a limitation of the MirrorMaker2 distributed mode, and has an >> improvement in-progress that I don't believe has been released yet. See >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters >> for more details. >> As a workaround, I believe you can restart the MirrorMaker2 connectors to >> force a reconfiguration. >> >> I hope this helps, >> Greg Harris >> >> On Tue, Feb 7, 2023, 10:47 PM Shirolkar, Anup >> <anup.shirol...@netapp.com.invalid> wrote: >> >> > Hi, >> > >> > I have deployed a 3-node mirror maker cluster version 3.2.1 >> > I have configured the connect-mirror-maker.properties file and started >> the >> > mirror service using connect-mirror-maker.sh >> > >> > It runs fine but one of the three workers always gets below exception. >> > If I restart the connect worker with the error, another worker gets the >> > same error. >> > >> > >> > [2023-02-08 06:11:03,509] ERROR [Worker clientId=connect-2, >> > groupId=ruh-mm2] Request to leader to reconfigure connector tasks failed >> > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1610) >> > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Error >> > trying to forward REST request: Invalid URI host: null (authority: null) >> > at >> > >> org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:147) >> > at >> > >> org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607) >> > at >> > >> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) >> > at >> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> > at >> > >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> > at >> > >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> > at java.base/java.lang.Thread.run(Thread.java:829) >> > Caused by: java.lang.IllegalArgumentException: Invalid URI host: null >> > (authority: null) >> > at >> > org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521) >> > at >> > org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506) >> > at >> > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464) >> > at >> > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453) >> > at >> > >> org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107) >> > ... 6 more >> > >> > What could be wrong here can you please advise. >> > >> > >> > Thanks, >> > Anup Shirolkar. >> > >> > >> > >> > >> >
