Did anyone face it before? The connector URL is giving 500 request time out.
On Thu, Apr 1, 2021 at 9:55 AM Himanshu Shukla <himanshushukla...@gmail.com> wrote: > Hi, > I am using kafka-connect-file-pulse connector and scanning around 20K > files. After the scan step, the whole connect cluster is becoming > unresponsive. I can not even access localhost:8083/connectors/ URL. It is > giving request timeout. > > > I have observed the below errors from the connect logs. Did anyone face > this issue? > > Please advise if I am doing something wrong. > > > [2021-03-31 16:21:58,920] INFO Scanning local file system directory > '/apps/datafiles_1/cm_dir/QA1/' > (io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner:241) > [2021-03-31 16:22:57,586] WARN [Worker clientId=connect-1, > groupId=connect-cluster] This member will leave the group because consumer > poll timeout has expired. This means the time between subsequent calls to > poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time processing > messages. You can address this either by increasing max.poll.interval.ms > or by reducing the maximum size of batches returned in poll() with > max.poll.records. > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1051) > [2021-03-31 16:22:57,586] INFO [Worker clientId=connect-1, > groupId=connect-cluster] *Member > connect-1-064cf0bf-b834-40d2-9e72-e61b229157c4 sending LeaveGroup request > to coordinator URL:9092* (id: 2147483646 rack: null) > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:822) > [2021-03-31 16:23:24,562] ERROR Request to leader to reconfigure connector > tasks failed > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1037) > > > *org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: > Request timed out* > at > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [2021-03-31 16:23:24,562] ERROR* Failed to reconfigure connector's tasks, > retrying after backoff: > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:958)* > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request > timed out > at > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51 > > -- > Regards, > Himanshu Shukla > > > -- Regards, Himanshu Shukla
