bootstrap.servers=b-1:9092,b-2:9092 group.id=connect-cluster key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=true offset.storage.topic=connect-offsets-2 offset.storage.replication.factor=2 #offset.storage.partitions=25 config.storage.topic=connect-configs-2 config.storage.replication.factor=2 status.storage.topic=connect-status-2 status.storage.replication.factor=2 #status.storage.partitions=5 offset.flush.interval.ms=10000 #rest.host.name= #rest.port=8083 rest.advertised.host.name=172.16.234.122 rest.advertised.port=8083 plugin.path=/apps/libs/streamthoughts-kafka-connect-file-pulse-1.6.0/ consumer.max.poll.records=100 consumer.max.poll.interval.ms=600000
These are the connect-distributed.properties, I am using. I have changed the last two fields but still having the same issue. is it related to the source connecter(file pulse in my case). It is scanning around 20K files and stuck. With smaller scan directory, it is properly running. On Mon, Apr 5, 2021 at 2:52 PM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Himanshu, > > Have you adjusted your consumer properties as the error message suggested? > > Alternatively reduce your your consumer.max.poll.records in the worker > config. > > Basically, the sink you're using is spending too much time processing in > the poll loop, so either tweak the properties as mentioned in the error > message, or reduce the number of records processed in a batch so that it > doesn't hit that time out. > > If you have adjusted these properties, and still have issues, please > respond with your current worker properties to make it easier to debug. > > Please note that for any KC sink or source connector, response times from > the underlying data source/store can impact performance, so you may also > need to look into that aspect also. > > Cheers, > > Liam Clarke-Hutchinson > > On Thu, 1 Apr. 2021, 5:26 pm Himanshu Shukla, <himanshushukla...@gmail.com > > > wrote: > > > Hi, > > I am using kafka-connect-file-pulse connector and scanning around 20K > > files. After the scan step, the whole connect cluster is becoming > > unresponsive. I can not even access localhost:8083/connectors/ URL. It is > > giving request timeout. > > > > > > I have observed the below errors from the connect logs. Did anyone face > > this issue? > > > > Please advise if I am doing something wrong. > > > > > > [2021-03-31 16:21:58,920] INFO Scanning local file system directory > > '/apps/datafiles_1/cm_dir/QA1/' > > > > > (io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner:241) > > [2021-03-31 16:22:57,586] WARN [Worker clientId=connect-1, > > groupId=connect-cluster] This member will leave the group because > consumer > > poll timeout has expired. This means the time between subsequent calls to > > poll() was longer than the configured max.poll.interval.ms, which > > typically > > implies that the poll loop is spending too much time processing messages. > > You can address this either by increasing max.poll.interval.ms or by > > reducing the maximum size of batches returned in poll() with > > max.poll.records. > > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1051) > > [2021-03-31 16:22:57,586] INFO [Worker clientId=connect-1, > > groupId=connect-cluster] *Member > > connect-1-064cf0bf-b834-40d2-9e72-e61b229157c4 sending LeaveGroup request > > to coordinator URL:9092* (id: 2147483646 rack: null) > > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:822) > > [2021-03-31 16:23:24,562] ERROR Request to leader to reconfigure > connector > > tasks failed > > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1037) > > > > > > *org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: > Request > > timed out* > > at > > > > > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97) > > at > > > > > org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > [2021-03-31 16:23:24,562] ERROR* Failed to reconfigure connector's tasks, > > retrying after backoff: > > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:958)* > > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: > Request > > timed out > > at > > > > > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97) > > at > > > > > org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51 > > > > -- > > Regards, > > Himanshu Shukla > > > -- Regards, Himanshu Shukla
