Hi,
Nothing interesting on Kafka side.Just sone partition delete/create logs.
Also I can't understand why all task managers stop at the same time without
any error log.
Thanks
Kenan
On Sun, Aug 20, 2023 at 10:49 AM liu ron <ron9....@gmail.com> wrote:
> Hi,
>
> Maybe you need to check what changed on the Kafka side at that time.
>
> Best,
> Ron
>
> Kenan Kılıçtepe <kkilict...@gmail.com> 于2023年8月20日周日 08:51写道：
>
>> Hi,
>>
>> I have 4 task manager working on 4 servers.
>> They all crush at the same time without any useful error logs.
>> Only log I can see is some disconnection from Kafka for both consumer and
>> producers.
>> Any idea or any help is appreciated.
>>
>> Some logs from all taskmanagers:
>>
>> I think first server 4 is crushing and it causes crush for all
>> taskmanagers.
>>
>> JobManager:
>>
>> 2023-08-18 15:16:46,528 INFO org.apache.kafka.clients.NetworkClient
>> [] - [AdminClient clientId=47539-enumerator-admin-client]
>> Node 2 disconnected.
>> 2023-08-18 15:19:00,303 INFO org.apache.kafka.clients.NetworkClient
>> [] - [AdminClient
>> clientId=tf_25464-enumerator-admin-client] Node 4 disconnected.
>> 2023-08-18 15:19:16,668 INFO org.apache.kafka.clients.NetworkClient
>> [] - [AdminClient
>> clientId=cpu_59942-enumerator-admin-client] Node 1 disconnected.
>> 2023-08-18 15:19:16,764 INFO org.apache.kafka.clients.NetworkClient
>> [] - [AdminClient
>> clientId=cpu_55128-enumerator-admin-client] Node 3 disconnected.
>> 2023-08-18 15:19:27,913 WARN akka.remote.transport.netty.NettyTransport
>> [] - Remote connection to [/10.11.0.51:42778] failed
>> with java.io.IOException: Connection reset by peer
>> 2023-08-18 15:19:27,963 WARN akka.remote.ReliableDeliverySupervisor
>> [] - Association with remote system
>> [akka.tcp://flink@tef-prod-flink-04:38835] has failed, address is now
>> gated for [50] ms. Reason: [Disassociated]
>> 2023-08-18 15:19:27,967 WARN akka.remote.ReliableDeliverySupervisor
>> [] - Association with remote system
>> [akka.tcp://flink-metrics@tef-prod-flink-04:46491] has failed, address
>> is now gated for [50] ms. Reason: [Disassociated]
>> 2023-08-18 15:19:29,225 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
>> RouterReplacementAlgorithm -> kafkaSink_sinkFaultyRouter_windowMode: Writer
>> -> kafkaSink_sinkFaultyRouter_windowMode: Committer (3/4)
>> (f6fd65e3fc049bd9021093d8f532bbaf_a47f4a3b960228021159de8de51dbb1f_2_0)
>> switched from RUNNING to FAILED on
>> injection-assia-3-pro-cloud-tef-gcp-europe-west1:39011-b24b1d @
>> injection-assia-3-pro-cloud-tef-gcp-europe-west1 (dataPort=35223).
>> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
>> Connection unexpectedly closed by remote task manager 'tef-prod-flink-04/
>> 10.11.0.51:37505 [ tef-prod-flink-04:38835-e3ca4d ] '. This might
>> indicate that the remote task manager was lost.
>> at
>> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:134)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:94)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at java.lang.Thread.run(Thread.java:829) ~[?:?]
>>
>>
>> 2023-08-18 15:19:29,225 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
>> GroupAggregate[50] (1/1)
>> (7afc36d5660c8070d4d1e426428904a3_b2294522dce77da76437a18d2de62f66_0_0)
>> switched from RUNNING to FAILED on
>> injection-assia-3-pro-cloud-tef-gcp-europe-west1:39011-b24b1d @
>> injection-assia-3-pro-cloud-tef-gcp-europe-west1 (dataPort=35223).
>> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
>> Connection unexpectedly closed by remote task manager 'tef-prod-flink-04/
>> 10.11.0.51:37505 [ tef-prod-flink-04:38835-e3ca4d ] '. This might
>> indicate that the remote task manager was lost.
>> at
>> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:134)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:94)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> ~[flink-dist-1.16.2.jar:1.16.2]
>> at java.lang.Thread.run(Thread.java:829) ~[?:?]
>> 2023-08-18 15:19:29,294 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster [] - 16 tasks
>> will be restarted to recover the failed task
>> f6fd65e3fc049bd9021093d8f532bbaf_a47f4a3b960228021159de8de51dbb1f_2_0.
>> 2023-08-18 15:19:29,294 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster [] - 114
>> tasks will be restarted to recover the failed task
>> 7afc36d5660c8070d4d1e426428904a3_b2294522dce77da76437a18d2de62f66_0_0.
>> 2023-08-18 15:19:29,294 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
>> Health Algorithms V1.2 IntervalMin:180 EmbeddedRocksDBStateBackend x4
>> (8be8bada21ee7859c596deb49f29bbfd) switched from state RUNNING to
>> RESTARTING.
>> 2023-08-18 15:19:29,294 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
>> Temperature Algorithms V1.2 IntervalMin:180 EmbeddedRocksDBStateBackend x4
>> (f6b3823cc5b0d9462f09ca7e21a468b2) switched from state RUNNING to
>> RESTARTING.
>>
>>
>>
>> Server1
>>
>>
>> 2023-08-18 15:18:39,898 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-102] Resetting the last
>> seen epoch of partition alerts.wifiqos-14 to 0 since the associated topicId
>> changed from null to hGzbb6Q9R-eApP_OwyFNfw
>> 2023-08-18 15:18:39,898 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-102] Resetting the last
>> seen epoch of partition alerts.wifiqos-4 to 0 since the associated topicId
>> changed from null to hGzbb6Q9R-eApP_OwyFNfw
>> 2023-08-18 15:18:48,568 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-36] Node 4 disconnected.
>> 2023-08-18 15:18:49,864 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-66] Node 2 disconnected.
>> 2023-08-18 15:18:49,912 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-67] Node 2 disconnected.
>> 2023-08-18 15:18:50,156 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-64] Node 4 disconnected.
>> 2023-08-18 15:19:01,240 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-105] Node 4 disconnected.
>> 2023-08-18 15:19:17,252 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-41] Node 1 disconnected.
>> 2023-08-18 15:19:27,612 ERROR
>> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue [] -
>> Encountered error while consuming partitions
>> org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
>> readAddress(..) failed: Connection reset by peer
>> 2023-08-18 15:19:28,053 INFO
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend [] -
>> Closed RocksDB State Backend. Cleaning up RocksDB working directory
>> /home/assia/data2/flink1/job_9f2d6dc13df39c9f5ed1a0fe69e5deae_op_KeyedProcessOperator_0456b63cf4d1645b74bf4c5eed8d03f0__1_4__uuid_bbb68a27-a502-4fa9-b698-2894a4f6c8f0.
>> 2023-08-18 15:19:28,053 INFO
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend [] -
>> Closed RocksDB State Backend. Cleaning up RocksDB working directory
>> /home/assia/data1/flink3/job_9f2d6dc13df39c9f5ed1a0fe69e5deae_op_KeyedProcessOperator_b3fea86a6e448bb11ab554a395c1d49f__1_4__uuid_db93d58c-86ad-4985-b6d7-8dfa5a223628.
>> 2023-08-18 15:19:28,053 INFO
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend [] -
>> Closed RocksDB State Backend. Cleaning up RocksDB working directory
>> /home/assia/data3/flink1/job_b21089450115e102d5f8502110fd8b41_op_SlicingWindowOperator_28af45bb9be1c3c8edc302ec9b4b2ee1__15_18__uuid_6750135d-1896-4083-9016-56e782cff254.
>> 2023-08-18 15:19:28,053 INFO
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend [] -
>> Closed RocksDB State Backend. Cleaning up RocksDB working directory
>> /home/assia/data1/flink2/job_9f2d6dc13df39c9f5ed1a0fe69e5deae_op_KeyedProcessOperator_29b51ba77663f1dcf21f382adc455811__1_4__uuid_04368858-c202-41f2-a4d9-788dfe0f6e3c.
>> 2023-08-18 15:19:28,053 INFO
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend [] -
>> Closed RocksDB State Backend. Cleaning up RocksDB working directory
>> /home/assia/data3/flink2/job_b21089450115e102d5f8502110fd8b41_op_SlicingWindowOperator_28af45bb9be1c3c8edc302ec9b4b2ee1__6_18__uuid_62ca2b18-560a-45ae-887e-b3060abe58d3.
>>
>> ...
>> ...
>> ...
>>
>> 2023-08-18 15:19:29,550 WARN org.apache.flink.runtime.taskmanager.Task
>> [] - WifiStationAndInterfaceQoECalculator ->
>> Timestamps/Watermarks (6/18)#0
>> (f003b7bae04aeade2276538c66237dfe_547178fd75c00c015d9d73f02082a7cd_5_0)
>> switched from RUNNING to FAILED with failure cause:
>> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
>> Lost connection to task manager 'tef-prod-flink-04/10.11.0.51:37505 [
>> tef-prod-flink-04:38835-e3ca4d ] '. This indicates that the remote task
>> manager was lost.
>> at
>> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.exceptionCaught(CreditBasedPartitionRequestClientHandler.java:165)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:821)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> Caused by:
>> org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
>> readAddress(..) failed: Connection reset by peer
>>
>> 2023-08-18 15:19:29,550 WARN org.apache.flink.runtime.taskmanager.Task
>> [] - GroupAggregate[18] -> ConstraintEnforcer[19] ->
>> TableToDataSteam (6/18)#0
>> (f003b7bae04aeade2276538c66237dfe_2a967f37b689e0cf1eb34feac9c5b6c3_5_0)
>> switched from RUNNING to FAILED with failure cause:
>> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
>> Lost connection to task manager 'tef-prod-flink-04/10.11.0.51:37505 [
>> tef-prod-flink-04:38835-e3ca4d ] '. This indicates that the remote task
>> manager was lost.
>> at
>> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.exceptionCaught(CreditBasedPartitionRequestClientHandler.java:165)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:821)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
>> at
>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
>> at
>> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>>
>>
>>
>> Server 4:
>>
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-10 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-15 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-2 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-11 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-3 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-6 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-14 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-5 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> 2023-08-18 15:07:18,225 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-89] Resetting the last
>> seen epoch of partition alerts.noise-13 to 0 since the associated topicId
>> changed from null to mjudIQc3Tg688a11hC78Zw
>> ---
>> ---
>>
>> 2023-08-18 15:08:50,878 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-5] Node 4 disconnected.
>> 2023-08-18 15:09:05,583 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-22] Node 2 disconnected.
>> 2023-08-18 15:09:07,602 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-58] Node 1 disconnected.
>> 2023-08-18 15:09:13,006 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-24] Node 3 disconnected.
>> 2023-08-18 15:09:13,106 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-24] Node 1 disconnected.
>> 2023-08-18 15:09:14,359 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-8] Node 3 disconnected.
>> 2023-08-18 15:09:14,359 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-17] Node 4 disconnected.
>> 2023-08-18 15:09:14,362 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-16] Node 2 disconnected.
>> 2023-08-18 15:09:15,998 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-20] Node 3 disconnected.
>>
>>
>>
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-1,
>> groupId=memalgogroup1-prod] Cancelled in-flight FETCH request with
>> correlation id 1987537 due to node 3 being disconnected (elapsed time since
>> creation: 365797ms, elapsed time since send: 365797ms, request timeout:
>> 30000ms)
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-1,
>> groupId=memalgogroup1-prod] Disconnecting from node 4 due to request
>> timeout.
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-1,
>> groupId=memalgogroup1-prod] Cancelled in-flight FETCH request with
>> correlation id 1987536 due to node 4 being disconnected (elapsed time since
>> creation: 365800ms, elapsed time since send: 365800ms, request timeout:
>> 30000ms)
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-0,
>> groupId=memalgogroup1-prod] Disconnecting from node 2 due to request
>> timeout.
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-0,
>> groupId=memalgogroup1-prod] Cancelled in-flight FETCH request with
>> correlation id 1985751 due to node 2 being disconnected (elapsed time since
>> creation: 365801ms, elapsed time since send: 365801ms, request timeout:
>> 30000ms)
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-0,
>> groupId=memalgogroup1-prod] Cancelled in-flight METADATA request with
>> correlation id 1985752 due to node 2 being disconnected (elapsed time since
>> creation: 0ms, elapsed time since send: 0ms, request timeout: 30000ms)
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-0,
>> groupId=memalgogroup1-prod] Disconnecting from node 4 due to request
>> timeout.
>> 2023-08-18 15:12:51,108 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Consumer clientId=mem_2731-0,
>> groupId=memalgogroup1-prod] Cancelled in-flight FETCH request with
>> correlation id 1985750 due to node 4 being disconnected (elapsed time since
>> creation: 365803ms, elapsed time since send: 365803ms, request timeout:
>> 30000ms)
>> 2023-08-18 15:12:51,162 INFO
>> org.apache.kafka.clients.FetchSessionHandler [] -
>> [Consumer clientId=mem_2731-0, groupId=memalgogroup1-prod] Error sending
>> fetch request (sessionId=475536967, epoch=111003) to node 2:
>> org.apache.kafka.common.errors.DisconnectException: null
>> 2023-08-18 15:12:51,162 INFO
>> org.apache.kafka.clients.FetchSessionHandler [] -
>> [Consumer clientId=mem_2731-1, groupId=memalgogroup1-prod] Error sending
>> fetch request (sessionId=294196072, epoch=110689) to node 3:
>> org.apache.kafka.common.errors.DisconnectException: null
>> 2023-08-18 15:12:51,162 INFO
>> org.apache.kafka.clients.FetchSessionHandler [] -
>> [Consumer clientId=mem_2731-0, groupId=memalgogroup1-prod] Error sending
>> fetch request (sessionId=985795042, epoch=111503) to node 4:
>> org.apache.kafka.common.errors.DisconnectException: null
>> 2023-08-18 15:12:51,162 INFO
>> org.apache.kafka.clients.FetchSessionHandler [] -
>> [Consumer clientId=mem_2731-1, groupId=memalgogroup1-prod] Error sending
>> fetch request (sessionId=1034101367, epoch=110920) to node 4:
>> org.apache.kafka.common.errors.DisconnectException: null
>> 2023-08-18 15:12:51,282 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-127] Node 1 disconnected.
>> 2023-08-18 15:12:57,362 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-12] Node 1 disconnected.
>> 2023-08-18 15:12:57,363 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-12] Node 4 disconnected.
>> 2023-08-18 15:13:02,690 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-57] Node 2 disconnected.
>>
>> ...
>> ...
>> ...
>>
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-13 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-15 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-2 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-14 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-10 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:23,910 INFO org.apache.kafka.clients.Metadata
>> [] - [Producer clientId=producer-15] Resetting the last
>> seen epoch of partition messages.invalid-9 to 0 since the associated
>> topicId changed from null to axH1mwlLQNGmYe95C78Bew
>> 2023-08-18 15:18:28,270 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-173] Node 3 disconnected.
>> 2023-08-18 15:18:28,429 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-175] Node 4 disconnected.
>> 2023-08-18 15:18:28,982 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-174] Node 4 disconnected.
>> 2023-08-18 15:18:36,480 INFO
>> org.apache.kafka.clients.consumer.internals.Fetcher [] -
>> [Consumer clientId=wifialgogroup1WifiInterface-6,
>> groupId=wifialgogroup1WifiInterface] Fetch position
>> FetchPosition{offset=69028579, offsetEpoch=Optional[0],
>> currentLeader=LeaderAndEpoch{leader=Optional[10.11.0.41:9092 (id: 2
>> rack: null)], epoch=0}} is out of range for partition
>> telemetry.wifi.interface-13, resetting offset
>> 2023-08-18 15:18:36,481 INFO
>> org.apache.kafka.clients.consumer.internals.SubscriptionState [] -
>> [Consumer clientId=wifialgogroup1WifiInterface-6,
>> groupId=wifialgogroup1WifiInterface] Resetting offset for partition
>> telemetry.wifi.interface-13 to position FetchPosition{offset=71593021,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 10.11.0.41:9092 (id: 2 rack: null)], epoch=0}}.
>> 2023-08-18 15:18:36,546 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-15] Node 3 disconnected.
>> 2023-08-18 15:18:36,547 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-15] Node 1 disconnected.
>> 2023-08-18 15:18:37,170 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-13] Node 4 disconnected.
>> 2023-08-18 15:18:44,718 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-14] Node 2 disconnected.
>> 2023-08-18 15:18:46,521 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-59] Node 3 disconnected.
>> 2023-08-18 15:18:48,530 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-47] Node 4 disconnected.
>> 2023-08-18 15:18:50,018 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-61] Node 4 disconnected.
>> 2023-08-18 15:18:50,219 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-130] Node 3 disconnected.
>> 2023-08-18 15:18:55,206 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-5] Node 2 disconnected.
>> 2023-08-18 15:19:06,115 INFO org.apache.kafka.clients.NetworkClient
>> [] - [Producer clientId=producer-22] Node 4 disconnected.
>>
>>
>> And system crushes here.
>>
>>
>> TaskManager2:
>>
>>
