One other pointer: Martijn mentioned in FLINK-24923 [1] tools like Nessus could generate traffic while scanning for ports. It's just the size of the request that is suspicious.
[1] https://issues.apache.org/jira/browse/FLINK-24923 On Thu, Mar 17, 2022 at 5:29 PM Ori Popowski <ori....@gmail.com> wrote: > This issue did not repeat, so it may be a network issue > > On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <map...@apache.org> wrote: > >> Hi Ori, >> that looks odd. The message seems to exceed the maximum size >> of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in >> Jira that supports a bug in Flink. Could it be that there was some network >> issue? >> >> Matthias >> >> On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <ori....@gmail.com> wrote: >> >>> I am running a production job for at least 1 year, and I got to day this >>> error: >>> >>> >>> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >>> Adjusted frame length exceeds 2147483647: 2969686273 - discarded >>> (connection to >>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >>> 10.208.65.38:40737') >>> >>> Nothing was changed in the code for a long time. What's causing this >>> error and how to fix it? I am running Flink 1.10.3 on YARN. >>> >>> This is the full stack trace: >>> >>> 2022-03-15 03:22:13 >>> org.apache.flink.runtime.io.network.netty.exception. >>> LocalTransportException: Adjusted frame length exceeds 2147483647: >>> 2969686273 - discarded (connection to >>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >>> 10.208.65.38:40737') >>> at org.apache.flink.runtime.io.network.netty. >>> CreditBasedPartitionRequestClientHandler.exceptionCaught( >>> CreditBasedPartitionRequestClientHandler.java:165) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeExceptionCaught( >>> AbstractChannelHandlerContext.java:297) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeExceptionCaught( >>> AbstractChannelHandlerContext.java:276) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireExceptionCaught( >>> AbstractChannelHandlerContext.java:268) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> ChannelInboundHandlerAdapter.exceptionCaught( >>> ChannelInboundHandlerAdapter.java:143) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeExceptionCaught( >>> AbstractChannelHandlerContext.java:297) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.notifyHandlerException( >>> AbstractChannelHandlerContext.java:831) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:376) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:360) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:352) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline >>> .java:1421) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:374) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:360) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio. >>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java: >>> 163) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKey(NioEventLoop.java:697) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKeysOptimized(NioEventLoop.java:632) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKeys(NioEventLoop.java:549) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .run(NioEventLoop.java:511) >>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) >>> at org.apache.flink.shaded.netty4.io.netty.util.internal. >>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> TooLongFrameException: Adjusted frame length exceeds 2147483647: >>> 2969686273 - discarded >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> LengthFieldBasedFrameDecoder.failIfNecessary( >>> LengthFieldBasedFrameDecoder.java:491) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> LengthFieldBasedFrameDecoder.exceededFrameLength( >>> LengthFieldBasedFrameDecoder.java:378) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java: >>> 421) >>> at org.apache.flink.runtime.io.network.netty. >>> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java: >>> 334) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder >>> .java:505) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:374) >>> ... 14 more >>> >>> >>>
