This issue did not repeat, so it may be a network issue On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <map...@apache.org> wrote:
> Hi Ori, > that looks odd. The message seems to exceed the maximum size of 2147483647 > bytes (2GB). I couldn't find anything similar in the ML or in Jira that > supports a bug in Flink. Could it be that there was some network issue? > > Matthias > > On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <ori....@gmail.com> wrote: > >> I am running a production job for at least 1 year, and I got to day this >> error: >> >> >> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: >> Adjusted frame length exceeds 2147483647: 2969686273 - discarded >> (connection to >> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >> 10.208.65.38:40737') >> >> Nothing was changed in the code for a long time. What's causing this >> error and how to fix it? I am running Flink 1.10.3 on YARN. >> >> This is the full stack trace: >> >> 2022-03-15 03:22:13 >> org.apache.flink.runtime.io.network.netty.exception. >> LocalTransportException: Adjusted frame length exceeds 2147483647: >> 2969686273 - discarded (connection to >> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/ >> 10.208.65.38:40737') >> at org.apache.flink.runtime.io.network.netty. >> CreditBasedPartitionRequestClientHandler.exceptionCaught( >> CreditBasedPartitionRequestClientHandler.java:165) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeExceptionCaught( >> AbstractChannelHandlerContext.java:297) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeExceptionCaught( >> AbstractChannelHandlerContext.java:276) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.fireExceptionCaught( >> AbstractChannelHandlerContext.java:268) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter >> .java:143) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeExceptionCaught( >> AbstractChannelHandlerContext.java:297) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.notifyHandlerException( >> AbstractChannelHandlerContext.java:831) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:376) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:360) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.fireChannelRead( >> AbstractChannelHandlerContext.java:352) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline >> .java:1421) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:374) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:360) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) >> at org.apache.flink.shaded.netty4.io.netty.channel.nio. >> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163 >> ) >> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >> .processSelectedKey(NioEventLoop.java:697) >> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >> .processSelectedKeysOptimized(NioEventLoop.java:632) >> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >> .processSelectedKeys(NioEventLoop.java:549) >> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >> .run(NioEventLoop.java:511) >> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) >> at org.apache.flink.shaded.netty4.io.netty.util.internal. >> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> at java.lang.Thread.run(Thread.java:748) >> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec. >> TooLongFrameException: Adjusted frame length exceeds 2147483647: >> 2969686273 - discarded >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder >> .java:491) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> LengthFieldBasedFrameDecoder.exceededFrameLength( >> LengthFieldBasedFrameDecoder.java:378) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421 >> ) >> at org.apache.flink.runtime.io.network.netty. >> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334 >> ) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder >> .java:505) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) >> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) >> at org.apache.flink.shaded.netty4.io.netty.channel. >> AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:374) >> ... 14 more >> >> >>
