[
https://issues.apache.org/jira/browse/FLINK-25670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493017#comment-17493017
]
Kyle commented on FLINK-25670:
------------------------------
Hi Igal, with increasing "client_max_size", the memory size of task heap should
be also enlarged accordingly. After that the large state can be generated and
updated by remote function and sent back to StateFun cluster. Thank you very
much for suggestion and support!
Another question for my case is whether large data is allowed to be sent from
ingress to StateFun cluster and then to remote function. Currently I don't have
a chance to try this case but per your expertise do you have any idea?
> StateFun: Unable to handle oversize HTTP message if state size is large
> -----------------------------------------------------------------------
>
> Key: FLINK-25670
> URL: https://issues.apache.org/jira/browse/FLINK-25670
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
> Affects Versions: statefun-3.1.1
> Reporter: Kyle
> Priority: Major
> Attachments: 00-module.yaml, functions.py
>
>
> Per requirement we need to handle state which is about 500MB large (72MB
> state allocated in commented code as attached). However the HTTP message
> limit disallows us to send back large state to StateFun cluster after saving
> state in Stateful Function.
> Another question is whether large data is allowed to send to Stateful
> Function from ingress.
>
> 2022-01-17 07:57:18,416 WARN
> org.apache.flink.statefun.flink.core.nettyclient.NettyRequest [] - Exception
> caught while trying to deliver a message: (attempt
> #10)ToFunctionRequestSummary(address=Address(example, hello, 5555),
> batchSize=1, totalSizeInBytes=80, numberOfStates=2)
> org.apache.flink.shaded.netty4.io.netty.handler.codec.TooLongFrameException:
> Response entity too large: DefaultHttpResponse(decodeResult: success,
> version: HTTP/1.1)
> HTTP/1.1 200 OK
> Content-Type: application/octet-stream
> Content-Length: 40579630
> Date: Mon, 17 Jan 2022 07:57:18 GMT
> Server: Python/3.9 aiohttp/3.8.1
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:276)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:87)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.invokeHandleOversizedMessage(MessageAggregator.java:404)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:254)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [statefun-flink-distribution.jar:3.1.1]
> at java.lang.Thread.run(Unknown Source) [?:?]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
