There are quite a few reason why the framesize could be exceeded.
The most common one we see is due to the parallelism being so high that
tasks can't be deployed in the first place. When a task is deployed the
RPC payload also contains information about all downstream tasks this
task sends data to; when those are a few thousand (usually in case of a
shuffle) the amount of data can quickly add up.
Other causes could be tasks having thousands of accumulators or there
being too many metrics on one TaskExecutor (which would result in
metrics not being queryable from the WebUI/REST API).
Overall though, the documentation is pretty accurate. The framesize
being exceeded is usually not because the user did anything wrong, but
just operating at a scale that the default framesize cannot support. The
only solution to that is to increase the framesize.
On 9/18/2020 12:34 PM, shravan wrote:
Thanks again for the quick response.
In that case, could you tell me what are the possible factors that warrant a
framesize increase? I see the official documentation and it simply states
"If Flink fails because messages exceed this limit, then you should increase
it", which isn't very convincing.
Regards,
M S Shravan
Chesnay Schepler wrote
If you use 1.10.0 or above the framesize for which it failed is part of
the exception message, see FLINK-14618.
If you are using older version, then I'm afraid there is no way to tell.
On 9/18/2020 12:11 PM, shravan wrote:
Thanks for the quick response.
I might have wrongly phrased one of the questions.
/"> how can we know the expected size for which it is failing?
If you did not configure akka.framesize yourself then it is set to the
documented default value. See the configuration documentation for the
release you are using."/
We found out the default size from the configuration but we are unable to
identify the size for which it fails. Could you help out on this?
Awaiting a response.
Regards,
Shravan
Chesnay Schepler wrote
how can we know the expected size for which it is failing?
If you did not configure akka.framesize yourself then it is set to the
documented default value. See the configuration documentation for the
release you are using.
> Does the operator state have any impact on the expected Akka frame
size?
If you are using the MemoryStateBackend, yes. Otherwise, the impact of
using any form of state on the framesize should be negligible.
> What is the impact of increasing it?
Increase in memory consumption, probably around 1-2x the increased
amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)
On 9/18/2020 9:50 AM, shravan wrote:
Hi,
This is in continuation to an already raised request, (had replied to
the
same thread but couldn't get any response yet, hence posting a new
request)
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
We are observing the same error as well with regard to "The rpc
invocation
size exceeds the maximum akka framesize.", and have follow-up questions
on
the same.
Why we face this issue, how can we know the expected size for which it
is
failing? The error message does not indicate that. Does the operator
state
have any impact on the expected Akka frame size? What is the impact of
increasing it?
Awaiting a response.
Regards,
Shravan
--
Sent from:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
--
Sent from:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
