Hi Ajinkya

I think you can try to decrease the size of batch shuffle with config
`taskmanager.memory.framework.off-heap.batch-shuffle.size` if the data
volume of your job is small, the default value is `64M`. You can find more
information in doc [1]

[1]
https://github.com/apache/flink/blob/master/docs/content/docs/ops/batch/batch_shuffle.md#sort-shuffle

Best,
Shammon FY

On Tue, Mar 21, 2023 at 12:22 AM Ajinkya Pathrudkar <
ajinkya.pathrudka...@gmail.com> wrote:

>
> Ajinkya Pathrudkar <ajinkya.pathrudka...@gmail.com>
> 10:53 AM (1 hour ago)
> to user-info
> I hope this email finds you well. I am writing to inform you of a recent
> update we made to our Flink version, upgrading from 1.14 to 1.15, along
> with a shift from Java 8 to Java 11. Since the update, we have encountered
> out-of-memory (direct memory) exceptions.
>
> The exception message suggests that increasing the task-off-heap memory
> could resolve the issue, but we are uncertain if there have been any
> changes to the memory model that may be contributing to this problem. The
> error message we received is as follows:
>
> java.lang.OutOfMemoryError: Can't allocate enough direct buffer for batch
> shuffle read buffer pool (bytes allocated: 26017792, bytes still needed:
> 41091072). To avoid the exception, you need to do one of the following
> adjustments: 1) If you have ever decreased
> taskmanager.memory.framework.off-heap.size, you need to undo the decrement;
> 2) If you ever increased
> taskmanager.memory.framework.off-heap.batch-shuffle.size, you should also
> increase taskmanager.memory.framework.off-heap.size; 3) If neither the
> above cases, it usually means some other parts of your application have
> consumed too many direct memory and the value of
> taskmanager.memory.task.off-heap.size should be increased.
>
> We would appreciate any insights you may have regarding this issue, and
> any suggestions on how to proceed would be greatly appreciated. Please let
> us know if there are any additional details we can provide to assist you in
> diagnosing the issue.
>
> Thank you for your time and consideration.
>
>
> Thanks,
> Ajinkya
>

Reply via email to