Hi all,

Is there any way to increase the sort buffer size other than increasing the overall TaskManager memory? The following error comes up running a job with huge matrix block objects on a cluster:

Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 100499456 bytes).

Every TM has at least 40 GB of memory while the maximum sort buffer size is at 100 MB. What is the reason for this limit? Sorry if I'm missing something, but I have not found any related discussion or documentation yet.

Cheers,
Gabor

Reply via email to