Hi all,
Is there any way to increase the sort buffer size other than increasing
the overall TaskManager memory?
The following error comes up running a job with huge matrix block
objects on a cluster:
Error obtaining the sorted input: Thread 'SortMerger Reading Thread'
terminated due to an exception: The record exceeds the maximum size of a
sort buffer (current maximum: 100499456 bytes).
Every TM has at least 40 GB of memory while the maximum sort buffer size
is at 100 MB. What is the reason for this limit? Sorry if I'm missing
something, but I have not found any related discussion or documentation yet.
Cheers,
Gabor
- Any way to increase sort buffer size? Gábor Hermann
-