Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-05 Thread kalyan
Hi Cheng, Is there some place where I can get more details on this, or if you could give a couple of lines explaining about it. > But given memory usage from writers is non-visible to spark now, it seems > to me that there’s no other good way to model the memory usage for write. regards kalyan.

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-05 Thread Cheng Su
Hi Kalyan, > Is there some place where I can get more details on this, or if you could > give a couple of lines explaining about it. But given memory usage from writers is non-visible to spark now, it seems to me that there’s no other good way to model the memory usage for write. Data source v