Quizhang,
This is a known issue that ExternalAppendOnlyMap can do tons of tiny spills
in certain situations. SPARK-4452 aims to deal with this issue, but we
haven't finalized a solution yet.
Dinesh's solution should help as a workaround, but you'll likely experience
suboptimal performance when tr
Hi Qiuzhuang,
This is a linux related issue. Please go through this [1] and change the
limits. hope this will solve your problem.
[1] https://rtcamp.com/tutorials/linux/increase-open-files-limit/
On Thu, Nov 20, 2014 at 9:45 AM, Qiuzhuang Lian
wrote:
> Hi All,
>
> While doing some ETL, I run
