Re: question about data skew and memory issues

Hi, also if you are using SPARK 3.2.x please try to see the documentation on handling skew using SPARK settings. Regards, Gourav Sengupta On Tue, Dec 14, 2021 at 6:01 PM David Diebold wrote: > Hello all, > > I was wondering if it possible to encounter out of memory exceptions on > spark executo

Re: question about data skew and memory issues

Hi david, Can you give us the example of code you are running and the way you are aggregating over keys? HTH view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destr

question about data skew and memory issues

Hello all, I was wondering if it possible to encounter out of memory exceptions on spark executors when doing some aggregation, when a dataset is skewed. Let's say we have a dataset with two columns: - key : int - value : float And I want to aggregate values by key. Let's say that we have a tons o