that's Total Nonsense , EMR is total crap , use kubernetes i will help you . can you please provide whats the size of the shuffle file that is getting generated in each task . What's the total number of Partitions that you have ? What machines are you using ? Are you using an SSD ?
Best Tufan On Thu, 29 Sept 2022 at 20:28, Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > why not use EMR or data proc, kubernetes does not provide any benefit at > all for such scale of work. It is a classical case of over engineering and > over complication just for the heck of it. > > Also I think that in case you are in AWS, Redshift Spectrum or Athena for > 90% of use cases are way optimal. > > Regards, > Gourav > > On Thu, Sep 29, 2022 at 7:13 PM Igor Calabria <igor.calab...@gmail.com> > wrote: > >> Hi Everyone, >> >> I'm running spark 3.2 on kubernetes and have a job with a decently sized >> shuffle of almost 4TB. The relevant cluster config is as follows: >> >> - 30 Executors. 16 physical cores, configured with 32 Cores for spark >> - 128 GB RAM >> - shuffle.partitions is 18k which gives me tasks of around 150~180MB >> >> The job runs fine but I'm bothered by how underutilized the cluster gets >> during the reduce phase. During the map(reading data from s3 and writing >> the shuffle data) CPU usage, disk throughput and network usage is as >> expected, but during the reduce phase it gets really low. It seems the main >> bottleneck is reading shuffle data from other nodes, task statistics >> reports values ranging from 25s to several minutes(the task sizes are >> really close, they aren't skewed). I've tried increasing >> "spark.reducer.maxSizeInFlight" and >> "spark.shuffle.io.numConnectionsPerPeer" and it did improve performance by >> a little, but not enough to saturate the cluster resources. >> >> Did I miss some more tuning parameters that could help? >> One obvious thing would be to vertically increase the machines and use >> less nodes to minimize traffic, but 30 nodes doesn't seem like much even >> considering 30x30 connections. >> >> Thanks in advance! >> >>
