Could you explain more what your job is doing and what data types you are using? These numbers alone don't necessarily indicate something is wrong. The relationship between the in-memory and on-disk shuffle amount is definitely a bit strange, the data gets compressed when written to disk, but unless you have a weird dataset (E.g. all zeros) I wouldn't expect it to compress _that_ much.
On Mon, Apr 28, 2014 at 1:18 AM, Liu, Raymond <[email protected]> wrote: > Hi > > > I am running a simple word count program on spark standalone > cluster. The cluster is made up of 6 node, each run 4 worker and each > worker own 10G memory and 16 core thus total 96 core and 240G memory. ( > well, also used to configed as 1 worker with 40G memory on each node ) > > I run a very small data set (2.4GB on HDFS on total) to confirm > the problem here as below: > > As you can read from part of the task metrics as below, I noticed > that the shuffle spill part of metrics indicate that there are something > wrong. > > Executor ID Address Task Time Total Tasks Failed Tasks > Succeeded Tasks Shuffle Read Shuffle Write Shuffle Spill (Memory) > Shuffle Spill (Disk) > 0 sr437:42139 29 s 4 0 4 0.0 B 4.3 MB > 23.6 GB 4.3 MB > 1 sr433:46935 1.1 min 4 0 4 0.0 B 4.2 MB > 19.0 GB 3.4 MB > 10 sr436:53277 26 s 4 0 4 0.0 B 4.3 MB > 25.6 GB 4.6 MB > 11 sr437:58872 32 s 4 0 4 0.0 B 4.3 MB > 25.0 GB 4.4 MB > 12 sr435:48358 27 s 4 0 4 0.0 B 4.3 MB > 25.1 GB 4.4 MB > > > You can see that the Shuffle Spill (Memory) is pretty high, almost 5000x > of the actual shuffle data and Shuffle Spill (Disk), and also it seems to > me that by no means that the spill should trigger, since the memory is not > used up at all. > > To verify that I further reduce the data size to 240MB on total > > And here is the result: > > > Executor ID Address Task Time Total Tasks Failed Tasks > Succeeded Tasks Shuffle Read Shuffle Write Shuffle Spill (Memory) > Shuffle Spill (Disk) > 0 sr437:50895 15 s 4 0 4 0.0 B 703.0 KB > 80.0 MB 43.2 KB > 1 sr433:50207 17 s 4 0 4 0.0 B 704.7 KB > 389.5 MB 90.2 KB > 10 sr436:56352 16 s 4 0 4 0.0 B 700.9 KB > 814.9 MB 181.6 KB > 11 sr437:53099 15 s 4 0 4 0.0 B 689.7 KB > 0.0 B 0.0 B > 12 sr435:48318 15 s 4 0 4 0.0 B 702.1 KB > 427.4 MB 90.7 KB > 13 sr433:59294 17 s 4 0 4 0.0 B 704.8 KB > 779.9 MB 180.3 KB > > Nothing prevent spill from happening. > > Now, there seems to me that there must be something wrong with the spill > trigger codes. > > So anyone encounter this issue? By the way, I am using latest trunk code. > > > Best Regards, > Raymond Liu >
