what's the dump info by jstack? Yours, Xuefeng Wu 吴雪峰 敬上
> On 2015年2月6日, at 上午10:20, Michael Albert <[email protected]> > wrote: > > My apologies for following up my own post, but I thought this might be of > interest. > > I terminated the java process corresponding to executor which had opened the > stderr file mentioned below (kill <pid>). > Then my spark job completed without error (it was actually almost finished). > > Now I am completely confused :-). > > Thanks! > -Mike > > > From: Michael Albert <[email protected]> > To: "[email protected]" <[email protected]> > Sent: Thursday, February 5, 2015 9:04 PM > Subject: Spark stalls or hangs: is this a clue? remote fetches seem to never > return? > > Greetings! > > Again, thanks to all who have given suggestions. > I am still trying to diagnose a problem where I have processes than run for > one or several hours but intermittently stall or hang. > By "stall" I mean that there is no CPU usage on the workers or the driver, > nor network activity, nor do I see disk activity. > It just hangs. > > Using the Application Master to find which workers still had active tasks, I > then went to that machine and looked in the user logs. > In one of the users log's "stderr" files, it ends with "Started 50 remote > fetches...." > Should there be a message saying that the fetch was completed? > Any suggestions as to how I might diagnose why the fetch was not completed? > > Thanks! > -Mike > > Here is the last part of the log: > 15/02/06 01:33:46 INFO storage.MemoryStore: ensureFreeSpace(5368) called with > curMem=875861, maxMem=2315649024 > 15/02/06 01:33:46 INFO storage.MemoryStore: Block broadcast_10 stored as > values in memory (estimated size 5.2 KB, free 2.2 GB) > 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs > for shuffle 5, fetching them > 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker > actor = > Actor[akka.tcp://[email protected]:44124/user/MapOutputTracker#-878402310] > 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs > for shuffle 5, fetching them > 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Got the output locations > 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 > non-empty blocks out of 300 blocks > 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 > non-empty blocks out of 300 blocks > 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote > fetches in 47 ms > 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote > fetches in 48 ms > It's been like that for half and hour. > > Thanks! > -Mike > > > >
