Which Spark version? Are you using RDDs? Or datasets? What type are the features? If string how large? Is it spark standalone? How do you train/configure the algorithm. How do you initially parse the data? The standard driver and executor logs could be helpful.
> On 12 Oct 2016, at 09:24, 陈哲 <czhenj...@gmail.com> wrote: > > Hi > I'm using spark ml to train RandomForest Model . There is about over 200, > 000 lines in the training data file and about 100 features. I'm running > spark in local mode and with JAVA_OPTS like: -Xms1024m -Xmx10296m > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps, but OOM error keep coming out, I > tried with spark configuration change to avoid this but failed. > My spark conf: > spark.memory.fraction:0.85 > spark.executor.instances: 16 > spark.executor.heartbeatInterval:120 > spark.driver.maxResultSize:0 > spark.ui.retainedJobs=0 ... // I put all the similar conf to 0 , > spark.ui.retainedStages ... > > Here is some GC log: > 4876.028: [Full GC [PSYoungGen: 439296K->417358K(878592K)] [ParOldGen: > 9139497K->9139487K(9225216K)] 9578793K->9556845K(10103808K) [PSPermGen: > 81436K->81436K(81920K)], 2.0203540 secs] [Times: user=49.85 sys=0.12, > real=2.02 secs] > 4878.100: [Full GC [PSYoungGen: 417930K->187983K(878592K)] [ParOldGen: > 9139487K->9166111K(9225216K)] 9557418K->9354094K(10103808K) [PSPermGen: > 81436K->81436K(81920K)], 4.2368530 secs] [Times: user=53.77 sys=0.09, > real=4.23 secs] > 4882.414: [Full GC [PSYoungGen: 428018K->202158K(878592K)] [ParOldGen: > 9211167K->9196569K(9225216K)] 9639185K->9398727K(10103808K) [PSPermGen: > 81436K->81436K(81920K)], 4.4419950 secs] [Times: user=54.75 sys=0.08, > real=4.44 secs] > 4886.886: [Full GC [PSYoungGen: 425657K->397128K(878592K)] [ParOldGen: > 9196569K->9196568K(9225216K)] 9622227K->9593697K(10103808K) [PSPermGen: > 81436K->81436K(81920K)], 2.3522140 secs] [Times: user=51.41 sys=0.09, > real=2.35 secs] > 4889.239: [Full GC [PSYoungGen: 397128K->397128K(878592K)] [ParOldGen: > 9196568K->9196443K(9225216K)] 9593697K->9593572K(10103808K) [PSPermGen: > 81436K->81289K(81408K)], 30.5637160 secs] [Times: user=767.29 sys=2.98, > real=30.57 secs] > > the Full GC failed to collect enough memory , so OOM > > I have two questions : > 1. why spark log always show free memory like: > 4834409 [dispatcher-event-loop-5] INFO > org.apache.spark.storage.BlockManagerInfo - Removed broadcast_266_piece0 on > 172.17.1.235:9948 in memory (size: 642.5 KB, free: 7.1 GB) > Is this Wrong ? > > 2. How to avoid OOM here ? do I have to increase -Xmx to large value ? How > does spark use these memory , what's in those memory ? anyone can guide to > some docs ? > > > Thanks > > Patrick
