are you calling sc.stop() at the end of your applications? The history server only displays completed applications, but if you don't call sc.stop(), it doesn't know that those applications have been stopped.
Note that in spark 1.3, the history server can also display running applications (including completed applications, but that it thinks are still running), which improves things a little bit. On Fri, Apr 17, 2015 at 10:13 AM, Serega Sheypak <serega.shey...@gmail.com> wrote: > Hi, started history-server > Here is UI output: > > > - *Event log directory:* file:/var/log/spark/applicationHistory/ > > No completed applications found! > > Did you specify the correct logging directory? Please verify your setting > of spark.history.fs.logDirectory and whether you have the permissions to > access it. > It is also possible that your application did not run to completion or did > not stop the SparkContext. > > Spark 1.2.0 > > I goto node where server runs and: > > ls -la /var/log/spark/applicationHistory/ > > total 44 > > drwxrwxrwx 11 root root 4096 Apr 17 14:50 . > > drwxrwxrwx 3 cassandra root 4096 Apr 16 15:31 .. > > drwxrwxrwx 2 vagrant vagrant 4096 Apr 17 10:06 app-20150417100630-0000 > > drwxrwxrwx 2 vagrant vagrant 4096 Apr 17 11:01 app-20150417110140-0001 > > drwxrwxrwx 2 vagrant vagrant 4096 Apr 17 11:12 app-20150417111216-0002 > > drwxrwxrwx 2 vagrant vagrant 4096 Apr 17 11:14 app-20150417111441-0003 > > drwxrwx--- 2 vagrant vagrant 4096 Apr 17 11:20 > *app-20150417112028-0004* > > drwxrwx--- 2 vagrant vagrant 4096 Apr 17 14:17 > *app-20150417141733-0005* > > drwxrwx--- 2 vagrant vagrant 4096 Apr 17 14:32 > *app-20150417143237-0006* > > drwxrwx--- 2 vagrant vagrant 4096 Apr 17 14:49 > *app-20150417144902-0007* > > drwxrwx--- 2 vagrant vagrant 4096 Apr 17 14:50 > *app-20150417145025-0008* > > > So there are logs, but history-server doesn't want to display them. > > I've checked workers, they are pointed to that dir also, I run app, I see > new log. > > > Here is history-server log output: > > vagrant@dsenode01:/usr/lib/spark/logs$ cat > spark-root-org.apache.spark.deploy.history.HistoryServer-1-dsenode01.out > > Spark assembly has been built with Hive, including Datanucleus jars on > classpath > > Spark Command: java -cp > ::/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar > -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true > -Dspark.history.fs.logDirectory=/var/log/spark/applicationHistory > -Dspark.eventLog.enabled=true -Xms512m -Xmx512m > org.apache.spark.deploy.history.HistoryServer > > ======================================== > > > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > > 15/04/17 09:55:21 INFO HistoryServer: Registered signal handlers for > [TERM, HUP, INT] > > 15/04/17 09:55:21 INFO SecurityManager: Changing view acls to: root > > 15/04/17 09:55:21 INFO SecurityManager: Changing modify acls to: root > > 15/04/17 09:55:21 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root); users > with modify permissions: Set(root) > > 15/04/17 09:55:22 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 15/04/17 09:55:24 INFO Utils: Successfully started service on port 18080. > > 15/04/17 09:55:24 INFO HistoryServer: Started HistoryServer at > http://dsenode01:18080 > > > What could be wrong with it? >
