On 27 Feb 2016, at 20:40, Prabhu Joseph
<[email protected]<mailto:[email protected]>> wrote:
Hi All,
When i change the spark log4j.properties conversion pattern to know the
fully qualified class name, all the logs has the FQCN as
org.apache.spark.Logging. The actual fully qualified class name is overwritten.
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %C:
%m%n
FWIW, my test patter is:
log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c
(%F:%M(%L)) - %m%n
Gives me things like
2016-02-23 18:54:24,645 [EventPoster] DEBUG
org.apache.spark.deploy.history.yarn.YarnHistoryService
(Logging.scala:logDebug(58)) - entity successfully posted
2016-02-23 18:54:24,645 [EventPoster] DEBUG
org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter
(FileSystemTimelineWriter.java:flush(286)) - Flushing cache
2016-02-23 18:54:24,646 [EventPoster] DEBUG
org.apache.spark.deploy.history.yarn.YarnHistoryService
(Logging.scala:logDebug(58)) - Queue stopped
2016-02-23 18:54:24,646 [EventPoster] DEBUG
org.apache.spark.deploy.history.yarn.YarnHistoryService
(Logging.scala:logDebug(58)) - Queue shutdown, time limit= 18:54
2016-02-23 18:54:24,646 [EventPoster] INFO
org.apache.spark.deploy.history.yarn.YarnHistoryService
(Logging.scala:logInfo(54)) - Stopping dequeue service, final queue size is 0;
outstanding events to post count: 0
2016-02-23 18:54:24,646 [EventPoster] DEBUG
org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter
(FileSystemTimelineWriter.java:close(276)) - Closing cache
2016-02-23 18:54:24,648 [ScalaTest-main-running-IncompleteApplicationSuite]
INFO org.apache.spark.deploy.history.yarn.testtools.YarnTestUtils
(Logging.scala:logInfo(54)) - awaitServiceThreadStopped
Getting that thread in there is invaluable when there are lots of threads
about, having the line number in lets you track down where the code is. Here
you can see that there was a background thread [EventPoster], alongside the
test thread itself
As ted points out, this crawls in production, as it creates an exception on
every log, so that the stack trace can be examined —you would only use it in
production in an emergency. For test runs though, wonderful.
