v1.5.1.
Trying to enable CsvSink for metrics collecting, but I get the following
error as soon as kicking off a 'spark-submit' app:
15/12/08 11:24:02 INFO storage.BlockManagerMaster: Registered
BlockManager
15/12/08 11:24:02 ERROR metrics.MetricsSystem: Sink class
org.apache.spark.metrics.sink.CsvSink cannot be instantialized
15/12/08 11:24:02 ERROR spark.SparkContext: Error initializing
SparkContext.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance
(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance
(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance
(Constructor.java:526)
Only made one change -- edited my 'metrics.properties' file which now
contains the following settings:
worker.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
master.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
executor.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
driver.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
# Polling period for CsvSink
*.sink.csv.period=5
*.sink.csv.unit=second
# Polling directory for CsvSink
*.sink.csv.directory=/TestAutomation/results
According to documentation, what's all I needed to set ....
http://spark.apache.org/docs/latest/monitoring.html
Do I need to rebuild my spark distro with a special flag or something?
No much info on this on the Web at all.
Idea? Pointers? Thanks!
