Hi,
I am trying to submit a custom metrics.properties file to enable the
collection of spark metrics, but I am having a hard time even starting it
in local mode.
spark-submit \
...
--files "./metrics.properties"
--conf "spark.metrics.conf=metrics.properties"
...
However I am getting the following error :
java.io.FileNotFoundException: metrics.properties (No such file or
directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
I tried dropping the spark.metrics.conf just to see if the file gets moved
ok, but that results in an error as well :
...
15/10/29 11:58:13 INFO Utils: Successfully started service 'SparkUI' on
port 4040.
15/10/29 11:58:13 INFO SparkUI: Started SparkUI at http://192.168.0.2:4040
...
Exception in thread "main" java.io.FileNotFoundException: Added file
file:/Users/radu/dev/sparkTest/metrics.properties does not exist.
Is there a special handshake that I need to do to get this working ? I've
found some anecdotal reference online where people claim that they've got
this running using only the 2 flags mentioned above.
Thanks,
Radu
