Hi, I was trying to create a custom Query execution listener by extending the org.apache.spark.sql.util.QueryExecutionListener class. My custom listener just contains some logging statements. But i do not see those logging statements when i run a spark job.
Here are the steps that i did: 1. Create a custom listener by extending the QueryExecutionLIstener class 2. Created a jar file for the above project 3. Edited spark-defaults.conf to add the following properties: spark.sql.queryExecutionListeners com.customListener.spark.customSparkListener spark.driver.extraClassPath /pathtoJarFile/CustomListener-1.0-SNAPSHOT.jar 4. Restarted everything using SPARK-HOME/sbin/start-all.sh 5. Ran a sample job using spark-submit I do not see any of the logging statements from the custom listener being printed[i don't see them in the console] - Is there anything else that i need to do to make it work other than the above steps? - I'm adding this in the config properties because i need some info from all the spark jobs being executed on that cluster. My assumption is that this will prevent the need to do it from the code by adding an extra ExecutionListenerManager <https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-BaseSessionStateBuilder.html#listenerManager> .register(customListener) line. Is this assumption correct? - From which version of spark is this supported? (i'm using spark V 2.2.1) Can someone point me in the right direction?
