According to https://issues.apache.org/jira/browse/SPARK-19558 this feature was added in 2.3.
On Fri, Feb 16, 2018 at 12:43 AM, kurian vs <vskurian2...@gmail.com> wrote: > Hi, > > I was trying to create a custom Query execution listener by extending the > org.apache.spark.sql.util.QueryExecutionListener class. My custom listener > just contains some logging statements. But i do not see those logging > statements when i run a spark job. > > Here are the steps that i did: > > Create a custom listener by extending the QueryExecutionLIstener class > Created a jar file for the above project > Edited spark-defaults.conf to add the following properties: > spark.sql.queryExecutionListeners > com.customListener.spark.customSparkListener spark.driver.extraClassPath > /pathtoJarFile/CustomListener-1.0-SNAPSHOT.jar > Restarted everything using SPARK-HOME/sbin/start-all.sh > Ran a sample job using spark-submit > > I do not see any of the logging statements from the custom listener being > printed[i don't see them in the console] > > Is there anything else that i need to do to make it work other than the > above steps? > I'm adding this in the config properties because i need some info from all > the spark jobs being executed on that cluster. My assumption is that this > will prevent the need to do it from the code by adding an extra > ExecutionListenerManager.register(customListener) line. Is this assumption > correct? > From which version of spark is this supported? (i'm using spark V 2.2.1) > > > Can someone point me in the right direction? > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
