Hey guys, I need to capture runtime stats and metrics for every executed SQL query. For that I'm using two listeners: the QueryExecutionListener, that gives me information about executed QueryExecution, and the SparkListener, that provides me the info about the jobs, stages and tasks. The problem I have is that the stats provided by the SparkListener refer the execution ID, that is generated by SQLExecution class, but is not stored in the QueryExecution object itself. Subsequently, when I catch events from the QueryExecutionListener I have an execution object, but know it's ID that it has on the time of execution. And I cannot tell what captured jobs and tasks stats on the other layer relates to it.
Any suggestions on how could I link those pieces of information? Is there any reason why execution ID is not stored in the QueryExecution class? Thank you.
