Spark version is 3.2.2 I want to observe row count before write to jdbc using observation:
ds.observe("stepName", F.count(F.lit(1)).as("_rc")) And I add QueryExecutionListener to my spark session. code like : val metrics = qe.observedMetrics > logInfo(s"Function $funcName success. Try send metrics size: ${ > metrics.size}") > when i write data to jdbc. Log print as: > Function command success. Try send metrics size: 0 But when i write data to hdfs. *I can get the metric of row count*. I tried use *org.apache.spark.sql.Observation*. However, it is also not possible to get jdbc's observation information, but it is possible to get hdfs's. Whether there is a solution?