Spark version is 3.2.2

I want to observe row count before write to jdbc using observation:

ds.observe("stepName", F.count(F.lit(1)).as("_rc"))


And I add  QueryExecutionListener to my spark session. code like :

val metrics = qe.observedMetrics
> logInfo(s"Function $funcName success. Try send metrics size: ${
> metrics.size}")
>

when i write data to jdbc. Log print as:

> Function command success. Try send metrics size: 0


But when i write data to hdfs. *I can get the metric of row count*.

I tried use *org.apache.spark.sql.Observation*. However, it is also not
possible to get jdbc's observation information, but it is possible to get
hdfs's.

Whether there is a solution?

Reply via email to