Hi Ashwani,
Please verify input data by ensuring that the data being processed is
valid and free of null values or unexpected data types.
if data undergoes complex transformations before sorting review the
data Transformations, verify that data transformations don't introduce
inconsistencies or nul
Spark version is 3.2.2
I want to observe row count before write to jdbc using observation:
ds.observe("stepName", F.count(F.lit(1)).as("_rc"))
And I add QueryExecutionListener to my spark session. code like :
val metrics = qe.observedMetrics
> logInfo(s"Function $funcName success. Try send me
