Thanks for reporting. That might be possible it could be intentionally excluded as it could cause some confusion before introducing empty batch (given output rows are irrelevant to the input rows in current batch), but given we have empty batch I'm not seeing the reason why we don't deal with it. I'll file and submit a patch.
Btw, there's a metric bug with empty batch as well - see SPARK-29314 [1] which I've submitted a patch recently. Thanks, Jungtaek Lim (HeartSaVioR) 1. http://issues.apache.org/jira/browse/SPARK-29314 On Sun, Oct 13, 2019 at 1:12 AM Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > I use Spark 2.4.4 > > I've just noticed that the number of output rows metric > of StateStoreSaveExec physical operator does not seem to be measured for > Append output mode. In other words, whatever happens before or > after StateStoreSaveExec operator the metric is always 0. > > It is measured for the other modes - Complete and Update. > > See > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L329-L365 > > Is this intentional? Why? > > Pozdrawiam, > Jacek Laskowski > ---- > https://about.me/JacekLaskowski > The Internals of Spark SQL https://bit.ly/spark-sql-internals > The Internals of Spark Structured Streaming > https://bit.ly/spark-structured-streaming > The Internals of Apache Kafka https://bit.ly/apache-kafka-internals > Follow me at https://twitter.com/jaceklaskowski > >
