Hi, That was really quick! #impressed
Thanks HeartSaVioR for such prompt response! I'm fine with the current state of the issue = no need to change anything. Whatever makes Spark more shiny WFM! Jacek On Sun, 13 Oct 2019, 09:19 Jungtaek Lim, <kabhwan.opensou...@gmail.com> wrote: > Filed SPARK-29450 [1] and raised a patch [2]. Please let me know if you > would like to be assigned as a reporter of SPARK-29450. > > 1. https://issues.apache.org/jira/browse/SPARK-29450 > 2. https://github.com/apache/spark/pull/26104 > > On Sun, Oct 13, 2019 at 4:06 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Thanks for reporting. That might be possible it could be intentionally >> excluded as it could cause some confusion before introducing empty batch >> (given output rows are irrelevant to the input rows in current batch), but >> given we have empty batch I'm not seeing the reason why we don't deal with >> it. I'll file and submit a patch. >> >> Btw, there's a metric bug with empty batch as well - see SPARK-29314 [1] >> which I've submitted a patch recently. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> 1. http://issues.apache.org/jira/browse/SPARK-29314 >> >> >> On Sun, Oct 13, 2019 at 1:12 AM Jacek Laskowski <ja...@japila.pl> wrote: >> >>> Hi, >>> >>> I use Spark 2.4.4 >>> >>> I've just noticed that the number of output rows metric >>> of StateStoreSaveExec physical operator does not seem to be measured for >>> Append output mode. In other words, whatever happens before or >>> after StateStoreSaveExec operator the metric is always 0. >>> >>> It is measured for the other modes - Complete and Update. >>> >>> See >>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L329-L365 >>> >>> Is this intentional? Why? >>> >>> Pozdrawiam, >>> Jacek Laskowski >>> ---- >>> https://about.me/JacekLaskowski >>> The Internals of Spark SQL https://bit.ly/spark-sql-internals >>> The Internals of Spark Structured Streaming >>> https://bit.ly/spark-structured-streaming >>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals >>> Follow me at https://twitter.com/jaceklaskowski >>> >>>
