Filed SPARK-29450 [1] and raised a patch [2]. Please let me know if you
would like to be assigned as a reporter of SPARK-29450.
1. https://issues.apache.org/jira/browse/SPARK-29450
2. https://github.com/apache/spark/pull/26104
On Sun, Oct 13, 2019 at 4:06 PM Jungtaek Lim
wrote:
> Thanks for rep
Thanks for reporting. That might be possible it could be intentionally
excluded as it could cause some confusion before introducing empty batch
(given output rows are irrelevant to the input rows in current batch), but
given we have empty batch I'm not seeing the reason why we don't deal with
it. I
Hi,
That was really quick! #impressed
Thanks HeartSaVioR for such prompt response! I'm fine with the current
state of the issue = no need to change anything. Whatever makes Spark more
shiny WFM!
Jacek
On Sun, 13 Oct 2019, 09:19 Jungtaek Lim,
wrote:
> Filed SPARK-29450 [1] and raised a patch [
It might be due to empty batch (activated when there're stateful
operator(s) and the previous batch advances watermark), which has no input
so no moving watermark.
Did you only turn on DEBUG for WatermarkTracker? If you turn on DEBUG for
MicroBatchExecution as well, it would log "Completed batch "
