[
https://issues.apache.org/jira/browse/SPARK-51360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramakrishna updated SPARK-51360:
--------------------------------
Description:
It looks like Spark's
*numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
is double the number of actual records. This can be very misleading in
production scenarios.
Please find the screenshots attached .
The *numInputRows* is sometimes a multiple of actual no of records .
If you see the screenshot, I have df.count() inside forEachBatch and it matches
the rate stream's ingestion rate. 2 rows per second and 30 seconds trigger is
around 60 records , But *numInputRows* is double the value.
This seems to be a problem only with forEachBatch, otherwise it works fine.
I have observed this issue with Delta table as input source, and delta table as
output source.
However in my example I have used *rate* stream.
I have also zipped the project which reproduces the problem, (Minimal
Reproducible Example)
You need Java 8 and SBT to run this locally.
was:
It looks like Spark's
*numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
is double the number of actual records. This can be very misleading in
production scenarios.
Please find the screenshots attached .
The *numInputRows* is sometimes a multiple of actual no of records .
If you see the screenshot, I have df.count() inside forEachBatch and it matches
the rate stream's ingestion rate. 2 rows per second and 30 seconds trigger is
aroundd 60 records , But *numInputRows* is double the value.
This seems to be a problem only with forEachBatch, otherwise it works fine.
I have observed this issue with Delta table as input source, and delta table as
output source.
However in my example I have used *rate* stream.
I have also zipped the project which reproduces the problem, (Minimal
Reproducible Example)
You need Java 8 and SBT to run this locally.
> Spark counts the total no of records twice in forEachBatch
> ----------------------------------------------------------
>
> Key: SPARK-51360
> URL: https://issues.apache.org/jira/browse/SPARK-51360
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit, Structured Streaming
> Affects Versions: 3.5.1
> Reporter: Ramakrishna
> Priority: Critical
> Labels: SPARK
> Attachments: Scala_practice.zip, Screenshot 2025-03-01 at 1.48.11
> PM.png
>
>
> It looks like Spark's
> *numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
> is double the number of actual records. This can be very misleading in
> production scenarios.
>
> Please find the screenshots attached .
> The *numInputRows* is sometimes a multiple of actual no of records .
> If you see the screenshot, I have df.count() inside forEachBatch and it
> matches the rate stream's ingestion rate. 2 rows per second and 30 seconds
> trigger is around 60 records , But *numInputRows* is double the value.
>
> This seems to be a problem only with forEachBatch, otherwise it works fine.
>
> I have observed this issue with Delta table as input source, and delta table
> as output source.
> However in my example I have used *rate* stream.
>
> I have also zipped the project which reproduces the problem, (Minimal
> Reproducible Example)
> You need Java 8 and SBT to run this locally.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
