[
https://issues.apache.org/jira/browse/SPARK-51360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramakrishna updated SPARK-51360:
--------------------------------
Description:
It looks like Spark's
*numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
is double the number of actual records. This can be very misleading in
production scenarios.
Please find the screenshots attached .
I have observed this issue with Delta table as input source, and delta table as
output source. The *numInputRows* is sometimes a multiple of actual no of
records .
If you see the screenshot, I have df.count() inside forEachBatch and it matches
the rate stream's injection rate.
However in my example I have used *rate* stream.
I have also zipped the project which reproduces the problem
You need Java 8 and SBT to run this locally.
was:
It looks like Spark's
*numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
is double the number of actual records. This can be very misleading in
production scenarios.
Please find the screenshots attached .
I have observed this issue with Delta table as input source, and delta table as
output source.
However in my example I have used *rate* stream.
I have also zipped the project which reproduces the problem
You need Java 8 and SBT to run this locally.
> Spark counts the total no of records twice in forEachBatch
> ----------------------------------------------------------
>
> Key: SPARK-51360
> URL: https://issues.apache.org/jira/browse/SPARK-51360
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 3.5.1
> Reporter: Ramakrishna
> Priority: Critical
> Labels: pull-request-available
> Attachments: Screenshot 2025-03-01 at 1.24.26 PM.png
>
>
> It looks like Spark's
> *numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
> is double the number of actual records. This can be very misleading in
> production scenarios.
>
> Please find the screenshots attached .
> I have observed this issue with Delta table as input source, and delta table
> as output source. The *numInputRows* is sometimes a multiple of actual no of
> records .
> If you see the screenshot, I have df.count() inside forEachBatch and it
> matches the rate stream's injection rate.
>
>
> However in my example I have used *rate* stream.
>
> I have also zipped the project which reproduces the problem
> You need Java 8 and SBT to run this locally.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
