[
https://issues.apache.org/jira/browse/SPARK-51360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramakrishna updated SPARK-51360:
--------------------------------
Component/s: Structured Streaming
(was: PySpark)
(was: SQL)
> Spark counts the total no of records twice in forEachBatch
> ----------------------------------------------------------
>
> Key: SPARK-51360
> URL: https://issues.apache.org/jira/browse/SPARK-51360
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.5.1
> Reporter: Ramakrishna
> Priority: Critical
> Labels: SPARK
> Attachments: Scala_practice.zip, Screenshot 2025-03-01 at 1.24.26
> PM.png
>
>
> It looks like Spark's
> *numInputRows* in spark's streamingQuery Listener, is wrongly calculated , it
> is double the number of actual records. This can be very misleading in
> production scenarios.
>
> Please find the screenshots attached .
> I have observed this issue with Delta table as input source, and delta table
> as output source. The *numInputRows* is sometimes a multiple of actual no of
> records .
> If you see the screenshot, I have df.count() inside forEachBatch and it
> matches the rate stream's ingestion rate. But *numInputRows* is double the
> value.
>
>
>
> However in my example I have used *rate* stream.
>
> I have also zipped the project which reproduces the problem
> You need Java 8 and SBT to run this locally.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
