anishshri-db commented on code in PR #50520:
URL: https://github.com/apache/spark/pull/50520#discussion_r2029576191
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -1467,8 +1467,7 @@ class RocksDB(
// This is relative aggressive because that even if the uploading
succeeds,
// it is not necessarily the one written to the commit log. But we can
always load lineage
// from commit log so it is fine.
- lineageManager.resetLineage(lineageManager.getLineageForCurrVersion()
- .filter(i => i.version >= snapshot.version))
+ lineageManager.truncateFromVersion(snapshot.version)
Review Comment:
I guess its not easily possible. But any way to add a test to verify
correctness across threads here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
