attilapiros commented on code in PR #50122:
URL: https://github.com/apache/spark/pull/50122#discussion_r1986090749
##########
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala:
##########
@@ -474,6 +474,26 @@ class BlockManagerSuite extends SparkFunSuite with
Matchers with PrivateMethodTe
assert(!BlockManagerId("notADriverIdentifier", "XXX", 1).isDriver)
}
+ test("SPARK-43221: Host local block fetching should use a block status with
disk size") {
+ conf.set(IO_ENCRYPTION_ENABLED, true)
Review Comment:
Yes, it is needed for the reproduction of the error.
Otherwise we would read the complete file (use the file size instead of the
size get from `getLocationsAndStatus` which might be the in-memory block size
before this PR).
Size get from `BlockManagerMasterEndpoint#getLocationsAndStatus()` and
passed as `blockSize`:
https://github.com/apache/spark/blob/4231d58245251a34ae80a38ea4bbf7d720caa439/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1240
Filesize get as `file.length`:
https://github.com/apache/spark/blob/4231d58245251a34ae80a38ea4bbf7d720caa439/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
