Enrico Minack created SPARK-56199:
-------------------------------------
Summary: ShuffleBlockFetcherIterator should not read
FalbackStorage blocks as local blocks
Key: SPARK-56199
URL: https://issues.apache.org/jira/browse/SPARK-56199
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Enrico Minack
The ShuffleBlockFetcherIterator treats blocks stored on the FallbackStorage
(very likely a remote distributed storage like S3 or HDFS) as local block files.
The current implementation has the following disadvantages:
- blocks are read from fallback storage single threaded and synchronously (one
by one)
- waiting for fallback storage blocks being transferred over the network blocks
reading local blocks
- fallback storage blocks are read in the {{catch}} branch when trying to read
it as a local block (which cannot be found locally)
Reading fallback storage blocks should be treated like remote blocks: fetched
independently from local blocks, asynchronously and multi-threaded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
