gnanda commented on code in PR #54529:
URL: https://github.com/apache/spark/pull/54529#discussion_r2881385434
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3695,6 +3684,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STATE_STORE_FILE_CHECKSUM_THREAD_POOL_SIZE =
+ buildConf("spark.sql.streaming.stateStore.fileChecksumThreadPoolSize")
+ .internal()
+ .doc("Number of threads used to compute file checksums concurrently when
uploading " +
+ "state store checkpoints (e.g. main file and checksum file). " +
Review Comment:
Done
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/ChecksumCheckpointFileManager.scala:
##########
@@ -127,12 +128,14 @@ case class ChecksumFile(path: Path) {
* orphan checksum files. If using this, it is
your responsibility
* to clean up the potential orphan checksum
files.
* @param numThreads This is the number of threads to use for the thread pool,
for reading/writing
- * files. To avoid blocking, if the file manager instance is
being used by a
- * single thread, then you can set this to 2 (one thread for
main file, another
- * for checksum file).
- * If file manager is shared by multiple threads, you can
set it to
- * number of threads using file manager * 2.
- * Setting this differently can lead to file operation being
blocked waiting for
+ * files. Must be a non-negative integer.
+ * Setting this to 0 disables the thread pool and runs all
operations
+ * sequentially on the calling thread (no concurrency).
+ * To avoid blocking with a single concurrent caller, set
this to 2 (one thread
Review Comment:
Reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
