dongjoon-hyun commented on PR #52703: URL: https://github.com/apache/spark/pull/52703#issuecomment-3438278750
Thank you, @yaooqinn , @LuciferYang , @sarutak . Merged to master for Apache Spark 4.1.0-preview3. I manually verified on my Mac. It works. ``` $ sw_vers ProductName: macOS ProductVersion: 26.0.1 BuildVersion: 25A362 $ build/sbt "core/testOnly *.ShuffleNetty*Suite" ... [info] ShuffleNettyEpollSuite: [info] - groupByKey without compression [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - shuffle non-zero block size [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - shuffle serializer [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - zero sized blocks [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - zero sized blocks without kryo [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - shuffle on mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - sorting on mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - cogroup using mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - subtract mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - sort with Java non serializable class - Kryo [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - sort with Java non serializable class - Java [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - shuffle with different compression settings (SPARK-3426) [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - metrics for shuffle without aggregation [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - metrics for shuffle with aggregation [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - multiple simultaneous attempts for one task (SPARK-8029) [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - SPARK-34541: shuffle can be removed [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - SPARK-36206: shuffle checksum detect disk corruption [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] - SPARK-39771: warn when shuffle block number is too large [disabled on Mac OS X with EPOLL] !!! IGNORED !!! [info] ShuffleNettyNioSuite: [info] - groupByKey without compression (1 second, 253 milliseconds) [info] - shuffle non-zero block size (2 seconds, 705 milliseconds) [info] - shuffle serializer (2 seconds, 321 milliseconds) [info] - zero sized blocks (3 seconds, 414 milliseconds) [info] - zero sized blocks without kryo (3 seconds, 377 milliseconds) [info] - shuffle on mutable pairs (2 seconds, 205 milliseconds) [info] - sorting on mutable pairs (2 seconds, 205 milliseconds) [info] - cogroup using mutable pairs (2 seconds, 117 milliseconds) [info] - subtract mutable pairs (2 seconds, 24 milliseconds) [info] - sort with Java non serializable class - Kryo (2 seconds, 298 milliseconds) [info] - sort with Java non serializable class - Java (1 second, 618 milliseconds) [info] - shuffle with different compression settings (SPARK-3426) (170 milliseconds) [info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file (276 milliseconds) [info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle (38 milliseconds) [info] - metrics for shuffle without aggregation (132 milliseconds) [info] - metrics for shuffle with aggregation (200 milliseconds) [info] - multiple simultaneous attempts for one task (SPARK-8029) (31 milliseconds) [info] - SPARK-34541: shuffle can be removed (65 milliseconds) [info] - SPARK-36206: shuffle checksum detect disk corruption (3 seconds, 345 milliseconds) [info] - SPARK-39771: warn when shuffle block number is too large (75 milliseconds) [info] ShuffleNettyKQueueSuite: [info] - groupByKey without compression (344 milliseconds) [info] - shuffle non-zero block size (2 seconds, 198 milliseconds) [info] - shuffle serializer (2 seconds, 269 milliseconds) [info] - zero sized blocks (3 seconds, 131 milliseconds) [info] - zero sized blocks without kryo (3 seconds, 415 milliseconds) [info] - shuffle on mutable pairs (2 seconds, 382 milliseconds) [info] - sorting on mutable pairs (2 seconds, 350 milliseconds) [info] - cogroup using mutable pairs (2 seconds, 159 milliseconds) [info] - subtract mutable pairs (2 seconds, 257 milliseconds) [info] - sort with Java non serializable class - Kryo (2 seconds, 289 milliseconds) [info] - sort with Java non serializable class - Java (1 second, 710 milliseconds) [info] - shuffle with different compression settings (SPARK-3426) (130 milliseconds) [info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file (283 milliseconds) [info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle (39 milliseconds) [info] - metrics for shuffle without aggregation (81 milliseconds) [info] - metrics for shuffle with aggregation (133 milliseconds) [info] - multiple simultaneous attempts for one task (SPARK-8029) (25 milliseconds) [info] - SPARK-34541: shuffle can be removed (36 milliseconds) [info] - SPARK-36206: shuffle checksum detect disk corruption (3 seconds, 415 milliseconds) [info] - SPARK-39771: warn when shuffle block number is too large (31 milliseconds) [info] Run completed in 59 seconds, 880 milliseconds. [info] Total number of tests run: 40 [info] Suites: completed 3, aborted 0 [info] Tests: succeeded 40, failed 0, canceled 0, ignored 20, pending 0 [info] All tests passed. [success] Total time: 92 s (0:01:32.0), completed Oct 23, 2025, 10:35:14 AM ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
