dongjoon-hyun commented on PR #49476: URL: https://github.com/apache/spark/pull/49476#issuecomment-2588795124
Merged to master for Apache Spark 4.0.0. I manually tested. ``` $ build/sbt "core/testOnly org.apache.spark.deploy.history.*" ... [info] RollingEventLogFilesWriterSuite: [info] - create EventLogFileWriter with enable/disable rolling (184 milliseconds) [info] - initialize, write, stop - with codec None (73 milliseconds) [info] - initialize, write, stop - with codec Some(lz4) (176 milliseconds) [info] - initialize, write, stop - with codec Some(lzf) (39 milliseconds) [info] - initialize, write, stop - with codec Some(snappy) (157 milliseconds) [info] - initialize, write, stop - with codec Some(zstd) (44 milliseconds) [info] - Use the default value of spark.eventLog.compression.codec (5 milliseconds) [info] - Event log names (1 millisecond) [info] - Log overwriting (38 milliseconds) [info] - rolling event log files - codec None (177 milliseconds) [info] - rolling event log files - codec Some(lz4) (95 milliseconds) [info] - rolling event log files - codec Some(lzf) (92 milliseconds) [info] - rolling event log files - codec Some(snappy) (87 milliseconds) [info] - rolling event log files - codec Some(zstd) (91 milliseconds) [info] - rolling event log files - the max size of event log file size less than lower limit (6 milliseconds) [info] RocksDBBackendChromeUIHistoryServerSuite: [info] LevelDBBackendChromeUIHistoryServerSuite: [info] LevelDBBackendFsHistoryProviderSuite: [info] RocksDBBackendWithProtobufSerializerSuite: [info] - Parse application logs (inMemory = true) (196 milliseconds) [info] - Parse application logs (inMemory = false) (779 milliseconds) [info] - SPARK-31608: parse application logs with HybridStore (297 milliseconds) [info] - SPARK-41685: Verify the configurable serializer for history server (2 milliseconds) [info] - SPARK-3697: ignore files that cannot be read. (79 milliseconds) [info] - history file is renamed from inprogress to completed (70 milliseconds) [info] - SPARK-39439: Check final file if in-progress event log file does not exist (62 milliseconds) [info] - Parse logs that application is not started (57 milliseconds) [info] - SPARK-5582: empty log directory (59 milliseconds) [info] - apps with multiple attempts with order (426 milliseconds) [info] - log urls without customization (225 milliseconds) [info] - custom log urls, including FILE_NAME (151 milliseconds) [info] - custom log urls, excluding FILE_NAME (157 milliseconds) [info] - custom log urls with invalid attribute (161 milliseconds) [info] - custom log urls, LOG_FILES not available while FILE_NAME is specified (164 milliseconds) [info] - custom log urls, app not finished, applyIncompleteApplication: true (176 milliseconds) [info] - custom log urls, app not finished, applyIncompleteApplication: false (188 milliseconds) [info] - log cleaner (73 milliseconds) [info] - should not clean inprogress application with lastUpdated time less than maxTime (69 milliseconds) [info] - log cleaner for inProgress files (62 milliseconds) [info] - Event log copy (67 milliseconds) [info] - driver log cleaner (9 milliseconds) [info] - SPARK-8372: new logs with no app ID are ignored (56 milliseconds) OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended [info] - provider correctly checks whether fs is in safe mode (724 milliseconds) [info] - provider waits for safe mode to finish before initializing (88 milliseconds) [info] - provider reports error after FS leaves safe mode (106 milliseconds) [info] - ignore hidden files (89 milliseconds) [info] - support history server ui admin acls (482 milliseconds) [info] - mismatched version discards old listing (149 milliseconds) [info] - invalidate cached UI (245 milliseconds) [info] - clean up stale app information (181 milliseconds) [info] - SPARK-21571: clean up removes invalid history files (63 milliseconds) [info] - always find end event for finished apps (66 milliseconds) [info] - parse event logs with optimizations off (60 milliseconds) [info] - SPARK-24948: ignore files we don't have read permission on (118 milliseconds) [info] - check in-progress event logs absolute length (139 milliseconds) [info] - log cleaner with the maximum number of log files (384 milliseconds) [info] - backwards compatibility with LogInfo from Spark 2.4 (5 milliseconds) [info] - SPARK-29755 LogInfo should be serialized/deserialized by jackson properly (1 millisecond) [info] - SPARK-29755 AttemptInfoWrapper should be serialized/deserialized by jackson properly (2 milliseconds) [info] - SPARK-29043: clean up specified event log (59 milliseconds) [info] - compact event log files (134 milliseconds) [info] - SPARK-33146: don't let one bad rolling log folder prevent loading other applications (71 milliseconds) [info] - SPARK-39225: Support spark.history.fs.update.batchSize (98 milliseconds) [info] - SPARK-36354: EventLogFileReader should skip rolling event log directories with no logs (42 milliseconds) [info] - SPARK-33215: check ui view permissions without retrieving ui (89 milliseconds) [info] - SPARK-41447: Reduce the number of doMergeApplicationListing invocations (54 milliseconds) [info] BasicEventFilterBuilderSuite: [info] - track live jobs (7 milliseconds) [info] - track live executors (1 millisecond) [info] LevelDBHybridStoreSuite: [info] RocksDBBackendFsHistoryProviderSuite: [info] - Parse application logs (inMemory = true) (22 milliseconds) [info] - Parse application logs (inMemory = false) (312 milliseconds) [info] - SPARK-31608: parse application logs with HybridStore (276 milliseconds) [info] - SPARK-41685: Verify the configurable serializer for history server (1 millisecond) [info] - SPARK-3697: ignore files that cannot be read. (88 milliseconds) [info] - history file is renamed from inprogress to completed (65 milliseconds) [info] - SPARK-39439: Check final file if in-progress event log file does not exist (62 milliseconds) [info] - Parse logs that application is not started (69 milliseconds) [info] - SPARK-5582: empty log directory (56 milliseconds) [info] - apps with multiple attempts with order (474 milliseconds) [info] - log urls without customization (185 milliseconds) [info] - custom log urls, including FILE_NAME (149 milliseconds) [info] - custom log urls, excluding FILE_NAME (151 milliseconds) [info] - custom log urls with invalid attribute (157 milliseconds) [info] - custom log urls, LOG_FILES not available while FILE_NAME is specified (148 milliseconds) [info] - custom log urls, app not finished, applyIncompleteApplication: true (144 milliseconds) [info] - custom log urls, app not finished, applyIncompleteApplication: false (157 milliseconds) [info] - log cleaner (66 milliseconds) [info] - should not clean inprogress application with lastUpdated time less than maxTime (68 milliseconds) [info] - log cleaner for inProgress files (66 milliseconds) [info] - Event log copy (68 milliseconds) [info] - driver log cleaner (9 milliseconds) [info] - SPARK-8372: new logs with no app ID are ignored (58 milliseconds) [info] - provider correctly checks whether fs is in safe mode (61 milliseconds) [info] - provider waits for safe mode to finish before initializing (60 milliseconds) [info] - provider reports error after FS leaves safe mode (65 milliseconds) [info] - ignore hidden files (77 milliseconds) [info] - support history server ui admin acls (486 milliseconds) [info] - mismatched version discards old listing (146 milliseconds) [info] - invalidate cached UI (298 milliseconds) [info] - clean up stale app information (191 milliseconds) [info] - SPARK-21571: clean up removes invalid history files (63 milliseconds) [info] - always find end event for finished apps (61 milliseconds) [info] - parse event logs with optimizations off (59 milliseconds) [info] - SPARK-24948: ignore files we don't have read permission on (70 milliseconds) [info] - check in-progress event logs absolute length (52 milliseconds) [info] - log cleaner with the maximum number of log files (371 milliseconds) [info] - backwards compatibility with LogInfo from Spark 2.4 (1 millisecond) [info] - SPARK-29755 LogInfo should be serialized/deserialized by jackson properly (1 millisecond) [info] - SPARK-29755 AttemptInfoWrapper should be serialized/deserialized by jackson properly (1 millisecond) [info] - SPARK-29043: clean up specified event log (59 milliseconds) [info] - compact event log files (113 milliseconds) [info] - SPARK-33146: don't let one bad rolling log folder prevent loading other applications (66 milliseconds) [info] - SPARK-39225: Support spark.history.fs.update.batchSize (87 milliseconds) [info] - SPARK-36354: EventLogFileReader should skip rolling event log directories with no logs (34 milliseconds) [info] - SPARK-33215: check ui view permissions without retrieving ui (79 milliseconds) [info] - SPARK-41447: Reduce the number of doMergeApplicationListing invocations (54 milliseconds) [info] HistoryServerDiskManagerUseLevelDBSuite: [info] HistoryServerArgumentsSuite: [info] - No Arguments Parsing (2 milliseconds) [info] - Properties File Arguments Parsing --properties-file (1 millisecond) [info] - SPARK-48471: all history configurations should have documentations (2 milliseconds) [info] RollingEventLogFilesReaderSuite: [info] - Retrieve EventLogFileReader correctly (55 milliseconds) [info] - get information, list event log files, zip log files - with codec None (38 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(lz4) (39 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(lzf) (33 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(snappy) (32 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(zstd) (32 milliseconds) [info] - SPARK-46012: appStatus file should exist (65 milliseconds) [info] - rolling event log files - codec None (272 milliseconds) [info] - rolling event log files - codec Some(lz4) (63 milliseconds) [info] - rolling event log files - codec Some(lzf) (62 milliseconds) [info] - rolling event log files - codec Some(snappy) (69 milliseconds) [info] - rolling event log files - codec Some(zstd) (62 milliseconds) [info] BasicEventFilterSuite: [info] - filter out events for finished jobs (1 millisecond) [info] - accept all events for block manager addition/removal on driver (1 millisecond) [info] - filter out events for dead executors (1 millisecond) [info] - other events should be left to other filters (0 milliseconds) [info] RocksDBHybridStoreSuite: [info] - test multiple objects write read delete (5 milliseconds) [info] - test metadata (0 milliseconds) [info] - test update (1 millisecond) [info] - test basic iteration (2 milliseconds) [info] - test delete after switch (1 millisecond) [info] - test klassMap (3 milliseconds) [info] RocksDBBackendHistoryServerSuite: [info] - application list json (382 milliseconds) [info] - completed app list json (15 milliseconds) [info] - running app list json (5 milliseconds) [info] - minDate app list json (6 milliseconds) [info] - maxDate app list json (4 milliseconds) [info] - maxDate2 app list json (3 milliseconds) [info] - minEndDate app list json (4 milliseconds) [info] - maxEndDate app list json (3 milliseconds) [info] - minEndDate and maxEndDate app list json (2 milliseconds) [info] - minDate and maxEndDate app list json (2 milliseconds) [info] - limit app list json (4 milliseconds) [info] - one app json (15 milliseconds) [info] - one app multi-attempt json (3 milliseconds) [info] - job list json (181 milliseconds) [info] - job list from multi-attempt app json(1) (115 milliseconds) [info] - job list from multi-attempt app json(2) (106 milliseconds) [info] - one job json (2 milliseconds) [info] - succeeded job list json (2 milliseconds) [info] - succeeded&failed job list json (4 milliseconds) [info] - executor list json (4 milliseconds) [info] - executor list with executor metrics json (152 milliseconds) [info] - stage list json (17 milliseconds) [info] - complete stage list json (4 milliseconds) [info] - failed stage list json (3 milliseconds) [info] - one stage json (24 milliseconds) [info] - one stage json with details (7 milliseconds) [info] - one stage attempt json (4 milliseconds) [info] - one stage attempt json details with failed task (2 milliseconds) [info] - one stage json with partitionId (154 milliseconds) [info] - stage task summary w shuffle write (156 milliseconds) [info] - stage task summary w shuffle read (8 milliseconds) [info] - stage task summary w/ custom quantiles (11 milliseconds) [info] - stage task list (7 milliseconds) [info] - stage task list w/ offset & length (9 milliseconds) [info] - stage task list w/ sortBy (5 milliseconds) [info] - stage task list w/ sortBy short names: -runtime (5 milliseconds) [info] - stage task list w/ sortBy short names: runtime (4 milliseconds) [info] - stage task list w/ status (130 milliseconds) [info] - stage task list w/ status & offset & length (2 milliseconds) [info] - stage task list w/ status & sortBy short names: runtime (3 milliseconds) [info] - stage task list with partitionId (2 milliseconds) [info] - stage list with accumulable json (5 milliseconds) [info] - stage with accumulable json (5 milliseconds) [info] - stage task list from multi-attempt app json(1) (3 milliseconds) [info] - stage task list from multi-attempt app json(2) (5 milliseconds) [info] - excludeOnFailure for stage (127 milliseconds) [info] - excludeOnFailure node for stage (135 milliseconds) [info] - rdd list storage json (5 milliseconds) [info] - executor node excludeOnFailure (123 milliseconds) [info] - executor node excludeOnFailure unexcluding (2 milliseconds) [info] - executor memory usage (1 millisecond) [info] - executor resource information (137 milliseconds) [info] - multiple resource profiles (117 milliseconds) [info] - stage list with peak metrics (146 milliseconds) [info] - stage with peak metrics (6 milliseconds) [info] - stage with summaries (16 milliseconds) [info] - app environment (7 milliseconds) [info] - one rdd storage json (2 milliseconds) [info] - miscellaneous process (2 milliseconds) [info] - stage with speculation summary (131 milliseconds) [info] - download all logs for app with multiple attempts (11 milliseconds) [info] - download one log for app with multiple attempts (12 milliseconds) [info] - response codes on bad paths (17 milliseconds) [info] - automatically retrieve uiRoot from request through Knox (46 milliseconds) [info] - static relative links are prefixed with uiRoot (spark.ui.proxyBase) (2 milliseconds) [info] - /version api endpoint (2 milliseconds) [info] - security manager starts with spark.authenticate set (5 milliseconds) [info] - incomplete apps get refreshed (6 seconds, 603 milliseconds) [info] - ui and api authorization checks (2 seconds, 501 milliseconds) [info] - SPARK-33215: speed up event log download by skipping UI rebuild (223 milliseconds) [info] - access history application defaults to the last attempt id (302 milliseconds) [info] - Redirect URLs should end with a slash (1 millisecond) [info] - SPARK-31697: HistoryServer should set Content-Type (1 millisecond) [info] - Redirect to the root page when accessed to /history/ (1 millisecond) [info] HistoryServerDiskManagerUseRocksDBSuite: [info] - leasing space (44 milliseconds) [info] - tracking active stores (5 milliseconds) [info] - approximate size heuristic (1 millisecond) [info] - SPARK-32024: update ApplicationStoreInfo.size during initializing (7 milliseconds) [info] - SPARK-38095: appStorePath should use backend extensions (1 millisecond) [info] SingleFileEventLogFileReaderSuite: [info] - Retrieve EventLogFileReader correctly (54 milliseconds) [info] - get information, list event log files, zip log files - with codec None (11 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(lz4) (10 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(lzf) (9 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(snappy) (9 milliseconds) [info] - get information, list event log files, zip log files - with codec Some(zstd) (10 milliseconds) [info] EventLogFileCompactorSuite: [info] - No event log files (1 millisecond) [info] - No compact file, less origin files available than max files to retain (32 milliseconds) [info] - No compact file, more origin files available than max files to retain (46 milliseconds) [info] - compact file exists, less origin files available than max files to retain (33 milliseconds) [info] - compact file exists, number of origin files are same as max files to retain (39 milliseconds) [info] - compact file exists, more origin files available than max files to retain (68 milliseconds) [info] - events for finished job are dropped in new compact file (48 milliseconds) [info] - Don't compact file if score is lower than threshold (61 milliseconds) [info] - rewrite files with test filters (19 milliseconds) [info] SingleEventLogFileWriterSuite: [info] - create EventLogFileWriter with enable/disable rolling (10 milliseconds) [info] - initialize, write, stop - with codec None (9 milliseconds) [info] - initialize, write, stop - with codec Some(lz4) (9 milliseconds) [info] - initialize, write, stop - with codec Some(lzf) (9 milliseconds) [info] - initialize, write, stop - with codec Some(snappy) (8 milliseconds) [info] - initialize, write, stop - with codec Some(zstd) (9 milliseconds) [info] - Use the default value of spark.eventLog.compression.codec (3 milliseconds) [info] - Log overwriting (14 milliseconds) [info] - Event log name (0 milliseconds) [info] HistoryServerPageSuite: [info] - SPARK-39620: should behaves the same as REST API when filtering applications (4 seconds, 339 milliseconds) [info] - SPARK-49128: Support custom History Server UI title (4 seconds, 116 milliseconds) [info] HistoryServerMemoryManagerSuite: [info] - lease and release memory (4 milliseconds) [info] ApplicationCacheSuite: [info] - Completed UI get (68 milliseconds) [info] - Test that if an attempt ID is set, it must be used in lookups (0 milliseconds) [info] - Incomplete apps refreshed (1 millisecond) [info] - Large Scale Application Eviction (17 milliseconds) [info] - Attempts are Evicted (1 millisecond) [info] - redirect includes query params (61 milliseconds) [info] - SPARK-43403: Load new SparkUI during old one is detaching (5 seconds, 296 milliseconds) [info] LevelDBBackendHistoryServerSuite: [info] LevelDBBackendWithProtobufSerializerSuite: [info] Run completed in 43 seconds, 649 milliseconds. [info] Total number of tests run: 249 [info] Suites: completed 23, aborted 0 [info] Tests: succeeded 249, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 52 s, completed Jan 13, 2025, 6:50:53 PM ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
