L. C. Hsieh created SPARK-60110:
-----------------------------------
Summary: Skip event log compaction when replay skips lines over
the line-length limit
Key: SPARK-60110
URL: https://issues.apache.org/jira/browse/SPARK-60110
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
SPARK-59407 bounded event log replay with
spark.history.fs.eventLog.maxLineLength, but event log compaction only applies
the limit in its first phase. EventLogFileCompactor.initializeBuilders replays
through ReplayListenerBus, which skips over-long lines, while
EventFilter.applyFilterToFile reads every line unbounded during rewrite.
When a skipped line is a live job's SparkListenerJobStart (or a live
SparkListenerSQLExecutionStart), the filter builders never learn the job is
live, so the rewrite drops the job start and all of its stage and task events,
and the original files are then deleted, losing the data permanently. The
rewrite phase also bypasses the memory bound.
This only happens when compaction is enabled
(spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
