[
https://issues.apache.org/jira/browse/SPARK-60110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60110:
--------------------------------
Affects Version/s: 4.0.5
4.1.4
4.2.1
4.3.0
4.4.0
3.5.10
(was: 5.0.0)
> Skip event log compaction when replay skips lines over the line-length limit
> ----------------------------------------------------------------------------
>
> Key: SPARK-60110
> URL: https://issues.apache.org/jira/browse/SPARK-60110
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 4.3.0, 4.2.1, 4.0.5, 4.1.4, 3.5.10, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> SPARK-59407 bounded event log replay with
> spark.history.fs.eventLog.maxLineLength, but event log compaction only
> applies the limit in its first phase.
> EventLogFileCompactor.initializeBuilders replays through ReplayListenerBus,
> which skips over-long lines, while EventFilter.applyFilterToFile reads every
> line unbounded during rewrite.
> When a skipped line is a live job's SparkListenerJobStart (or a live
> SparkListenerSQLExecutionStart), the filter builders never learn the job is
> live, so the rewrite drops the job start and all of its stage and task
> events, and the original files are then deleted, losing the data permanently.
> The rewrite phase also bypasses the memory bound.
> This only happens when compaction is enabled
> (spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
> limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
