Yiping Deng created SPARK-59747:
-----------------------------------
Summary: History Server repeatedly retries compaction of rolled
event log files that contain malformed input
Key: SPARK-59747
URL: https://issues.apache.org/jira/browse/SPARK-59747
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 4.2.0, 4.0.1, 4.1.0
Reporter: Yiping Deng
When the rolled event log files that are eligible for compaction contain bytes
that are not valid UTF-8, EventLogFileCompactor fails with
java.nio.charset.MalformedInputException (a CharacterCodingException) while
replaying them to initialize the event filter builders.
FsHistoryProvider.compact() only records lastEvaluatedForCompaction after
compaction succeeds, and the exception reaches the generic handler, which logs
it at ERROR:
{code:java}
ERROR FsHistoryProvider: Exception while compacting log for
file:/.../eventlog_v2_app
java.nio.charset.MalformedInputException: Input length = 1{code}
The listing replay stops early, so it never reaches the malformed bytes and the
application keeps being listed. Each time the last (non-compacted) event log
file grows, the log is reprocessed and compaction of the same unchanged rolled
files is tried again. For a long-running application this logs a full stack
trace at ERROR on every scan until a new event log file is rolled. Rolled files
are no longer modified, so the retry can never succeed.
Proposed fix: when compaction fails with CharacterCodingException, log a WARN
and still record the index as evaluated, so compaction of the same files is not
retried until a new event log file is rolled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
