[
https://issues.apache.org/jira/browse/SPARK-60109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60109:
--------------------------------
Affects Version/s: 4.2.0
3.5.9
4.1.3
4.0.4
4.3.0
4.4.0
(was: 5.0.0)
> Apply the replay line-length limit and UTF-8-safe line alignment to the
> History Server end-event reparse
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60109
> URL: https://issues.apache.org/jira/browse/SPARK-60109
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 4.2.0, 3.5.9, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> When building the application listing, FsHistoryProvider looks for the
> application end event by reopening the last event log file, skipping to {{len
> - spark.history.fs.endEventReparseChunkSize}} bytes, wrapping the stream in
> {{Source.fromInputStream(in)(Codec.UTF8).getLines()}}, dropping the first
> (partial) line with {{source.next()}}, and replaying the rest through
> {{ReplayListenerBus.replay(Iterator[String], ...)}}. With the defaults
> (completed apps, 1m chunk), every completed application takes this path, and
> because the target is computed from the compressed length while the skip runs
> on the decompressed stream, most of a compressed log can be read there.
> This path has three problems:
> 1. The Iterator overload bypasses the bounded line reader added by
> SPARK-59407 / SPARK-59804, so {{spark.history.fs.eventLog.maxLineLength}}
> does not apply and an over-long line is fully materialized.
> 2. If the skip target lands inside a multi-byte UTF-8 character (e.g. CJK SQL
> text or job descriptions, which Jackson writes as raw UTF-8),
> {{source.next()}} throws {{MalformedInputException}} before replay starts.
> mergeApplicationListing only logs the error, and since the placeholder
> listing entry was already written with the same file size, the log is never
> retried, so the completed application stays missing from the listing (the
> target is deterministic, so restarting the History Server fails the same way).
> 3. Line numbers in the replay diagnostics (JSON parse errors, "Malformed line
> #N", the truncation warning and the over-long line warning) are relative to
> the skip point but are printed next to the full file path, which makes them
> misleading.
> Proposed fix: after skipping, discard the partial first line at the raw byte
> level (0x0A never occurs inside a multi-byte UTF-8 sequence) before any
> decoding, then replay through the InputStream overload so the line-length
> limit and strict decoding apply, and label the source name with the byte
> offset that line numbers are counted from.
> This is the tail-reparse follow-up discussed in
> https://github.com/apache/spark/pull/59074.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
