[
https://issues.apache.org/jira/browse/SPARK-60158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-60158.
---------------------------------
Fix Version/s: 4.2.2
4.1.5
4.0.6
4.3.0
3.5.10
Resolution: Fixed
Issue resolved by pull request 59360
[https://github.com/apache/spark/pull/59360]
> Avoid per-character synchronized reads in the bounded event log line reader
> ---------------------------------------------------------------------------
>
> Key: SPARK-60158
> URL: https://issues.apache.org/jira/browse/SPARK-60158
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.2, 4.1.5, 4.0.6, 4.3.0, 3.5.10
>
>
> SPARK-59407 replaced {{Source.getLines()}} in
> {{ReplayListenerBus.replay(InputStream, ...)}} with a bounded line reader so
> that over-long event log lines are skipped instead of materialized
> (SPARK-59804 later fixed its byte accounting). The reader calls
> {{BufferedReader.read()}} once per character, and {{BufferedReader.read()}}
> takes a lock on every call, so splitting lines is several times slower than
> the previous {{readLine()}}-based reading. In the review of SPARK-59804
> (https://github.com/apache/spark/pull/59074), line splitting was measured at
> about 14x slower.
> This slows down every replay through the {{InputStream}} overload: loading
> application UIs in the History Server, event log compaction, and listing with
> fast in-progress parsing. SPARK-60109 also routes the History Server
> end-event reparse through this reader. For compressed logs, that reparse
> reads most of the decompressed file, because the skip target is computed from
> the compressed length, so it makes the listing of completed applications
> slower as well. In a local measurement of that reparse path (210 MB of
> uncompressed event lines, lz4 and zstd), it took about 0.17 s with
> {{Source.getLines()}} and 0.9-1.05 s with the bounded reader.
> Proposed fix: read decoded characters in bulk (e.g. into a {{char[]}} buffer)
> and scan for '\n' there, instead of calling {{read()}} per character. Keep
> the current semantics: the UTF-8 content byte limit with the trailing-CR
> allowance, physical line indices including skipped lines, strict decoding,
> and releasing the buffer of an over-long line while it is drained. A
> prototype of the bulk scan brought the reparse path above back to about 0.17
> s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
