[
https://issues.apache.org/jira/browse/HIVE-29866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29866:
--------------------------------
Description:
Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
please find Eclipse MAT html report for the same:
[^heap-dump_Top_Components.zip]
h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
When LLAP ingests text and writes it into its ORC-encoded cache, the routing
gate in
[{{VectorDeserializeOrcWriter.create()}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java#L101]
today is:
{code:java}
if (... || !(serDe instanceof LazySimpleSerDe)) {
return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
}
{code}
({{DeserializerOrcWriter}} lives as an inner class in
[{{SerDeEncodedDataReader}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java#L1564].)
{{MultiDelimitSerDe}} fails the check, so every row travels through
{{DeserializerOrcWriter}}, the generic per-row fallback. For each row it:
# Calls {{serDe.deserialize(Writable)}} → a fresh
*[{{LazyStruct}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java]*
wrapping…
# …a fresh
*[{{LazyString}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java]
/
[{{LazyInteger}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyInteger.java]
/ …* object per column (writable + underlying byte buffer refs).
# Walks the struct through an {{ObjectInspector}} — every field getter
allocates a *{{Text}}/{{IntWritable}}/…* to hand to the ORC writer.
# ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
For a wide row (thousands of columns), that's *~1 struct + ~N lazy field
objects + ~N writables per row*, all short-lived. Combined with LLAP's normal
cache-fill pressure, the daemon runs out of heap.
The vectorized path avoids all of this:
[{{LazySimpleDeserializeRead.topLevelParse()}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/LazySimpleDeserializeRead.java#L419]
scans the raw bytes into a reused
[{{VectorizedRowBatch}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java#L40]
([{{DEFAULT_SIZE=1024}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java#L66]
rows at a time, fixed-size column vectors, *zero per-row allocation*). Wiring
MultiDelimit into that path is what the fix does.
please also find:
!Screenshot 2026-09-03 at 13.56.34.png|width=783,height=130!
12 instances most probably belong to 12 IO threads in the LLAP daemon where
this investigation took place
was:
Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
please find Eclipse MAT html report for the same:
[^heap-dump_Top_Components.zip]
h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
When LLAP ingests text and writes it into its ORC-encoded cache, the routing
gate in {{VectorDeserializeOrcWriter.create()}} today is:
{code:java}
if (!(serDe instanceof LazySimpleSerDe)) {
return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
}
{code}
{{MultiDelimitSerDe}} fails the check, so every row travels through
{{{}DeserializerOrcWriter{}}}, the generic per-row fallback. For each row it:
# Calls {{serDe.deserialize(Writable)}} → a fresh *{{LazyStruct}}* wrapping…
# …a fresh *{{{}LazyString{}}}/{{{}LazyInteger{}}}/…* object per column
(writable + underlying byte buffer refs).
# Walks the struct through an {{ObjectInspector}} — every field getter
allocates a *{{{}Text{}}}/{{{}IntWritable{}}}/…* to hand to the ORC writer.
# ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
For a 3000-column row, that's {*}~1 struct + ~3000 lazy field objects + ~3000
writables per row{*}, all short-lived. At ingest rates of tens of thousands of
rows/s per LLAP daemon:
* Young gen fills in seconds → back-to-back {*}minor GCs{*}.
* Objects that survive one collection under load get *promoted to old gen*
long before they'd naturally die → old gen fills → {*}long CMS/G1 mixed
collections{*}, then {*}full GC{*}, then {*}OOM{*}.
* Add cache pressure (LLAP is also holding encoded ORC chunks in the SSD/data
cache) and the heap has no headroom left. The daemon dies with
{{OutOfMemoryError: Java heap space}} — sometimes with {{GC overhead limit
exceeded}} first if it lingers.
The vectorized path avoids all of this: {{LazySimpleDeserializeRead}} scans the
raw bytes into a reused {{VectorizedRowBatch}} (1024 rows at a time, fixed-size
column vectors, {*}zero per-row allocation{*}). Wiring MultiDelimit into that
path is what the fix does.
please also find:
!Screenshot 2026-09-03 at 13.56.34.png|width=783,height=130!
12 instances most probably belong to 12 IO threads in the LLAP daemon where
this investigation took place
> Optimize LLAP IO cache encoding path for text tables with MultiDelimitSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-29866
> URL: https://issues.apache.org/jira/browse/HIVE-29866
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2026-09-03 at 13.56.34.png,
> heap-dump_Top_Components.zip
>
>
> Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
> please find Eclipse MAT html report for the same:
> [^heap-dump_Top_Components.zip]
> h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
> When LLAP ingests text and writes it into its ORC-encoded cache, the routing
> gate in
> [{{VectorDeserializeOrcWriter.create()}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java#L101]
> today is:
> {code:java}
> if (... || !(serDe instanceof LazySimpleSerDe)) {
> return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
> }
> {code}
> ({{DeserializerOrcWriter}} lives as an inner class in
> [{{SerDeEncodedDataReader}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java#L1564].)
> {{MultiDelimitSerDe}} fails the check, so every row travels through
> {{DeserializerOrcWriter}}, the generic per-row fallback. For each row it:
> # Calls {{serDe.deserialize(Writable)}} → a fresh
> *[{{LazyStruct}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java]*
> wrapping…
> # …a fresh
> *[{{LazyString}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java]
> /
> [{{LazyInteger}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyInteger.java]
> / …* object per column (writable + underlying byte buffer refs).
> # Walks the struct through an {{ObjectInspector}} — every field getter
> allocates a *{{Text}}/{{IntWritable}}/…* to hand to the ORC writer.
> # ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
> For a wide row (thousands of columns), that's *~1 struct + ~N lazy field
> objects + ~N writables per row*, all short-lived. Combined with LLAP's normal
> cache-fill pressure, the daemon runs out of heap.
> The vectorized path avoids all of this:
> [{{LazySimpleDeserializeRead.topLevelParse()}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/LazySimpleDeserializeRead.java#L419]
> scans the raw bytes into a reused
> [{{VectorizedRowBatch}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java#L40]
>
> ([{{DEFAULT_SIZE=1024}}|https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java#L66]
> rows at a time, fixed-size column vectors, *zero per-row allocation*).
> Wiring MultiDelimit into that path is what the fix does.
> please also find:
> !Screenshot 2026-09-03 at 13.56.34.png|width=783,height=130!
> 12 instances most probably belong to 12 IO threads in the LLAP daemon where
> this investigation took place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
