[
https://issues.apache.org/jira/browse/HIVE-29866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29866:
--------------------------------
Description:
Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
please find Eclipse MAT html report for the same:
[^heap-dump_Top_Components.zip]
h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
When LLAP ingests text and writes it into its ORC-encoded cache, the routing
gate in {{VectorDeserializeOrcWriter.create()}} today is:
{code:java}
if (!(serDe instanceof LazySimpleSerDe)) {
return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
}
{code}
{{MultiDelimitSerDe}} fails the check, so every row travels through
{{DeserializerOrcWriter}}, the generic per-row fallback. For each row it:
# Calls {{serDe.deserialize(Writable)}} → a fresh *{{LazyStruct}}* wrapping…
# …a fresh *{{LazyString}}/{{LazyInteger}}/…* object per column (writable +
underlying byte buffer refs).
# Walks the struct through an {{ObjectInspector}} — every field getter
allocates a *{{Text}}/{{IntWritable}}/…* to hand to the ORC writer.
# ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
For a 3000-column row, that's *~1 struct + ~3000 lazy field objects + ~3000
writables per row*, all short-lived. At ingest rates of tens of thousands of
rows/s per LLAP daemon:
* Young gen fills in seconds → back-to-back *minor GCs*.
* Objects that survive one collection under load get *promoted to old gen* long
before they'd naturally die → old gen fills → *long CMS/G1 mixed collections*,
then *full GC*, then *OOM*.
* Add cache pressure (LLAP is also holding encoded ORC chunks in the SSD/data
cache) and the heap has no headroom left. The daemon dies with
{{OutOfMemoryError: Java heap space}} — sometimes with {{GC overhead limit
exceeded}} first if it lingers.
The vectorized path avoids all of this: {{LazySimpleDeserializeRead}} scans the
raw bytes into a reused {{VectorizedRowBatch}} (1024 rows at a time, fixed-size
column vectors, *zero per-row allocation*). Wiring MultiDelimit into that path
is what the fix does.
was:
Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
please find Eclipse MAT html report [^heap-dump_Top_Components.zip]
h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
When LLAP ingests text and writes it into its ORC-encoded cache, the routing
gate in {{VectorDeserializeOrcWriter.create()}} today is:
{code:java}
if (!(serDe instanceof LazySimpleSerDe)) {
return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
}
{code}
{{MultiDelimitSerDe}} fails the check, so every row travels through
{{DeserializerOrcWriter}}, the generic per-row fallback. For each row it:
# Calls {{serDe.deserialize(Writable)}} → a fresh *{{LazyStruct}}* wrapping…
# …a fresh *{{LazyString}}/{{LazyInteger}}/…* object per column (writable +
underlying byte buffer refs).
# Walks the struct through an {{ObjectInspector}} — every field getter
allocates a *{{Text}}/{{IntWritable}}/…* to hand to the ORC writer.
# ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
For a 3000-column row, that's *~1 struct + ~3000 lazy field objects + ~3000
writables per row*, all short-lived. At ingest rates of tens of thousands of
rows/s per LLAP daemon:
* Young gen fills in seconds → back-to-back *minor GCs*.
* Objects that survive one collection under load get *promoted to old gen* long
before they'd naturally die → old gen fills → *long CMS/G1 mixed collections*,
then *full GC*, then *OOM*.
* Add cache pressure (LLAP is also holding encoded ORC chunks in the SSD/data
cache) and the heap has no headroom left. The daemon dies with
{{OutOfMemoryError: Java heap space}} — sometimes with {{GC overhead limit
exceeded}} first if it lingers.
The vectorized path avoids all of this: {{LazySimpleDeserializeRead}} scans the
raw bytes into a reused {{VectorizedRowBatch}} (1024 rows at a time, fixed-size
column vectors, *zero per-row allocation*). Wiring MultiDelimit into that path
is what the fix does.
> Optimize LLAP IO cache encoding path for text tables with MultiDelimitSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-29866
> URL: https://issues.apache.org/jira/browse/HIVE-29866
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Attachments: heap-dump_Top_Components.zip
>
>
> Long story short: LLAP OOM while encoding large rows from MultiDelimitSerde,
> please find Eclipse MAT html report for the same:
> [^heap-dump_Top_Components.zip]
> h2. Why MultiDelimitSerDe currently OOMs in LLAP text-cache encoding
> When LLAP ingests text and writes it into its ORC-encoded cache, the routing
> gate in {{VectorDeserializeOrcWriter.create()}} today is:
> {code:java}
> if (!(serDe instanceof LazySimpleSerDe)) {
> return new DeserializerOrcWriter(serDe, sourceOi, allocSize);
> }
> {code}
> {{MultiDelimitSerDe}} fails the check, so every row travels through
> {{DeserializerOrcWriter}}, the generic per-row fallback. For each row it:
> # Calls {{serDe.deserialize(Writable)}} → a fresh *{{LazyStruct}}* wrapping…
> # …a fresh *{{LazyString}}/{{LazyInteger}}/…* object per column (writable +
> underlying byte buffer refs).
> # Walks the struct through an {{ObjectInspector}} — every field getter
> allocates a *{{Text}}/{{IntWritable}}/…* to hand to the ORC writer.
> # ORC's {{TreeWriter}} then re-encodes those objects into columnar buffers.
> For a 3000-column row, that's *~1 struct + ~3000 lazy field objects + ~3000
> writables per row*, all short-lived. At ingest rates of tens of thousands of
> rows/s per LLAP daemon:
> * Young gen fills in seconds → back-to-back *minor GCs*.
> * Objects that survive one collection under load get *promoted to old gen*
> long before they'd naturally die → old gen fills → *long CMS/G1 mixed
> collections*, then *full GC*, then *OOM*.
> * Add cache pressure (LLAP is also holding encoded ORC chunks in the SSD/data
> cache) and the heap has no headroom left. The daemon dies with
> {{OutOfMemoryError: Java heap space}} — sometimes with {{GC overhead limit
> exceeded}} first if it lingers.
> The vectorized path avoids all of this: {{LazySimpleDeserializeRead}} scans
> the raw bytes into a reused {{VectorizedRowBatch}} (1024 rows at a time,
> fixed-size column vectors, *zero per-row allocation*). Wiring MultiDelimit
> into that path is what the fix does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
