[
https://issues.apache.org/jira/browse/IMPALA-13940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115506#comment-18115506
]
Aleksandr Efimov edited comment on IMPALA-13940 at 9/15/26 10:29 AM:
---------------------------------------------------------------------
The uncounted field is {{QueryStateRecord::fragments}}: {{EstimateSize}} counts
it as {{sizeof(TPlanFragment) * fragments.capacity()}}
(query-state-record.cc:208) and does not traverse nested thrift, while
{{stmt}}, {{plan}} and {{compressed_profile}} are all counted at capacity.
A wide {{INSERT ... VALUES}} hits it harder than many fragments do: every
literal lands in {{TUnionNode.const_expr_lists}} while the text plan prints
only {{constant-operands=N}}. On a build from master (2026-08-25) with
{{query_log_size=20000}}, ~500 such inserts moved tcmalloc bytes-in-use by
19.4GiB against 1.05GiB of estimate growth, about 39MiB per record for ~2MB of
estimate.
One dead end worth naming: estimating from the serialized size does not help,
because thrift omits unset optional fields. Storing {{fragments}} serialized,
the way {{compressed_profile}} already is, would make the size exact and
counted.
was (Author: JIRAUSER313444):
The unaccounted field is {{QueryStateRecord::fragments}}. {{EstimateSize}}
counts it as {{sizeof(TPlanFragment) * fragments.capacity()}}
(query-state-record.cc:208) and its own comment says it does not traverse
nested thrift, so everything inside a fragment is invisible to the byte cap.
{{stmt}}, {{plan}} and {{compressed_profile}} are all counted at capacity, so
the blind spot is exactly a workload whose weight sits in the plan rather than
in the text or the profile.
A wide {{INSERT ... VALUES}} hits it harder than many fragments do:
{{ValuesStmt}} plans as a {{UnionNode}}, every literal ends up in
{{TUnionNode.const_expr_lists}}, and the text plan prints only
{{constant-operands=N}}. Production numbers on a build from master
(2026-08-25), dedicated coordinator, {{query_log_size=20000}} and the default
2GiB byte cap: ~500 such inserts moved tcmalloc bytes-in-use by 19.4GiB while
the log estimate grew by 1.05GiB, so about 39MiB per record against ~2MB of
estimate. The coordinator later held 46.5GiB with the estimate at 1.80GiB, so
the byte bound never evicted anything.
On accounting: these bytes are not outside {{mem_limit}}. The process tracker
is built on {{AggregateMemoryMetrics::TOTAL_USED}} (exec-env.cc:636), a sum
over the allocator's used bytes, so they inflate consumption and are charged to
whoever allocates next, which is other queries. What is missing is a consume
against a tracker when the record is built.
Estimating from the serialized size would not close it: thrift omits unset
optional fields, so the wire size of a VALUES plan is about the text size,
which is what the estimate already reports. Storing {{fragments}} serialized in
the record, the way {{compressed_profile}} already is, would make the size
exact and counted, and would collapse the per-node overhead where the factor
lives. The single consumer is {{PlanToJson}} on the debug page
(impala-http-handler.cc:1426), which today copies the whole vector again on
every render.
> Excessive memory use when running targeted-perf's primitive_many_fragments
> --------------------------------------------------------------------------
>
> Key: IMPALA-13940
> URL: https://issues.apache.org/jira/browse/IMPALA-13940
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Attachments: primitive_many_fragments_100.sql
>
>
> When running targeted-perf's primitive_many_fragments test case in a loop in
> a single impala-shell session, I see memory continually increase until it
> hits the memory limit:
> {noformat}
> Query state can be monitored at:
> http://joemcdonnell-22743:25000/query_plan?query_id=514ae852aad5406e:aa67c15900000000
> 2025-04-07 17:14:59 [Exception] ERROR: Query
> 514ae852aad5406e:aa67c15900000000 failed:
> Failed to get minimum memory reservation of 1.11 GB on daemon
> joemcdonnell-22743:27000 for query 514ae852aad5406e:aa67c15900000000 due to
> following error: Memory limit exceeded: Could not allocate memory while
> trying to increase reservation.
> Query(514ae852aad5406e:aa67c15900000000) could not allocate 1.11 GB without
> exceeding limit.
> Error occurred on backend joemcdonnell-22743:27000
> Memory left in process limit: 2.05 GB
> Query(514ae852aad5406e:aa67c15900000000): Reservation=0 ReservationLimit=9.60
> GB OtherMemory=0 Total=0 Peak=0
> Memory is likely oversubscribed. Reducing query concurrency or configuring
> admission control may help avoid this error.
> {noformat}
> [https://github.com/apache/impala/blob/master/testdata/workloads/targeted-perf/queries/primitive_many_fragments.test]
> Untracked Memory: Total=9.05 GB
> Oddly enough, it doesn't fail if I run each query in a separate impala-shell
> invocation. I will attach a repro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
