[
https://issues.apache.org/jira/browse/CALCITE-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frank Chen updated CALCITE-7805:
--------------------------------
Description:
[CALCITE-7782|https://issues.apache.org/jira/browse/CALCITE-7782] eliminated
quadratic operand-list copying while validating large ARRAY and MAP
constructors, allowing a 1,000,000-element string ARRAY to complete. Planning
remains allocation-heavy: Druid's benchmark allocates approximately 15.4 GB and
takes approximately 7.8 seconds for one million string literals. Allocation is
cumulative bytes per operation, not retained or peak heap.
h2. Reproducer
{code:sql}
SELECT string1
FROM foo
WHERE string1 IN ('1', '2', ..., '1000000')
{code}
Druid rewrites this predicate to:
{code}
SCALAR_IN_ARRAY(string1, ARRAY['1', '2', ..., '1000000'])
{code}
The benchmark is {{InPlanningBenchmark.queryStringInSqlPlanOnly}}.
h2. Work items
The remaining allocation sources are tracked independently:
* [CALCITE-7806|https://issues.apache.org/jira/browse/CALCITE-7806] - Avoid
resolving generated CAST calls during ARRAY validation.
* [CALCITE-7807|https://issues.apache.org/jira/browse/CALCITE-7807] - Reuse
unchanged SQL types during least-restrictive inference.
* [CALCITE-7808|https://issues.apache.org/jira/browse/CALCITE-7808] - Cache
recently created and decorated SQL types.
* [CALCITE-7809|https://issues.apache.org/jira/browse/CALCITE-7809] - Avoid
copying unchanged SQL and Rex operands.
* [CALCITE-7810|https://issues.apache.org/jira/browse/CALCITE-7810] - Write Rex
string literal digests directly.
Combined prototypes reduced allocation by approximately 70% in this Druid
workload. Each sub-task contains its isolated measurement and scope.
was:
CALCITE-7782 and [PR #5263|https://github.com/apache/calcite/pull/5263]
eliminated quadratic operand-list copying while validating large ARRAY and MAP
constructors. That correction allows a 1,000,000-element string ARRAY to
complete instead of running for more than 20 minutes.
After that correction, validation is linear, but planning still creates many
temporary objects for every string literal. In the benchmark below, planning a
1,000,000-element string IN expression allocates approximately 15.4 GB during
one operation and takes approximately 7.8 seconds.
The allocation values below are cumulative allocation during one planning
operation, not retained memory or peak heap usage.
h2. Reproducer
Apache Druid exposes the remaining allocation through a large string IN
predicate:
{code:sql}
SELECT string1
FROM foo
WHERE string1 IN ('1', '2', ..., '1000000')
{code}
Druid rewrites the predicate to an expression containing a string ARRAY:
{code}
SCALAR_IN_ARRAY(string1, ARRAY['1', '2', ..., '1000000'])
{code}
The benchmark is {{InPlanningBenchmark.queryStringInSqlPlanOnly}}.
Parameters:
* {{inSubQueryThreshold = 2147483647}}
* {{rowsPerSegment = 500000}}
* 2 JMH forks
* 2 one-second warmup iterations per fork
* 5 one-second measurement iterations per fork
* GC profiler enabled
* Temurin 25.0.4.1
The Calcite baseline is commit {{38413ece6}}, which includes the merged
CALCITE-7782 change.
h2. Proposed delivery
The implementation is split into five independently reviewable sub-tasks:
# [CALCITE-7806|https://issues.apache.org/jira/browse/CALCITE-7806] - Avoid
resolving generated CAST calls during ARRAY validation.
# [CALCITE-7807|https://issues.apache.org/jira/browse/CALCITE-7807] - Reuse
unchanged SQL types during least-restrictive inference.
# [CALCITE-7808|https://issues.apache.org/jira/browse/CALCITE-7808] - Cache
recently created and decorated SQL types.
# [CALCITE-7809|https://issues.apache.org/jira/browse/CALCITE-7809] - Avoid
copying unchanged SQL and Rex operands.
# [CALCITE-7810|https://issues.apache.org/jira/browse/CALCITE-7810] - Write Rex
string literal digests directly.
Each sub-task links its corresponding pull request and can be reviewed and
tested independently.
h2. Combined benchmark
When changes are stacked together, the benchmark shows about 70% allocation
saving.
|| String literals || CALCITE-7782 baseline || All proposed changes ||
Allocation saved ||
| 100,000 | 1.522 GB/op | 0.445 GB/op | 1.077 GB/op (70.8%) |
| 1,000,000 | 15.418 GB/op | 4.556 GB/op | 10.862 GB/op (70.5%) |
For each change, the separated benchmark data is attached in their descriptions
> Reduce temporary object allocation when validating large string ARRAYs to
> lower GC pressure
> -------------------------------------------------------------------------------------------
>
> Key: CALCITE-7805
> URL: https://issues.apache.org/jira/browse/CALCITE-7805
> Project: Calcite
> Issue Type: Improvement
> Components: core
> Affects Versions: 1.42.0
> Reporter: Frank Chen
> Priority: Major
> Labels: pull-request-available
>
> [CALCITE-7782|https://issues.apache.org/jira/browse/CALCITE-7782] eliminated
> quadratic operand-list copying while validating large ARRAY and MAP
> constructors, allowing a 1,000,000-element string ARRAY to complete. Planning
> remains allocation-heavy: Druid's benchmark allocates approximately 15.4 GB
> and takes approximately 7.8 seconds for one million string literals.
> Allocation is cumulative bytes per operation, not retained or peak heap.
> h2. Reproducer
> {code:sql}
> SELECT string1
> FROM foo
> WHERE string1 IN ('1', '2', ..., '1000000')
> {code}
> Druid rewrites this predicate to:
> {code}
> SCALAR_IN_ARRAY(string1, ARRAY['1', '2', ..., '1000000'])
> {code}
> The benchmark is {{InPlanningBenchmark.queryStringInSqlPlanOnly}}.
> h2. Work items
> The remaining allocation sources are tracked independently:
> * [CALCITE-7806|https://issues.apache.org/jira/browse/CALCITE-7806] - Avoid
> resolving generated CAST calls during ARRAY validation.
> * [CALCITE-7807|https://issues.apache.org/jira/browse/CALCITE-7807] - Reuse
> unchanged SQL types during least-restrictive inference.
> * [CALCITE-7808|https://issues.apache.org/jira/browse/CALCITE-7808] - Cache
> recently created and decorated SQL types.
> * [CALCITE-7809|https://issues.apache.org/jira/browse/CALCITE-7809] - Avoid
> copying unchanged SQL and Rex operands.
> * [CALCITE-7810|https://issues.apache.org/jira/browse/CALCITE-7810] - Write
> Rex string literal digests directly.
> Combined prototypes reduced allocation by approximately 70% in this Druid
> workload. Each sub-task contains its isolated measurement and scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
