This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 224ba5f20b8 [improvement](parser) Factor LIMIT clause common prefix
(#67455)
224ba5f20b8 is described below
commit 224ba5f20b88263e460ee73cf3107dd591ce3706
Author: morrySnow <[email protected]>
AuthorDate: Thu Sep 3 10:28:46 2026 +0800
[improvement](parser) Factor LIMIT clause common prefix (#67455)
### What problem does this PR solve?
Problem Summary:
The three `limitClause` alternatives repeat the `LIMIT INTEGER_VALUE`
prefix. ANTLR therefore inspects up to four or five tokens before
selecting an alternative, even though the three forms can be
distinguished after consuming the common prefix.
Factor the prefix once and dispatch on `OFFSET`, comma, or the rule end.
A local grammar action preserves the existing generated `limit` and
`offset` token fields for MySQL's `LIMIT offset, count` form. The
accepted SQL syntax and FE semantics do not change.
This PR also adds reusable JMH workloads for evaluating the P5
local-rule candidates and a focused test for all three LIMIT forms.
Predicate, relation, primary-expression, SHOW, and DDL candidates were
profiled and benchmarked; candidates without a stable compatible
improvement were discarded.
### Benchmark
Environment and method:
- Baseline: `807454f5d92` (benchmark present, before the grammar change)
- Candidate: `cba9bef4b07`
- Baseline parser artifact SHA-256:
`7ab3247fc905dcb519f80fd7a95f3a12e038bdc9d3b25a0825eeda2fb7df321d`
- Candidate parser artifact SHA-256:
`cfa2b289c199bbbb36a42615ce46b6633a0bbe16f595bd178796c976a08c4ebb`
- macOS 15 arm64, OpenJDK 17.0.20.1, JMH 1.37, 1 GiB heap
- 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Runs were interleaved as B1-C1-C2-B2. Values below are `us/op`;
individual values show JMH's 99.9% confidence error.
| Benchmark | Baseline (B1 / B2) |
Candidate (C1 / C2) | Mean Latency Change | Allocation (B/op): Baseline →
Candidate |
| :-------------------- | -----------------------------: |
------------------------------: | ------------------: |
--------------------------------------: |
| Target `limitClause` | 0.2876±0.0398 / 0.2530±0.0288 | 0.2622±0.0304
/ 0.2338±0.0056 | -8.2% | 1,136 → 1,136
|
| LIMIT end-to-end | 341.998±96.097 / 286.276±5.505 | 292.862±5.802 /
303.246±32.851 | -5.1% | 532,244 → 531,343 (-0.2%) |
| LIMIT pre-tokenized | 294.078±23.005 / 284.602±6.383 | 294.394±18.844 /
282.062±17.540 | -0.4% | 447,332 → 447,332 |
| Control end-to-end | 54.008±1.936 / 55.684±4.762 | 53.994±2.805
/ 53.091±2.520 | -2.4% | 107,292 → 105,905 (-1.3%) |
| Control pre-tokenized | 51.786±0.928 / 50.017±0.690 | 51.062±1.075
/ 49.902±0.574 | -0.8% | 97,043 → 97,385 (+0.4%) |
`ProfilingATNSimulator` shows that the three baseline LIMIT forms
consume 128/160/128 lookahead tokens over 32 invocations, with maximum
lookahead 4/5/4. The candidate consumes 32 tokens with maximum lookahead
1. The generated parser class also shrinks by 130 bytes. The improvement
therefore comes from removing the repeated adaptive lookahead;
allocation is effectively unchanged.
Correctness corpus:
- 4,610 tracked SQL files were compared in legacy and ANSI modes (9,220
parser cases).
- Baseline and candidate match for accept/reject, statement count,
exception type, and first error line/position.
- Result signature SHA-256:
`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`.
- Targeted valid and invalid LIMIT error-signature comparisons also have
no differences.
- The lexer grammar is unchanged, so tokenization is unaffected.
---
.../benchmark/LocalRulePrefixBenchmark.java | 193 +++++++++++++++++++++
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 8 +-
.../doris/sqlparser/LimitClausePrefixTest.java | 64 +++++++
3 files changed, 262 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LocalRulePrefixBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LocalRulePrefixBenchmark.java
new file mode 100644
index 00000000000..4e4aa601691
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LocalRulePrefixBenchmark.java
@@ -0,0 +1,193 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ListTokenSource;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures local grammar-prefix changes in isolation and through the public
parser facade. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class LocalRulePrefixBenchmark {
+ private static final int STATEMENT_COUNT = 32;
+
+ @Param({"control", "limit", "predicate", "relation", "tvf", "primary",
"show", "ddl"})
+ public String workload;
+
+ private final DorisSqlParser facade = new DorisSqlParser();
+ private final PostProcessor postProcessor = new PostProcessor();
+ private final ParseErrorListener errorListener = new ParseErrorListener();
+
+ private String sql;
+ private List<Token> tokens;
+ private List<Token> targetTokens;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ sql = repeat(statement(workload));
+ CommonTokenStream stream = new CommonTokenStream(facade.newLexer(sql));
+ stream.fill();
+ tokens = List.copyOf(stream.getTokens());
+ CommonTokenStream targetStream = new
CommonTokenStream(facade.newLexer(target(workload)));
+ targetStream.fill();
+ targetTokens = List.copyOf(targetStream.getTokens());
+ }
+
+ @Benchmark
+ public Object parseEndToEnd() {
+ return facade.parseStatements(sql);
+ }
+
+ @Benchmark
+ public Object parsePreTokenized() {
+ return newParser(tokens).multiStatements();
+ }
+
+ @Benchmark
+ public Object parseTargetRule() {
+ DorisParser parser = newParser(targetTokens);
+ switch (workload) {
+ case "control":
+ return parser.singleStatement();
+ case "limit":
+ return parser.limitClause();
+ case "predicate":
+ return parser.predicate();
+ case "relation":
+ case "tvf":
+ return parser.relationPrimary();
+ case "primary":
+ return parser.primaryExpression();
+ case "show":
+ return parser.showStatement();
+ case "ddl":
+ return parser.createStatement();
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ }
+
+ private DorisParser newParser(List<Token> input) {
+ CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(input));
+ DorisParser parser = new DorisParser(stream);
+ parser.addParseListener(postProcessor);
+ parser.removeErrorListeners();
+ parser.addErrorListener(errorListener);
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parser;
+ }
+
+ private static String statement(String workload) {
+ switch (workload) {
+ case "control":
+ return "SELECT 1";
+ case "limit":
+ return "SELECT c FROM t LIMIT 100; SELECT c FROM t LIMIT 100
OFFSET 20;"
+ + " SELECT c FROM t LIMIT 20, 100";
+ case "predicate":
+ return "SELECT c FROM t WHERE c NOT BETWEEN 1 AND 10"
+ + " AND s NOT LIKE 'x%' ESCAPE '\\\\'"
+ + " AND r NOT REGEXP 'a.*'"
+ + " AND i NOT IN (1, 2, 3)"
+ + " AND j IN (SELECT j FROM u)"
+ + " AND n IS NOT NULL AND b IS NOT FALSE";
+ case "relation":
+ return "SELECT t0.c FROM catalog.db.t0 AS t0"
+ + IntStream.range(1, 12)
+ .mapToObj(i -> " JOIN catalog.db.t" + i + " AS
t" + i
+ + " ON t" + (i - 1) + ".k = t" + i +
".k")
+ .collect(Collectors.joining());
+ case "tvf":
+ return "SELECT number FROM numbers(\"number\" = \"100\") AS n";
+ case "primary":
+ return "SELECT " + IntStream.range(0, 48)
+ .mapToObj(i -> i % 4 == 0 ? "db.fn" + i + "(c" + i +
")"
+ : i % 4 == 1 ? "c" + i + ".field"
+ : i % 4 == 2 ? "c" + i + "[1].field"
+ : "c" + i)
+ .collect(Collectors.joining(", ")) + " FROM
catalog.db.t";
+ case "show":
+ return "SHOW TABLES FROM db LIKE 'fact%'";
+ case "ddl":
+ return "CREATE TABLE IF NOT EXISTS db.t (k BIGINT, v
VARCHAR(32))"
+ + " DISTRIBUTED BY HASH(k) BUCKETS 8"
+ + " PROPERTIES (\"replication_num\" = \"1\")";
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ }
+
+ private static String repeat(String statement) {
+ return IntStream.range(0, STATEMENT_COUNT)
+ .mapToObj(ignored -> statement)
+ .collect(Collectors.joining("; "));
+ }
+
+ private static String target(String workload) {
+ switch (workload) {
+ case "control":
+ return "SELECT 1";
+ case "limit":
+ return "LIMIT 100 OFFSET 20";
+ case "predicate":
+ return "NOT IN (1, 2, 3)";
+ case "relation":
+ return "catalog.db.t AS t";
+ case "tvf":
+ return "numbers(\"number\" = \"100\") AS n";
+ case "primary":
+ return "db.fn(c)[1].field";
+ case "show":
+ return "SHOW TABLES FROM db LIKE 'fact%'";
+ case "ddl":
+ return "CREATE TABLE IF NOT EXISTS db.t (k BIGINT, v
VARCHAR(32))"
+ + " DISTRIBUTED BY HASH(k) BUCKETS 8"
+ + " PROPERTIES (\"replication_num\" = \"1\")";
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 148563daf8e..b03f6835717 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -1627,9 +1627,11 @@ sortItem
;
limitClause
- : (LIMIT limit=INTEGER_VALUE)
- | (LIMIT limit=INTEGER_VALUE OFFSET offset=INTEGER_VALUE)
- | (LIMIT offset=INTEGER_VALUE COMMA limit=INTEGER_VALUE)
+ : LIMIT limit=INTEGER_VALUE
+ (OFFSET offset=INTEGER_VALUE
+ // Preserve the existing semantic labels for MySQL's LIMIT offset, count
form.
+ | COMMA commaLimit=INTEGER_VALUE {$ctx.offset = $ctx.limit; $ctx.limit =
$ctx.commaLimit;}
+ )?
;
partitionClause
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/LimitClausePrefixTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/LimitClausePrefixTest.java
new file mode 100644
index 00000000000..2c54a6c830a
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/LimitClausePrefixTest.java
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParser.LimitClauseContext;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.stream.Stream;
+
+class LimitClausePrefixTest {
+ private final DorisSqlParser facade = new DorisSqlParser();
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("limitClauses")
+ void preservesLimitAndOffsetLabels(String sql, String expectedLimit,
String expectedOffset) {
+ DorisParser parser = new DorisParser(new
CommonTokenStream(facade.newLexer(sql)));
+ parser.addParseListener(new PostProcessor());
+ parser.removeErrorListeners();
+ parser.addErrorListener(new ParseErrorListener());
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+
+ LimitClauseContext context = parser.limitClause();
+
+ Assertions.assertEquals(Token.EOF, parser.getCurrentToken().getType());
+ Assertions.assertEquals(expectedLimit, context.limit.getText());
+ if (expectedOffset == null) {
+ Assertions.assertNull(context.offset);
+ } else {
+ Assertions.assertEquals(expectedOffset, context.offset.getText());
+ }
+ }
+
+ private static Stream<Arguments> limitClauses() {
+ return Stream.of(
+ Arguments.of("LIMIT 100", "100", null),
+ Arguments.of("LIMIT 100 OFFSET 20", "100", "20"),
+ Arguments.of("LIMIT 20, 100", "100", "20"));
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
