This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f054492cbb9 [improvement](parser) Factor common EXPLAIN and CTE
prefixes (#67427)
f054492cbb9 is described below
commit f054492cbb9fb76a0ac636ff4454a7303bae91c8
Author: morrySnow <[email protected]>
AuthorDate: Thu Sep 3 00:15:25 2026 +0800
[improvement](parser) Factor common EXPLAIN and CTE prefixes (#67427)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SQL grammar repeatedly predicts the optional
`EXPLAIN` and CTE prefixes in `queryOrDmlStatement`, `dmlStatement`, and
`query`, causing long adaptive lookahead for CTE DML. This change
consumes the common prefix once, dispatches the remaining query or DML
body, and passes the prefix contexts to the FE plan builder. It
preserves non-explainable LOAD/EXPORT/REPLAY/COPY/TRUNCATE branches,
nested source CTEs, error positions, and FE command semantics.
This is now an independent PR based directly on `master`. The seven
functional, test, and benchmark files have the same stable patch ID
(`db03fbcb3a0ca4cfee8a1422bc78e7f753e6ae51`) as the original P1 commit.
Only the TODO status update that depended on the earlier roadmap PR was
omitted.
### Benchmark
The benchmark results below are reused from the original P1 validation
because the replayed functional patch is identical. The benchmark
measures the public parser facade and a pre-tokenized parser-only path;
lower latency is better.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB
heap, `-prof gc`
- Standard run: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement
- Longer target run: 3 forks, 6 x 500 ms warmup, 10 x 700 ms measurement
- Measurement baseline: `d7f44fcfedd`; benchmark jar SHA-256
`2e33becac22a27d8c40eec7c821c506cbc9bcedfda277725fa8fee40c6d35116`
- Measurement candidate: `5e0eadb13e9`; benchmark jar SHA-256
`743c133321dbb1a2283ea1836e5a9568a14ecc2ae350b2fc239d354d7988b423`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java`
#### Longer parser-only target run
| Workload | Baseline us/op | Candidate us/op | Latency change |
Baseline B/op | Candidate B/op | Allocation change |
|---|---:|---:|---:|---:|---:|---:|
| CTE SELECT | 43.273 +/- 1.101 | 43.498 +/- 1.528 | +0.52% | 73,453.8 |
73,195.1 | -0.35% |
| CTE INSERT | 48.439 +/- 1.233 | 42.907 +/- 1.236 | -11.42% | 74,096.5
| 74,560.4 | +0.63% |
| EXPLAIN CTE INSERT | 48.032 +/- 1.446 | 42.356 +/- 0.580 | -11.82% |
74,360.5 | 74,963.1 | +0.81% |
#### Public-facade control cases from the standard run
| Workload | Baseline us/op | Candidate us/op | Latency change |
Baseline B/op | Candidate B/op |
|---|---:|---:|---:|---:|---:|
| SELECT control | 2.636 +/- 0.187 | 2.388 +/- 0.167 | -9.42% | 5,200.0
| 4,928.0 |
| EXPLAIN SELECT | 7.094 +/- 1.151 | 6.316 +/- 0.274 | -10.97% |
10,730.8 | 10,728.1 |
The previous grammar performs about 56 tokens of top-level lookahead for
a long CTE and then about 62 more tokens in the DML decision. Factoring
the prefix removes the repeated prediction, which accounts for the CTE
INSERT gains. CTE SELECT remains flat within overlapping confidence
intervals, and target allocation changes remain below 1%. The standard
end-to-end long-CTE forks were noisy, so this PR makes no precise
end-to-end long-CTE or whole-FE latency claim.
### Semantic differential
- Baseline: current `master` at `049410596f4d`; parser jar SHA-256
`923ed2a22142ee9b5dcfefbba5766b9696a653a4218e8208e42a61270e9d986f`
- Candidate parser jar SHA-256:
`3e43f2e025f4152d109a692ba68985c512a155fc2f21345a550911b1ef29af8f`
- Corpus: all 4,610 tracked `*.sql` files; manifest SHA-256
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Legacy and ANSI results: 4,275 parsed and 335 rejected in both
artifacts
- Per-file statement count, exception type, first error position, and
full error message are byte-identical; result SHA-256
`897b6167aa718d5d14887c1752d4bd849cb1c55f7926b474e5dd2f9562a26fbd`
- Lexer and token behavior are unchanged by construction.
### Release note
---
.../doris/nereids/parser/LogicalPlanBuilder.java | 66 +++++++-----
.../parser/LogicalPlanBuilderForEncryption.java | 25 +++--
.../parser/ParseInsertPartitionSpecTest.java | 5 +-
.../parser/QueryOrDmlCommonPrefixPlanTest.java | 95 +++++++++++++++++
.../benchmark/QueryOrDmlCommonPrefixBenchmark.java | 116 +++++++++++++++++++++
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 27 +++--
.../sqlparser/QueryOrDmlCommonPrefixTest.java | 106 +++++++++++++++++++
7 files changed, 400 insertions(+), 40 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 55e92d7c0d8..0382d4641b9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -221,6 +221,8 @@ import org.apache.doris.nereids.DorisParser.ExceptContext;
import org.apache.doris.nereids.DorisParser.ExceptOrReplaceContext;
import org.apache.doris.nereids.DorisParser.ExistContext;
import org.apache.doris.nereids.DorisParser.ExplainContext;
+import org.apache.doris.nereids.DorisParser.ExplainableDmlStatementContext;
+import org.apache.doris.nereids.DorisParser.ExplainableStatementContext;
import org.apache.doris.nereids.DorisParser.ExportContext;
import org.apache.doris.nereids.DorisParser.ExpressionWithEofContext;
import org.apache.doris.nereids.DorisParser.ExpressionWithOrderContext;
@@ -458,7 +460,6 @@ import
org.apache.doris.nereids.DorisParser.SortClauseContext;
import org.apache.doris.nereids.DorisParser.SortItemContext;
import org.apache.doris.nereids.DorisParser.SpecifiedPartitionContext;
import org.apache.doris.nereids.DorisParser.StarContext;
-import org.apache.doris.nereids.DorisParser.StatementDefaultContext;
import org.apache.doris.nereids.DorisParser.StatementScopeContext;
import org.apache.doris.nereids.DorisParser.StepPartitionDefContext;
import org.apache.doris.nereids.DorisParser.StringLiteralContext;
@@ -1229,14 +1230,24 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
}
@Override
- public LogicalPlan visitStatementDefault(StatementDefaultContext ctx) {
- LogicalPlan plan = plan(ctx.query());
+ public LogicalPlan visitExplainableStatement(ExplainableStatementContext
ctx) {
+ if (ctx.dmlStatementBody() != null) {
+ return plan(ctx.dmlStatementBody());
+ }
+ LogicalPlan plan = ParserUtils.withOrigin(
+ ctx.cteContext != null ? ctx.cteContext : ctx.queryTerm(),
+ () -> withCte(buildQuery(ctx.queryTerm(),
ctx.queryOrganization()), ctx.cteContext));
if (ctx.outFileClause() != null) {
plan = withOutFile(plan, ctx.outFileClause());
} else {
plan = new UnboundResultSink<>(plan);
}
- return withExplain(plan, ctx.explain());
+ return withExplain(plan, ctx.explainContext);
+ }
+
+ @Override
+ public LogicalPlan
visitExplainableDmlStatement(ExplainableDmlStatementContext ctx) {
+ return plan(ctx.dmlStatementBody());
}
@Override
@@ -1451,12 +1462,12 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
tvfName, properties, DMLCommandType.INSERT, plan);
Optional<LogicalPlan> cte = Optional.empty();
- if (ctx.cte() != null) {
- cte = Optional.ofNullable(withCte(plan, ctx.cte()));
+ if (ctx.cteContext != null) {
+ cte = Optional.ofNullable(withCte(plan, ctx.cteContext));
}
LogicalPlan command = new InsertIntoTVFCommand(sink, labelName, cte);
- return withExplain(command, ctx.explain());
+ return withExplain(command, ctx.explainContext);
}
/**
@@ -1507,8 +1518,8 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
plan,
partitionSpec.isStaticPartition() ?
partitionSpec.getStaticPartitionValues() : null);
Optional<LogicalPlan> cte = Optional.empty();
- if (ctx.cte() != null) {
- cte = Optional.ofNullable(withCte(plan, ctx.cte()));
+ if (ctx.cteContext != null) {
+ cte = Optional.ofNullable(withCte(plan, ctx.cteContext));
}
LogicalPlan command;
if (isOverwrite) {
@@ -1525,12 +1536,14 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
command = new InsertIntoTableCommand(sink, labelName,
Optional.empty(), cte, true, branchName);
}
}
- return withExplain(command, ctx.explain());
+ return withExplain(command, ctx.explainContext);
}
@Override
public Object visitMergeInto(MergeIntoContext ctx) {
- return ParserUtils.withOrigin(ctx, () -> {
+ ParserRuleContext originContext = ctx.explainContext != null
+ ? ctx.explainContext : ctx.cteContext != null ? ctx.cteContext
: ctx;
+ return ParserUtils.withOrigin(originContext, () -> {
List<String> targetNameParts =
visitMultipartIdentifier(ctx.targetTable);
Optional<String> targetAlias = Optional.ofNullable(
ctx.identifier() != null ? ctx.identifier().getText() :
null);
@@ -1540,11 +1553,11 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
List<MergeNotMatchedClause> notMatchedClauses =
visit(ctx.mergeNotMatchedClause(),
MergeNotMatchedClause.class);
Optional<LogicalPlan> cte = Optional.empty();
- if (ctx.cte() != null) {
- cte = Optional.ofNullable(withCte(source, ctx.cte()));
+ if (ctx.cteContext != null) {
+ cte = Optional.ofNullable(withCte(source, ctx.cteContext));
}
return withExplain(new MergeIntoCommand(targetNameParts,
targetAlias, cte,
- source, onClause, matchedClauses, notMatchedClauses),
ctx.explain());
+ source, onClause, matchedClauses, notMatchedClauses),
ctx.explainContext);
});
}
@@ -2077,11 +2090,11 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
tableAlias = ctx.tableAlias().strictIdentifier().getText();
}
Optional<LogicalPlan> cte = Optional.empty();
- if (ctx.cte() != null) {
- cte = Optional.ofNullable(withCte(query, ctx.cte()));
+ if (ctx.cteContext != null) {
+ cte = Optional.ofNullable(withCte(query, ctx.cteContext));
}
return withExplain(new
UpdateCommand(visitMultipartIdentifier(ctx.tableName), tableAlias,
- visitUpdateAssignmentSeq(ctx.updateAssignmentSeq()), query,
cte), ctx.explain());
+ visitUpdateAssignmentSeq(ctx.updateAssignmentSeq()), query,
cte), ctx.explainContext);
}
@Override
@@ -2106,7 +2119,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
&& (ctx.queryOrganization().sortClause() != null
|| ctx.queryOrganization().limitClause() != null);
Command deleteCommand;
- if (ctx.USING() == null && ctx.cte() == null && !hasQueryOrganization)
{
+ if (ctx.USING() == null && ctx.cteContext == null &&
!hasQueryOrganization) {
query = withFilter(query, Optional.ofNullable(ctx.whereClause()));
deleteCommand = new DeleteFromCommand(tableName, tableAlias,
partitionSpec.first,
partitionSpec.second, query);
@@ -2119,14 +2132,14 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
query = withQueryOrganization(query, ctx.queryOrganization());
query = convertSortOrdinalsToUnboundSlot(query);
Optional<LogicalPlan> cte = Optional.empty();
- if (ctx.cte() != null) {
- cte = Optional.ofNullable(withCte(query, ctx.cte()));
+ if (ctx.cteContext != null) {
+ cte = Optional.ofNullable(withCte(query, ctx.cteContext));
}
deleteCommand = new DeleteFromUsingCommand(tableName, tableAlias,
partitionSpec.first, partitionSpec.second, query, cte,
hasQueryOrganization);
}
- if (ctx.explain() != null) {
- return withExplain(deleteCommand, ctx.explain());
+ if (ctx.explainContext != null) {
+ return withExplain(deleteCommand, ctx.explainContext);
} else {
return deleteCommand;
}
@@ -2596,12 +2609,15 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
public LogicalPlan visitQuery(QueryContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
// TODO: need to add withQueryResultClauses and withCTE
- LogicalPlan query = plan(ctx.queryTerm());
- query = withQueryOrganization(query, ctx.queryOrganization());
- return withCte(query, ctx.cte());
+ return withCte(buildQuery(ctx.queryTerm(),
ctx.queryOrganization()), ctx.cte());
});
}
+ private LogicalPlan buildQuery(QueryTermContext queryTerm,
QueryOrganizationContext queryOrganization) {
+ LogicalPlan query = plan(queryTerm);
+ return withQueryOrganization(query, queryOrganization);
+ }
+
@Override
public LogicalPlan visitSetOperation(SetOperationContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForEncryption.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForEncryption.java
index 1e074463d94..c1552b58f83 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForEncryption.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForEncryption.java
@@ -22,6 +22,9 @@ import org.apache.doris.analysis.UserDesc;
import org.apache.doris.common.Pair;
import org.apache.doris.common.util.DatasourcePrintableMap;
import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParser.DmlStatementBodyContext;
+import org.apache.doris.nereids.DorisParser.DmlStatementContext;
+import org.apache.doris.nereids.DorisParser.ExplainableDmlStatementContext;
import org.apache.doris.nereids.DorisParser.InsertTableContext;
import org.apache.doris.nereids.DorisParser.JobFromToClauseContext;
import org.apache.doris.nereids.trees.plans.commands.info.SetVarOp;
@@ -47,14 +50,14 @@ public class LogicalPlanBuilderForEncryption extends
LogicalPlanBuilder {
// select into outfile clause
@Override
- public LogicalPlan
visitStatementDefault(DorisParser.StatementDefaultContext ctx) {
+ public LogicalPlan
visitExplainableStatement(DorisParser.ExplainableStatementContext ctx) {
if (ctx.outFileClause() != null &&
ctx.outFileClause().propertyClause() != null) {
DorisParser.PropertyClauseContext propertyClauseContext =
ctx.outFileClause().propertyClause();
encryptProperty(visitPropertyClause(propertyClauseContext),
propertyClauseContext.fileProperties.start.getStartIndex(),
propertyClauseContext.fileProperties.stop.getStopIndex());
}
- return super.visitStatementDefault(ctx);
+ return super.visitExplainableStatement(ctx);
}
// export into outfile clause
@@ -227,8 +230,9 @@ public class LogicalPlanBuilderForEncryption extends
LogicalPlanBuilder {
// create job select tvf
@Override
public LogicalPlan
visitCreateScheduledJob(DorisParser.CreateScheduledJobContext ctx) {
- if (ctx.dmlStatement() instanceof InsertTableContext) {
- visitInsertTable((InsertTableContext) ctx.dmlStatement());
+ InsertTableContext insertTableContext =
getInsertTableContext(ctx.dmlStatement());
+ if (insertTableContext != null) {
+ visitInsertTable(insertTableContext);
} else if (ctx.jobFromToClause() != null) {
JobFromToClauseContext jobFromToClauseContext =
ctx.jobFromToClause();
encryptProperty(visitPropertyItemList(jobFromToClauseContext.sourceProperties),
@@ -242,8 +246,9 @@ public class LogicalPlanBuilderForEncryption extends
LogicalPlanBuilder {
// alter job select tvf
@Override
public LogicalPlan visitAlterJob(DorisParser.AlterJobContext ctx) {
- if (ctx.dmlStatement() instanceof InsertTableContext) {
- visitInsertTable((InsertTableContext) ctx.dmlStatement());
+ InsertTableContext insertTableContext =
getInsertTableContext(ctx.dmlStatement());
+ if (insertTableContext != null) {
+ visitInsertTable(insertTableContext);
} else if (ctx.jobFromToClause() != null) {
JobFromToClauseContext jobFromToClauseContext =
ctx.jobFromToClause();
encryptProperty(visitPropertyItemList(jobFromToClauseContext.sourceProperties),
@@ -254,6 +259,14 @@ public class LogicalPlanBuilderForEncryption extends
LogicalPlanBuilder {
return super.visitAlterJob(ctx);
}
+ private InsertTableContext getInsertTableContext(DmlStatementContext ctx) {
+ if (!(ctx instanceof ExplainableDmlStatementContext)) {
+ return null;
+ }
+ DmlStatementBodyContext body = ((ExplainableDmlStatementContext)
ctx).dmlStatementBody();
+ return body instanceof InsertTableContext ? (InsertTableContext) body
: null;
+ }
+
@Override
public LogicalPlan visitCreateResource(DorisParser.CreateResourceContext
ctx) {
if (ctx.properties != null) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ParseInsertPartitionSpecTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ParseInsertPartitionSpecTest.java
index 70a3d0e6375..e1b74cd9d2c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ParseInsertPartitionSpecTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ParseInsertPartitionSpecTest.java
@@ -83,8 +83,11 @@ public class ParseInsertPartitionSpecTest {
* Helper method to parse SQL and extract PartitionSpecContext.
*/
private DorisParser.PartitionSpecContext parsePartitionSpec(String
insertSql) {
- DorisParser.InsertTableContext insertTableContext =
(DorisParser.InsertTableContext) NereidsParser.toAst(
+ DorisParser.DmlStatementContext dmlStatementContext =
(DorisParser.DmlStatementContext) NereidsParser.toAst(
insertSql, DorisParser::dmlStatement);
+ DorisParser.DmlStatementBodyContext dmlStatementBody =
+ ((DorisParser.ExplainableDmlStatementContext)
dmlStatementContext).dmlStatementBody();
+ DorisParser.InsertTableContext insertTableContext =
(DorisParser.InsertTableContext) dmlStatementBody;
return insertTableContext.partitionSpec();
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrDmlCommonPrefixPlanTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrDmlCommonPrefixPlanTest.java
new file mode 100644
index 00000000000..259b04478da
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrDmlCommonPrefixPlanTest.java
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.nereids.analyzer.UnboundResultSink;
+import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.nereids.trees.plans.commands.DeleteFromUsingCommand;
+import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.commands.UpdateCommand;
+import
org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand;
+import org.apache.doris.nereids.trees.plans.commands.merge.MergeIntoCommand;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTE;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.stream.Stream;
+
+public class QueryOrDmlCommonPrefixPlanTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("statementPlans")
+ public void buildsExpectedPlanAfterCommonPrefix(
+ String description, String sql, Class<? extends LogicalPlan>
expectedClass) {
+ Assertions.assertInstanceOf(expectedClass, parser.parseSingle(sql));
+ }
+
+ private static Stream<Arguments> statementPlans() {
+ return Stream.of(
+ Arguments.of("CTE query", "WITH c AS (SELECT 1) SELECT * FROM
c", UnboundResultSink.class),
+ Arguments.of("explain CTE query", "EXPLAIN WITH c AS (SELECT
1) SELECT * FROM c",
+ ExplainCommand.class),
+ Arguments.of("CTE insert", "WITH c AS (SELECT 1) INSERT INTO
db.t SELECT * FROM c",
+ InsertIntoTableCommand.class),
+ Arguments.of("CTE update",
+ "WITH c AS (SELECT 1 AS id) UPDATE db.t SET v = 1 FROM
c WHERE t.id = c.id",
+ UpdateCommand.class),
+ Arguments.of("CTE delete",
+ "WITH c AS (SELECT 1 AS id) DELETE FROM db.t USING c
WHERE t.id = c.id",
+ DeleteFromUsingCommand.class),
+ Arguments.of("CTE merge",
+ "WITH c AS (SELECT 1 AS id) MERGE INTO db.t USING c ON
t.id = c.id "
+ + "WHEN MATCHED THEN UPDATE SET v = 1",
+ MergeIntoCommand.class));
+ }
+
+ @Test
+ public void preservesExplainAndCtePlanShape() {
+ ExplainCommand explain = (ExplainCommand) parser.parseSingle(
+ "EXPLAIN WITH c AS (SELECT 1) SELECT * FROM c");
+ UnboundResultSink<?> sink = (UnboundResultSink<?>)
explain.getLogicalPlan();
+ Assertions.assertInstanceOf(LogicalCTE.class, sink.child());
+ }
+
+ @Test
+ public void preservesExplainAndCteForInsert() {
+ ExplainCommand explain = (ExplainCommand) parser.parseSingle(
+ "EXPLAIN WITH c AS (SELECT 1) INSERT INTO db.t SELECT * FROM
c");
+ Assertions.assertInstanceOf(InsertIntoTableCommand.class,
explain.getLogicalPlan());
+
Assertions.assertTrue(explain.getLogicalPlan().toDigest().contains("WITH\n"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("invalidPrefixCombinations")
+ public void rejectsInvalidPrefixCombinations(String sql) {
+ Assertions.assertThrows(ParseException.class, () ->
parser.parseSingle(sql));
+ }
+
+ private static Stream<String> invalidPrefixCombinations() {
+ return Stream.of(
+ "EXPLAIN TRUNCATE TABLE t",
+ "WITH c AS (SELECT 1) TRUNCATE TABLE t",
+ "WITH c AS (SELECT 1)");
+ }
+}
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
new file mode 100644
index 00000000000..b2700dce7f8
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ListTokenSource;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures common EXPLAIN/CTE prefix dispatch in isolation and through the
public parser facade. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class QueryOrDmlCommonPrefixBenchmark {
+ @Param({"control", "explain", "cteSelect", "cteInsert",
"explainCteInsert"})
+ public String workload;
+
+ private final DorisSqlParser facade = new DorisSqlParser();
+ private final PostProcessor postProcessor = new PostProcessor();
+ private final ParseErrorListener errorListener = new ParseErrorListener();
+
+ private String sql;
+ private List<Token> tokens;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ String cte = buildCte(12);
+ switch (workload) {
+ case "control":
+ sql = "SELECT 1";
+ break;
+ case "explain":
+ sql = "EXPLAIN SELECT a, b FROM t WHERE a > 1";
+ break;
+ case "cteSelect":
+ sql = cte + " SELECT * FROM c11";
+ break;
+ case "cteInsert":
+ sql = cte + " INSERT INTO target_table SELECT * FROM c11";
+ break;
+ case "explainCteInsert":
+ sql = "EXPLAIN " + cte + " INSERT INTO target_table SELECT *
FROM c11";
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+
+ CommonTokenStream stream = new CommonTokenStream(facade.newLexer(sql));
+ stream.fill();
+ tokens = List.copyOf(stream.getTokens());
+ }
+
+ @Benchmark
+ public Object parseEndToEnd() {
+ return facade.parseStatement(sql);
+ }
+
+ @Benchmark
+ public Object parsePreTokenized() {
+ CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
+ DorisParser parser = new DorisParser(stream);
+ parser.addParseListener(postProcessor);
+ parser.removeErrorListeners();
+ parser.addErrorListener(errorListener);
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parser.singleStatement();
+ }
+
+ private static String buildCte(int count) {
+ return "WITH " + IntStream.range(0, count)
+ .mapToObj(index -> index == 0
+ ? "c0 AS (SELECT 0 AS k)"
+ : "c" + index + " AS (SELECT k + 1 AS k FROM c" +
(index - 1) + ")")
+ .collect(Collectors.joining(", "));
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index b2cdefd8901..148563daf8e 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -105,8 +105,10 @@ statementBase
;
queryOrDmlStatement
- : explain? query outFileClause? #statementDefault
- | dmlStatement #dmlStatementAlias
+ : explainContext=explain? cteContext=cte?
+ (queryTerm queryOrganization outFileClause?
+ | dmlStatementBody[$explainContext.ctx, $cteContext.ctx])
#explainableStatement
+ | nonExplainableDmlStatement #dmlStatementAlias
| describeStatement #describeStatementAlias
| otherStatement #otherStatementAlias
| loadDmlStatement #loadStatementAlias
@@ -274,31 +276,40 @@ optSpecBranch
;
dmlStatement
- : explain? cte? INSERT INTO tvfName=identifier
+ : explainContext=explain? cteContext=cte?
+ dmlStatementBody[$explainContext.ctx, $cteContext.ctx]
#explainableDmlStatement
+ | nonExplainableDmlStatement
#nonExplainableDmlStatementAlias
+ ;
+
+dmlStatementBody[ExplainContext explainContext, CteContext cteContext]
+ : INSERT INTO tvfName=identifier
LEFT_PAREN tvfProperties=propertyItemList RIGHT_PAREN
(WITH LABEL labelName=identifier)?
query
#insertIntoTVF
- | explain? cte? INSERT (INTO | OVERWRITE TABLE)
+ | INSERT (INTO | OVERWRITE TABLE)
(tableName=multipartIdentifier (optSpecBranch)? |
DORIS_INTERNAL_TABLE_ID LEFT_PAREN tableId=INTEGER_VALUE RIGHT_PAREN)
partitionSpec? // partition define
(WITH LABEL labelName=identifier)? cols=identifierList? // label and
columns define
(LEFT_BRACKET hints=identifierSeq RIGHT_BRACKET)? // hint define
query
#insertTable
- | explain? cte? UPDATE tableName=multipartIdentifier tableAlias
+ | UPDATE tableName=multipartIdentifier tableAlias
SET updateAssignmentSeq
fromClause?
whereClause?
queryOrganization #update
- | explain? cte? DELETE FROM tableName=multipartIdentifier
+ | DELETE FROM tableName=multipartIdentifier
partitionSpec? tableAlias
(USING relations)?
whereClause?
queryOrganization #delete
- | explain? cte? MERGE INTO targetTable=multipartIdentifier
+ | MERGE INTO targetTable=multipartIdentifier
(AS? identifier)? USING srcRelation=relationPrimary
ON expression
(mergeMatchedClause | mergeNotMatchedClause)+
#mergeInto
- | LOAD LABEL lableName=multipartIdentifier
+ ;
+
+nonExplainableDmlStatement
+ : LOAD LABEL lableName=multipartIdentifier
LEFT_PAREN dataDescs+=dataDesc (COMMA dataDescs+=dataDesc)* RIGHT_PAREN
(withRemoteStorageSystem)?
propertyClause?
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrDmlCommonPrefixTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrDmlCommonPrefixTest.java
new file mode 100644
index 00000000000..cb69398ddd5
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrDmlCommonPrefixTest.java
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser.SingleStatementContext;
+import org.apache.doris.nereids.exceptions.ParseException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+class QueryOrDmlCommonPrefixTest {
+ private final DorisSqlParser parser = new DorisSqlParser();
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("explainableStatements")
+ void parsesExplainAndCtePrefixes(String description, String sql) {
+ SingleStatementContext context = parser.parseStatement(sql);
+ Assertions.assertNotNull(context.statement());
+ }
+
+ private static Stream<Arguments> explainableStatements() {
+ String longCte = buildCte(20);
+ return Stream.of(
+ Arguments.of("query", "SELECT 1"),
+ Arguments.of("explain query", "EXPLAIN SELECT 1"),
+ Arguments.of("CTE query", "WITH c AS (SELECT 1) SELECT * FROM
c"),
+ Arguments.of("explain CTE query", "EXPLAIN WITH c AS (SELECT
1) SELECT * FROM c"),
+ Arguments.of("long CTE query", longCte + " SELECT * FROM c19"),
+ Arguments.of("CTE insert", "WITH c AS (SELECT 1) INSERT INTO t
SELECT * FROM c"),
+ Arguments.of("long CTE insert", longCte + " INSERT INTO t
SELECT * FROM c19"),
+ Arguments.of("explain CTE insert",
+ "EXPLAIN WITH c AS (SELECT 1) INSERT OVERWRITE TABLE t
SELECT * FROM c"),
+ Arguments.of("outer and source CTE",
+ "WITH c AS (SELECT 1) INSERT INTO t WITH d AS (SELECT
* FROM c) SELECT * FROM d"),
+ Arguments.of("CTE update",
+ "WITH c AS (SELECT 1 AS id) UPDATE t SET v = 1 FROM c
WHERE t.id = c.id"),
+ Arguments.of("CTE delete",
+ "WITH c AS (SELECT 1 AS id) DELETE FROM t USING c
WHERE t.id = c.id"),
+ Arguments.of("CTE merge",
+ "WITH c AS (SELECT 1 AS id) MERGE INTO t USING c ON
t.id = c.id "
+ + "WHEN MATCHED THEN UPDATE SET v = 1"),
+ Arguments.of("job DML prefix",
+ "CREATE JOB db.job ON SCHEDULE AT '2026-01-01
00:00:00' DO "
+ + "EXPLAIN WITH c AS (SELECT 1) INSERT INTO t
SELECT * FROM c"),
+ Arguments.of("non-explainable DML", "TRUNCATE TABLE t"),
+ Arguments.of("warm-up explain", "EXPLAIN WARM UP SELECT * FROM
t"),
+ Arguments.of("describe", "DESC t"));
+ }
+
+ @ParameterizedTest(name = "rejects: {0}")
+ @MethodSource("invalidPrefixCombinations")
+ void rejectsPrefixesThatWereNotPreviouslyAccepted(String sql) {
+ Assertions.assertThrows(ParseException.class, () ->
parser.parseStatement(sql));
+ }
+
+ private static Stream<String> invalidPrefixCombinations() {
+ return Stream.of(
+ "EXPLAIN TRUNCATE TABLE t",
+ "WITH c AS (SELECT 1) TRUNCATE TABLE t",
+ "CREATE JOB db.job ON SCHEDULE AT '2026-01-01 00:00:00' DO "
+ + "WITH c AS (SELECT 1) SELECT * FROM c");
+ }
+
+ @ParameterizedTest(name = "truncated: {0}")
+ @MethodSource("truncatedPrefixes")
+ void reportsTruncatedPrefixesAtEndOfInput(String sql) {
+ ParseException exception =
Assertions.assertThrows(ParseException.class, () -> parser.parseStatement(sql));
+ Assertions.assertTrue(exception.getMessage().contains("line 1, pos " +
sql.length()), exception::getMessage);
+ }
+
+ private static Stream<String> truncatedPrefixes() {
+ return Stream.of(
+ "EXPLAIN",
+ "WITH c AS (SELECT 1)",
+ "EXPLAIN WITH c AS (SELECT 1) INSERT INTO t");
+ }
+
+ private static String buildCte(int count) {
+ return "WITH " + IntStream.range(0, count)
+ .mapToObj(index -> index == 0
+ ? "c0 AS (SELECT 0 AS k)"
+ : "c" + index + " AS (SELECT k + 1 AS k FROM c" +
(index - 1) + ")")
+ .collect(Collectors.joining(", "));
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
