This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a6a7e52a733 [improvement](parser) Add case-insensitive stream fast
paths (#67452)
a6a7e52a733 is described below
commit a6a7e52a7336fcca8dd1564cbeb0f3d726304cc4
Author: morrySnow <[email protected]>
AuthorDate: Thu Sep 10 12:07:59 2026 +0800
[improvement](parser) Add case-insensitive stream fast paths (#67452)
### What problem does this PR solve?
Problem Summary:
Every SQL lexer currently copies its input through
`CharStreams.fromString()` and calls `Character.toUpperCase()` for every
case-insensitive lookahead. This is visible in lexer allocation and
identifier-heavy parsing time.
Add two strictly compatible fast paths:
- Fold ASCII `a-z` with arithmetic and retain `Character.toUpperCase()`
for every other code point.
- Read strings without UTF-16 surrogates directly, avoiding ANTLR's
copied code-point buffer. Inputs containing any surrogate retain the
original ANTLR stream so code-point indices, `getText`, navigation, and
errors remain unchanged.
The public arbitrary-`CharStream` constructor is preserved. All
production String entry points now use the factory. ANTLR's native
`caseInsensitive` option was evaluated but rejected because it changed
existing Unicode behavior such as the handling of `ſelect`.
### Benchmark
Environment and method:
- Baseline: `cf33a08bcd5`, artifact SHA-256
`51700bef953361218912abc6aec348cecdec3fe61031eba10cc1ebc534bc9183`
- Candidate: `7aa7b5e6487`, artifact SHA-256
`6e8edf78befc0dc349be398aae454c172b5cb968ee98d88a0e056d4c21a9c5d0`
- macOS 15.0.1 arm64, OpenJDK 17.0.20.1, 1 GiB heap, JMH 1.37
- 2 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Interleaved baseline/candidate runs; one candidate run affected by
unrelated host load was discarded and repeated. The table averages two
valid runs per artifact. Individual JMH scores use 99.9% confidence
intervals.
```shell
java -Xms1g -Xmx1g -jar <benchmark.jar> \
'CaseInsensitiveStreamBenchmark.(createLexer|foldPrebuiltCharacters|parseStatement|tokenize)'
\
-p workload=shortQuery,lowercaseIdentifiers,stringAndComment,unicode \
-f 2 -wi 4 -i 7 -w 300ms -r 400ms -prof gc -rf json
```
| Benchmark | Baseline us/op (B1 / B2) |
Candidate us/op (C1 / C3) | Mean change | Baseline → candidate B/op |
| -------------------------------------- | --------------------------: |
--------------------------: | ----------: | ------------------------: |
| Tokenize 64 lowercase identifiers | 10.826±0.078 / 11.408±0.125 |
8.708±0.071 / 8.704±0.096 | -21.7% | 14,799 → 10,464 (-29.3%) |
| Parse 64 lowercase identifiers | 54.775±7.360 / 56.365±4.224 |
49.860±0.191 / 52.340±1.160 | -8.0% | 99,395 → 95,017 (-4.4%) |
| Tokenize `select 1` | 0.451±0.004 / 0.485±0.011 |
0.348±0.069 / 0.324±0.010 | -28.2% | 1,008 → 816 (-19.0%) |
| Parse strings/comments with Unicode | 27.047±5.371 / 28.435±2.932 |
24.744±0.401 / 24.743±0.425 | -10.8% | 24,167 → 23,784 (-1.6%) |
| Tokenize supplementary Unicode control | 32.034±1.398 / 33.910±0.752 |
32.303±1.108 / 32.020±0.738 | -2.5% | 24,630 → 24,594 (-0.1%) |
The isolated `createLexer` measurement for the surrogate-containing
string/comment workload regresses by 5.9% because the compatibility
guard scans for surrogates before falling back. The corresponding
complete tokenize and parse paths improve by 3.6% and 10.8%; no
end-to-end control workload regressed. Prebuilt lowercase character
folding improves by 10.6%.
The performance gains come from eliminating the copied input buffer for
BMP-only SQL, avoiding its allocation, and replacing the common
lowercase ASCII `Character.toUpperCase()` call with an arithmetic
branch.
Correctness corpus:
- 4,610 tracked SQL files; baseline and candidate parse signatures match
in legacy and ANSI modes. SHA-256:
`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`.
- 9,220 lexer cases (4,610 SQL files x both `noBackslashEscapes` modes);
complete token tuple and lexer-error signatures match. SHA-256:
`66780d4e1d9224ea27c8f715ab33edd247faed1571030ba9e90a8ef3f6212180`.
---
.../httpv2/websql/SingleStatementValidator.java | 3 +-
.../apache/doris/nereids/parser/NereidsParser.java | 7 +-
.../benchmark/CaseInsensitiveStreamBenchmark.java | 140 ++++++++++++
.../nereids/parser/CaseInsensitiveStream.java | 91 +++++++-
.../org/apache/doris/sqlparser/DorisSqlParser.java | 3 +-
.../doris/sqlparser/CaseInsensitiveStreamTest.java | 245 +++++++++++++++++++++
6 files changed, 480 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/websql/SingleStatementValidator.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/websql/SingleStatementValidator.java
index a04746bc855..a2ea4c01338 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/websql/SingleStatementValidator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/websql/SingleStatementValidator.java
@@ -23,7 +23,6 @@ import
org.apache.doris.nereids.DorisParser.MultiStatementsContext;
import org.apache.doris.nereids.parser.CaseInsensitiveStream;
import org.apache.doris.nereids.parser.NereidsParser;
-import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
/**
@@ -45,7 +44,7 @@ public final class SingleStatementValidator {
}
try {
- DorisLexer lexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ DorisLexer lexer = new
DorisLexer(CaseInsensitiveStream.fromString(sql));
lexer.isNoBackslashEscapes = noBackslashEscapes;
CommonTokenStream tokens = new CommonTokenStream(lexer);
MultiStatementsContext parsed = (MultiStatementsContext)
NereidsParser.toAst(
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
index 66ca0ecdc2f..4184c0f421c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
@@ -42,7 +42,6 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
-import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Recognizer;
@@ -143,7 +142,7 @@ public class NereidsParser {
* for example: select id from tbl return Tokens: ['select', 'id', 'from',
'tbl']
*/
public static TokenSource scan(String sql) {
- return new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ return new DorisLexer(CaseInsensitiveStream.fromString(sql));
}
/**
@@ -392,7 +391,7 @@ public class NereidsParser {
while (hintToken != null && hintToken.getType() != DorisLexer.EOF) {
if (hintToken.getChannel() == 2 &&
sql.charAt(hintToken.getStartIndex() + 2) == '+') {
String hintSql = sql.substring(hintToken.getStartIndex() + 3,
hintToken.getStopIndex() + 1);
- DorisLexer newHintLexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(hintSql)));
+ DorisLexer newHintLexer = new
DorisLexer(CaseInsensitiveStream.fromString(hintSql));
CommonTokenStream newHintTokenStream = new
CommonTokenStream(newHintLexer);
DorisParser hintParser = new DorisParser(newHintTokenStream);
ParserRuleContext hintContext =
parseFunction.apply(hintParser);
@@ -477,7 +476,7 @@ public class NereidsParser {
}
private static CommonTokenStream parseTokens(String sql, boolean
leanTokenMode) {
- DorisLexer lexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ DorisLexer lexer = new
DorisLexer(CaseInsensitiveStream.fromString(sql));
lexer.isNoBackslashEscapes = SqlModeHelper.hasNoBackSlashEscapes();
lexer.isLeanTokenMode = leanTokenMode;
CommonTokenStream tokenStream = leanTokenMode
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/CaseInsensitiveStreamBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/CaseInsensitiveStreamBenchmark.java
new file mode 100644
index 00000000000..a60b3e64b55
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/CaseInsensitiveStreamBenchmark.java
@@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CharStream;
+import org.antlr.v4.runtime.Token;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.Locale;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures case-folding stream construction, lookahead, lexing, and
end-to-end parsing. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class CaseInsensitiveStreamBenchmark {
+ @Param({
+ "shortQuery",
+ "lowercaseIdentifiers",
+ "mixedCaseIdentifiers",
+ "uppercaseIdentifiers",
+ "stringAndComment",
+ "unicode"
+ })
+ public String workload;
+
+ private DorisSqlParser parser;
+ private String sql;
+ private CharStream prebuiltStream;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ parser = new DorisSqlParser();
+ switch (workload) {
+ case "shortQuery":
+ sql = "select 1";
+ break;
+ case "lowercaseIdentifiers":
+ sql = projection("lowercase_column_", false);
+ break;
+ case "mixedCaseIdentifiers":
+ sql = projection("mixedCaseColumn_", false);
+ break;
+ case "uppercaseIdentifiers":
+ sql = projection("UPPERCASE_COLUMN_", true);
+ break;
+ case "stringAndComment":
+ sql = "select 'lowercase MixedCase 中文😀' as text_value,
lower_name, mixedName "
+ + "from lower_table /* lowercase MixedCase 中文😀 */ "
+ + "where lower_name = 'another string' -- trailing
comment\nlimit 10";
+ break;
+ case "unicode":
+ String supplementary = new String(Character.toChars(0x10428));
+ sql = "select café_列, élève_列, " + supplementary + "_column,
'ı ſ 中文 😀' "
+ + "from 数据表 where café_列 > 10";
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ prebuiltStream = parser.newLexer(sql).getInputStream();
+ }
+
+ @Benchmark
+ public Object createLexer() {
+ return parser.newLexer(sql);
+ }
+
+ @Benchmark
+ public int foldPrebuiltCharacters() {
+ prebuiltStream.seek(0);
+ int checksum = 1;
+ int character;
+ while ((character = prebuiltStream.LA(1)) != Token.EOF) {
+ checksum = 31 * checksum + character;
+ prebuiltStream.consume();
+ }
+ return checksum;
+ }
+
+ @Benchmark
+ public int tokenize() {
+ DorisLexer lexer = parser.newLexer(sql);
+ int checksum = 1;
+ Token token;
+ do {
+ token = lexer.nextToken();
+ checksum = 31 * checksum + token.getType();
+ checksum = 31 * checksum + token.getStartIndex();
+ checksum = 31 * checksum + token.getStopIndex();
+ } while (token.getType() != Token.EOF);
+ return checksum;
+ }
+
+ @Benchmark
+ public Object parseStatement() {
+ return parser.parseStatement(sql);
+ }
+
+ private static String projection(String prefix, boolean uppercase) {
+ String projection = IntStream.range(0, 64)
+ .mapToObj(index -> prefix + index)
+ .collect(Collectors.joining(", "));
+ String sql = "select " + projection + " from identifier_heavy_table
where " + prefix + "0 > 10";
+ return uppercase ? sql.toUpperCase(Locale.ROOT) : sql;
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
index 54fe54ab8ba..63190f2e0fc 100644
---
a/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
@@ -18,6 +18,7 @@
package org.apache.doris.nereids.parser;
import org.antlr.v4.runtime.CharStream;
+import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.IntStream;
import org.antlr.v4.runtime.misc.Interval;
@@ -31,6 +32,19 @@ public class CaseInsensitiveStream implements CharStream {
this.stream = stream;
}
+ /**
+ * Avoid copying strings whose UTF-16 indices already equal code-point
indices.
+ * Strings containing surrogates keep ANTLR's code-point stream and
indexing semantics.
+ */
+ public static CharStream fromString(String input) {
+ for (int index = 0; index < input.length(); index++) {
+ if (Character.isSurrogate(input.charAt(index))) {
+ return new
CaseInsensitiveStream(CharStreams.fromString(input));
+ }
+ }
+ return new CaseInsensitiveStringStream(input);
+ }
+
@Override
public String getText(Interval interval) {
return stream.getText(interval);
@@ -43,8 +57,13 @@ public class CaseInsensitiveStream implements CharStream {
@Override
public int LA(int i) {
- int result = stream.LA(i);
+ return toUpperCase(stream.LA(i));
+ }
+ private static int toUpperCase(int result) {
+ if (result >= 'a' && result <= 'z') {
+ return result - ('a' - 'A');
+ }
switch (result) {
case 0:
case IntStream.EOF:
@@ -83,4 +102,74 @@ public class CaseInsensitiveStream implements CharStream {
public String getSourceName() {
return stream.getSourceName();
}
+
+ private static final class CaseInsensitiveStringStream implements
CharStream {
+ private final String input;
+ private int position;
+
+ CaseInsensitiveStringStream(String input) {
+ this.input = input;
+ }
+
+ @Override
+ public String getText(Interval interval) {
+ int start = Math.min(interval.a, input.length());
+ int length = Math.min(interval.b - interval.a + 1, input.length()
- start);
+ return input.substring(start, start + length);
+ }
+
+ @Override
+ public void consume() {
+ if (position == input.length()) {
+ throw new IllegalStateException("cannot consume EOF");
+ }
+ position++;
+ }
+
+ @Override
+ public int LA(int offset) {
+ int index;
+ switch (Integer.signum(offset)) {
+ case -1:
+ index = position + offset;
+ return index < 0 ? IntStream.EOF :
toUpperCase(input.charAt(index));
+ case 0:
+ return 0;
+ case 1:
+ index = position + offset - 1;
+ return index >= input.length() ? IntStream.EOF :
toUpperCase(input.charAt(index));
+ default:
+ throw new UnsupportedOperationException("Not reached");
+ }
+ }
+
+ @Override
+ public int mark() {
+ return -1;
+ }
+
+ @Override
+ public void release(int marker) {
+ }
+
+ @Override
+ public int index() {
+ return position;
+ }
+
+ @Override
+ public void seek(int index) {
+ position = index;
+ }
+
+ @Override
+ public int size() {
+ return input.length();
+ }
+
+ @Override
+ public String getSourceName() {
+ return IntStream.UNKNOWN_SOURCE_NAME;
+ }
+ }
}
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
index 6d62a273448..189c60c1f36 100644
---
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
@@ -26,7 +26,6 @@ import
org.apache.doris.nereids.DorisParser.SingleStatementContext;
import org.apache.doris.nereids.parser.CaseInsensitiveStream;
import org.apache.doris.nereids.parser.ParseErrorListener;
-import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.atn.PredictionMode;
@@ -77,7 +76,7 @@ public final class DorisSqlParser {
/** Build a freshly configured lexer for advanced callers that want to
walk tokens directly. */
public DorisLexer newLexer(String sql) {
- DorisLexer lexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ DorisLexer lexer = new
DorisLexer(CaseInsensitiveStream.fromString(sql));
lexer.isNoBackslashEscapes = noBackslashEscapes;
return lexer;
}
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/CaseInsensitiveStreamTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/CaseInsensitiveStreamTest.java
new file mode 100644
index 00000000000..7544fc6278a
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/CaseInsensitiveStreamTest.java
@@ -0,0 +1,245 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.nereids.parser.CaseInsensitiveStream;
+
+import org.antlr.v4.runtime.BaseErrorListener;
+import org.antlr.v4.runtime.CharStream;
+import org.antlr.v4.runtime.CharStreams;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.IntStream;
+import org.antlr.v4.runtime.RecognitionException;
+import org.antlr.v4.runtime.Recognizer;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.misc.Interval;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+
+class CaseInsensitiveStreamTest {
+ private static final String SOURCE_NAME = "case-insensitive-stream-test";
+ private static final String DESERET_SMALL_LONG_I = new
String(Character.toChars(0x10428));
+ private static final String UNPAIRED_HIGH_SURROGATE =
Character.toString((char) 0xD801);
+
+ @Test
+ void matchesReferenceLookaheadTextAndNavigation() {
+ List<String> inputs = Arrays.asList(
+ "",
+
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_$",
+ "éÉıſß中",
+ "a" + DESERET_SMALL_LONG_I + "Z",
+ "unpaired" + UNPAIRED_HIGH_SURROGATE,
+ "select 'MiXeD 中文😀' -- comment\nfrom `Table`");
+
+ for (String input : inputs) {
+ CharStream actual = new
CaseInsensitiveStream(CharStreams.fromString(input, SOURCE_NAME));
+ CharStream reference = new
ReferenceCaseInsensitiveStream(CharStreams.fromString(input, SOURCE_NAME));
+ assertEquivalentStream(input, actual, reference);
+
+ CharStream stringBacked = CaseInsensitiveStream.fromString(input);
+ CharStream defaultReference = new
ReferenceCaseInsensitiveStream(CharStreams.fromString(input));
+ assertEquivalentStream(input, stringBacked, defaultReference);
+ }
+ }
+
+ private static void assertEquivalentStream(String input, CharStream
actual, CharStream reference) {
+ Assertions.assertEquals(reference.size(), actual.size());
+ Assertions.assertEquals(reference.getSourceName(),
actual.getSourceName());
+ for (int start = 0; start <= reference.size(); start++) {
+ for (int end : new int[] {start - 1, start, reference.size() - 1,
reference.size() + 2}) {
+ Assertions.assertEquals(reference.getText(Interval.of(start,
end)),
+ actual.getText(Interval.of(start, end)),
+ "input=" + input + ", interval=" + start + ".." + end);
+ }
+ }
+
+ for (int position = 0; position <= reference.size(); position++) {
+ reference.seek(position);
+ actual.seek(position);
+ Assertions.assertEquals(reference.index(), actual.index());
+ for (int offset = -reference.size() - 2; offset <=
reference.size() + 2; offset++) {
+ Assertions.assertEquals(reference.LA(offset),
actual.LA(offset),
+ "input=" + input + ", position=" + actual.index() + ",
offset=" + offset);
+ }
+ Assertions.assertEquals(reference.mark(), actual.mark());
+ reference.release(-1);
+ actual.release(-1);
+ }
+
+ reference.seek(0);
+ actual.seek(0);
+ while (reference.LA(1) != IntStream.EOF) {
+ Assertions.assertEquals(reference.LA(1), actual.LA(1));
+ reference.consume();
+ actual.consume();
+ }
+ Assertions.assertEquals(reference.index(), actual.index());
+ Assertions.assertEquals(IntStream.EOF, actual.LA(1));
+ Assertions.assertThrows(IllegalStateException.class,
reference::consume);
+ Assertions.assertThrows(IllegalStateException.class, actual::consume);
+ }
+
+ @Test
+ void matchesReferenceTokensAcrossSqlAndStringModes() {
+ List<String> inputs = new ArrayList<>(Arrays.asList(
+ "",
+ "select lower_name, Mixed_Name, UPPER_NAME from db.tbl where
id = 1",
+ "ſelect 1, ıd from café",
+ "select " + DESERET_SMALL_LONG_I + "_name, 中文列 from 数据表",
+ "select 1.2e3x, 1.2z, .5d, 2.3W",
+ "select 'lowerCase 中文😀', \"MiXeD\", `QuotedName` -- lower
comment\n"
+ + "from tbl /* Mixed 中文😀 */ where c = 'a\\'b'",
+ "/*+ SET_VAR(query_timeout=1) */ select value from table_name",
+ "'unterminated\\",
+ "-- comment without newline"));
+ inputs.addAll(randomInputs());
+
+ for (boolean noBackslashEscapes : new boolean[] {false, true}) {
+ for (String input : inputs) {
+ Assertions.assertEquals(referenceSnapshot(input,
noBackslashEscapes),
+ actualSnapshot(input, noBackslashEscapes),
+ () -> "input=" + input + ", noBackslashEscapes=" +
noBackslashEscapes);
+ }
+ }
+ }
+
+ private static List<String> randomInputs() {
+ String[] alphabet = {
+ "a", "z", "A", "Z", "0", "9", "_", "$", " ", "\t", "\n",
+ "'", "\"", "`", "\\", "-", "/", "*", ".", ",", "(", ")", "+",
"=",
+ "é", "ı", "ſ", "中", DESERET_SMALL_LONG_I
+ };
+ Random random = new Random(20260902L);
+ List<String> inputs = new ArrayList<>();
+ for (int caseIndex = 0; caseIndex < 200; caseIndex++) {
+ int length = random.nextInt(65);
+ StringBuilder input = new StringBuilder();
+ for (int index = 0; index < length; index++) {
+ input.append(alphabet[random.nextInt(alphabet.length)]);
+ }
+ inputs.add(input.toString());
+ }
+ return inputs;
+ }
+
+ private static List<String> actualSnapshot(String sql, boolean
noBackslashEscapes) {
+ DorisLexer lexer = new DorisSqlParser(noBackslashEscapes,
false).newLexer(sql);
+ return snapshot(lexer);
+ }
+
+ private static List<String> referenceSnapshot(String sql, boolean
noBackslashEscapes) {
+ DorisLexer lexer = new DorisLexer(
+ new
ReferenceCaseInsensitiveStream(CharStreams.fromString(sql)));
+ lexer.isNoBackslashEscapes = noBackslashEscapes;
+ return snapshot(lexer);
+ }
+
+ private static List<String> snapshot(DorisLexer lexer) {
+ ErrorCollector errors = new ErrorCollector();
+ lexer.removeErrorListeners();
+ lexer.addErrorListener(errors);
+ CommonTokenStream tokenStream = new CommonTokenStream(lexer);
+ tokenStream.fill();
+
+ List<String> snapshot = new ArrayList<>();
+ for (Token token : tokenStream.getTokens()) {
+ snapshot.add(token.getType() + "|" + token.getChannel() + "|" +
token.getText()
+ + "|" + token.getStartIndex() + "|" + token.getStopIndex()
+ + "|" + token.getLine() + "|" +
token.getCharPositionInLine()
+ + "|" + token.getTokenIndex());
+ }
+ snapshot.addAll(errors.errors);
+ return snapshot;
+ }
+
+ private static class ErrorCollector extends BaseErrorListener {
+ private final List<String> errors = new ArrayList<>();
+
+ @Override
+ public void syntaxError(Recognizer<?, ?> recognizer, Object
offendingSymbol, int line,
+ int charPositionInLine, String message, RecognitionException
exception) {
+ errors.add("ERROR|" + line + "|" + charPositionInLine + "|" +
message);
+ }
+ }
+
+ private static class ReferenceCaseInsensitiveStream implements CharStream {
+ private final CharStream stream;
+
+ ReferenceCaseInsensitiveStream(CharStream stream) {
+ this.stream = stream;
+ }
+
+ @Override
+ public String getText(Interval interval) {
+ return stream.getText(interval);
+ }
+
+ @Override
+ public void consume() {
+ stream.consume();
+ }
+
+ @Override
+ public int LA(int offset) {
+ int result = stream.LA(offset);
+ switch (result) {
+ case 0:
+ case IntStream.EOF:
+ return result;
+ default:
+ return Character.toUpperCase(result);
+ }
+ }
+
+ @Override
+ public int mark() {
+ return stream.mark();
+ }
+
+ @Override
+ public void release(int marker) {
+ stream.release(marker);
+ }
+
+ @Override
+ public int index() {
+ return stream.index();
+ }
+
+ @Override
+ public void seek(int index) {
+ stream.seek(index);
+ }
+
+ @Override
+ public int size() {
+ return stream.size();
+ }
+
+ @Override
+ public String getSourceName() {
+ return stream.getSourceName();
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
