morrySnow opened a new pull request, #67455:
URL: https://github.com/apache/doris/pull/67455
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The three `limitClause` alternatives repeat the `LIMIT INTEGER_VALUE`
prefix. ANTLR therefore inspects up to four or five tokens before selecting an
alternative, even though the three forms can be distinguished after consuming
the common prefix.
Factor the prefix once and dispatch on `OFFSET`, comma, or the rule end. A
local grammar action preserves the existing generated `limit` and `offset`
token fields for MySQL's `LIMIT offset, count` form. The accepted SQL syntax
and FE semantics do not change.
This PR also adds reusable JMH workloads for evaluating the P5 local-rule
candidates and a focused test for all three LIMIT forms. Predicate, relation,
primary-expression, SHOW, and DDL candidates were profiled and benchmarked;
candidates without a stable compatible improvement were discarded.
### Benchmark
Environment and method:
- Baseline: `807454f5d92` (benchmark present, before the grammar change)
- Candidate: `cba9bef4b07`
- Baseline parser artifact SHA-256:
`7ab3247fc905dcb519f80fd7a95f3a12e038bdc9d3b25a0825eeda2fb7df321d`
- Candidate parser artifact SHA-256:
`cfa2b289c199bbbb36a42615ce46b6633a0bbe16f595bd178796c976a08c4ebb`
- macOS 15 arm64, OpenJDK 17.0.20.1, JMH 1.37, 1 GiB heap
- 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Runs were interleaved as B1-C1-C2-B2. Values below are `us/op`; individual
values show JMH's 99.9% confidence error.
```shell
java -Xms1g -Xmx1g -jar <benchmark.jar> \
'LocalRulePrefixBenchmark.parseTargetRule' \
-p workload=control,limit -f 3 -wi 4 -i 7 -w 300ms -r 400ms \
-prof gc -rf json
java -Xms1g -Xmx1g -jar <benchmark.jar> \
'LocalRulePrefixBenchmark.(parseEndToEnd|parsePreTokenized)' \
-p workload=control,limit -f 3 -wi 4 -i 7 -w 300ms -r 400ms \
-prof gc -rf json
```
| Benchmark | Baseline (B1 / B2) | Candidate (C1 / C2) | Mean change |
Baseline -> candidate B/op |
|---|---:|---:|---:|---:|
| Target `limitClause` | 0.2876±0.0398 / 0.2530±0.0288 | 0.2622±0.0304 /
0.2338±0.0056 | -8.2% | 1,136 -> 1,136 |
| LIMIT end-to-end | 341.998±96.097 / 286.276±5.505 | 292.862±5.802 /
303.246±32.851 | -5.1% | 532,244 -> 531,343 (-0.2%) |
| LIMIT pre-tokenized | 294.078±23.005 / 284.602±6.383 | 294.394±18.844 /
282.062±17.540 | -0.4% | 447,332 -> 447,332 |
| Control end-to-end | 54.008±1.936 / 55.684±4.762 | 53.994±2.805 /
53.091±2.520 | -2.4% | 107,292 -> 105,905 (-1.3%) |
| Control pre-tokenized | 51.786±0.928 / 50.017±0.690 | 51.062±1.075 /
49.902±0.574 | -0.8% | 97,043 -> 97,385 (+0.4%) |
`ProfilingATNSimulator` shows that the three baseline LIMIT forms consume
128/160/128 lookahead tokens over 32 invocations, with maximum lookahead 4/5/4.
The candidate consumes 32 tokens with maximum lookahead 1. The generated parser
class also shrinks by 130 bytes. The improvement therefore comes from removing
the repeated adaptive lookahead; allocation is effectively unchanged.
Other evaluated P5 candidates were not retained:
- Predicate factoring regressed the target rule by 2.4%, the pre-tokenized
workload by 0.9%, and allocation by 0.3%.
- `relationPrimary` already dispatches with maximum lookahead 2; another
context layer had no justified benefit.
- The semantically equivalent primary-expression candidate regressed
representative workloads by 36%-48%. A faster variant accepted invalid `fn().*`
and `a[1].*` inputs and was rejected.
- The DDL candidate regressed its target rule by 4.5% and the pre-tokenized
workload by 2.4%; SHOW already has maximum lookahead 3.
Correctness corpus:
- 4,610 tracked SQL files were compared in legacy and ANSI modes (9,220
parser cases).
- Baseline and candidate match for accept/reject, statement count, exception
type, and first error line/position.
- Result signature SHA-256:
`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`.
- Targeted valid and invalid LIMIT error-signature comparisons also have no
differences.
- The lexer grammar is unchanged, so tokenization is unaffected.
Raw JMH JSON and corpus signatures are retained under
`/private/tmp/doris-p5-benchmark/` in the benchmark environment; artifact
hashes and complete reproduction commands are included above.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Tested with:
- `mvn -pl fe-sql-parser -am clean test`: 171 tests passed; Checkstyle
passed.
- `./run-fe-ut.sh --run org.apache.doris.nereids.parser.LimitClauseTest`: 3
tests passed.
- Parser corpus and targeted error comparisons described above: no
differences.
- FE-only `./build.sh --fe` with a fresh output directory: passed and
produced `Successfully build Doris`.
- JMH and ATN profiling described above.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
