ryanmeowy opened a new pull request, #13145:
URL: https://github.com/apache/seatunnel/pull/13145
### Purpose of this pull request
Fixes #13144
In the default ZETA SQL transform engine, a LIKE pattern whose `%` or `_` is
not at the very
beginning or the very end matches nothing, silently: rows are dropped with
no error and no
warning, and `NOT LIKE` with the same pattern keeps every row. Regex
metacharacters in the
pattern are interpreted as regex, and `LIKE NULL` throws
`NullPointerException`.
This is the out-of-the-box behaviour: `SQLTransform.java:68` defaults the
engine to ZETA, and
`ZetaSQLEngine.java:338` sends `where` predicates through `ZetaSQLFilter`.
**Root cause.** `likeExpr` (`:191-232`) and `notLikeExpr` (`:235-273`)
rewrite the pattern with
four positional checks — `startsWith("%")`, `endsWith("%")`,
`startsWith("_")`, `endsWith("_")`
plus one `"_.*"` special case — and feed the result to `Pattern.compile`.
Only the boundaries are
considered, so an interior wildcard stays in the string and is compiled as a
literal character;
and nothing is quoted, so user characters become regex syntax.
`rightVal.toString()` also runs
before the null check, which made that check dead.
**Fix.** Replace the positional rewriting with `likeMatches(pattern,
value)`, a single iterative
matcher shared by both methods: `%` consumes any sequence, `_` consumes
exactly one code point,
every other code point is compared literally. Both methods keep their
existing dispatch and now
check `rightVal` for null before using it.
Two deliberate properties of this matcher:
- *No regex escaping layer.* Matching characters directly means `'1.5'`,
`'a(b'` and `'$100^'`
are literal without any quoting, so the metacharacter subclass of the bug
cannot come back.
- *No catastrophic backtracking.* Translating `%` to `.*` — the obvious fix,
and what this change
did in its first revision — turns a pattern like `%a%a%a%a%a%a%a%a%a%a%b`
into an exponential
regex. Measured on a prototype of that approach: 11 wildcards against a
40-character value took
16 s, and the exact pattern used by the new unit test (10 wildcards,
64-character value) was
still running after 5 minutes. The iterative matcher keeps one backtrack
point per `%` and runs
in O(pattern × value): both cases finish in under 1 ms. A regex-based
translation would have
traded a silent wrong-answer bug for a hang, which is worse.
Behaviour kept from before:
- `Pattern.DOTALL` is still not applied, so `%` does not cross a newline
(`"a\nb" LIKE 'a%b'` is
false, as today). Aligning that with the SQL standard is a separate change.
- LIKE ... ESCAPE remains unsupported, so a literal `%` or `_` still cannot
be expressed. Also
separate; there is a `shortcut:` comment at the site.
- A pattern that legitimately starts and ends with a quote is no longer
stripped. The previous
`regex.startsWith("'") && regex.endsWith("'")` defence was dead code
(`StringValue` is already
unquoted by `getNotExcapedValue()`), and when it did fire it corrupted the
pattern.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased `dev` branch — no released version is affected
differently, since the
bug is present in every version that has ZETA LIKE support.
Previous behaviour versus behaviour after this PR, for a row `(1, "Alice",
20)` unless stated
otherwise. "SQL" is the expected SQL semantics.
| predicate | before | after | SQL |
|---|---|---|---|
| `name LIKE 'Al%'` | true | true | true |
| `name LIKE '%lic%'` | true | true | true |
| `name LIKE '_li%'` | true | true | true |
| `name LIKE 'A%ce'` | **false** | true | true |
| `name LIKE 'A%ic%e'` | **false** | true | true |
| `name LIKE 'A_i_e'` | **false** | true | true |
| `name LIKE '%li%ce'` | **false** | true | true |
| `name LIKE 'A%e'` | **false** | true | true |
| `name LIKE '%%%%'` | **false** | true | true |
| `name NOT LIKE 'A%ce'` | **true** | false | false |
| `'125' LIKE '1.5'` | **true** | false | false |
| `'axb' LIKE 'a.b'` | **true** | false | false |
| `name LIKE '''quoted'''` on `'quoted'` | **false** | true | true |
| `name LIKE 'a(b%'` | **PatternSyntaxException** | false | false |
| `name LIKE NULL` | **NullPointerException** | false | false |
Every change in that table moves the result toward the SQL truth value. In
particular the three
silent classes — interior wildcards, regex metacharacters, `LIKE NULL` —
stop producing wrong
answers or exceptions, and `NOT LIKE` stops inverting the result.
The SQL transform documentation does not spell out LIKE wildcard semantics
for the ZETA engine —
there is no `LIKE` section in `docs/en/transforms/sql-functions.md` to
contradict or to update, so
no documentation change is included here. No configuration option, public
API, data format or
default value is touched, so `incompatible-changes.md` is not affected: the
observable change is a
bug fix, not a contract change. The two known gaps (`%` not crossing
newlines, no
`LIKE ... ESCAPE`) are listed above and left as separate changes.
### How was this patch tested?
Unit tests, extending the existing `ZetaSQLFilterTest` (it already provides
`createFilter()`, so no
new test class). The two pre-existing LIKE tests only used `'Al%'` and
`'_li%'` — the only shapes
that worked before this PR. Six tests are added:
- `testLikeWithWildcardInTheMiddle` — `A%ce`, `A%ic%e`, `A_i_e`, `%li%ce`,
`A%e` match `Alice`
and not `Bob`; `NOT LIKE 'A%ce'` inverts.
- `testLikePatternsThatStillWork` — `Al%`, `%lic%`, `_li%` regression guard.
- `testLikeTreatsRegexMetacharactersAndBrokenGroupsAsLiterals` — `'125' LIKE
'1.5'` is false,
`'a(b%'` returns false instead of throwing, and a value that starts and
ends with a quote is not
stripped.
- `testLikeNullDoesNotThrow` — `LIKE NULL` and `NOT LIKE NULL` return false.
- `testLikeMatchesOneCodePointPerUnderscore` — `_` matches one code point,
so an astral character
(emoji) counts as one and `a__b` does not match `a😀b`.
- `testLikeWithManyWildcardsDoesNotBacktrackExponentially` — 10 interior
wildcards against a
64-character value complete within 5 s (`assertTimeoutPreemptively`). A
real guard, not
decoration: a regex-based implementation of the same pattern was still
running after 5 minutes.
If you want to check the behaviour by hand, the two-line version is:
```bash
./mvnw -pl seatunnel-transforms-v2 test -Dtest='ZetaSQLFilterTest'
```
Commands run locally:
```bash
# format (JDK 17)
./mvnw -pl seatunnel-transforms-v2 spotless:check
# BUILD SUCCESS
# the fix and its neighbouring tests
./mvnw -pl seatunnel-transforms-v2 test \
-Dtest='ZetaSQLFilterTest,DateTimeFunctionsTest,SQLDateTimeFunctionsTest'
# Tests run: 47, Failures: 0, Errors: 0
# ZetaSQLFilterTest 11 (5 before + 6 new)
# DateTimeFunctionsTest 21
# SQLDateTimeFunctionsTest 15
```
On the parent commit the new behaviour tests fail with `3 failures + 1 NPE`:
`A%ce` returns
false, `'125' LIKE '1.5'` returns true, the emoji case returns false, and
`LIKE NULL` throws —
so these tests do detect the bug rather than passing vacuously.
Full module (`seatunnel-transforms-v2`, 1901 tests) reports 2 failures / 16
errors, all in
`PythonTransformTest`, and the identical 2F/16E occur on an unmodified `dev`
checkout on this
machine: `python_executable=python3` resolves to `/usr/bin/python3`, which
is not listed in the
local `seatunnel.transform.python.allowed-executables` server property.
Environment only,
unrelated to this change.
Beyond the unit tests, the fix was cross-checked against an independently
written dynamic
programming LIKE matcher on 61,609 random pattern/value pairs: zero
mismatches (the only two
divergences involve `_` against an astral character, where the
code-point-aware implementation is
the correct one).
### Check list
- [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— not applicable, no new dependency or binary.
- [ ] If necessary, please update the documentation to describe the new
feature.
— not applicable, no new option or behaviour surface; this restores
documented-standard
LIKE semantics rather than adding a feature. Note that the ZETA LIKE
semantics are not
currently documented at all (no `LIKE` section in
`docs/en/transforms/sql-functions.md`),
which could be a separate docs improvement if maintainers want it.
- [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility
caused by this PR. — not applicable, no contract change (see above).
- [ ] Connector-specific checks (`plugin-mapping.properties`,
`seatunnel-dist` pom, label scope,
e2e, `plugin_config`) — not applicable, this is not a connector change.
Diff is limited to `ZetaSQLFilter.java` and `ZetaSQLFilterTest.java`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]