ODriscollBoys opened a new pull request, #20278:
URL: https://github.com/apache/hudi/pull/20278
### Describe the issue this Pull Request addresses
Closes #19449
The extended SQL parsers set `SQL_standard_keyword_behavior` from
`conf.ansiEnabled`, while Spark's own `AbstractParser` sets it from
`conf.enforceReservedKeywords` (`spark.sql.ansi.enabled &&
spark.sql.ansi.enforceReservedKeywords`, the latter defaulting to `false`).
Spark 4.x turns ANSI mode on by default, so every statement routed to the
extended parser (anything with a `BLOB` or `VECTOR` token) was parsed with ANSI
reserved keywords enforced, unlike the stock parser. The forked grammar then
cannot read `TIMESTAMP` as a type name or typed-literal prefix:
```sql
CREATE TABLE t (id BIGINT, ts TIMESTAMP, data BLOB) USING hudi
-- ParseException: no viable alternative at input 'TIMESTAMP'
```
### Summary and Changelog
Statements with a `BLOB` or `VECTOR` column now accept the same keywords as
stock Spark under ANSI mode, including a `TIMESTAMP` column type, `TIMESTAMP
'...'` literals and ANSI-reserved identifiers.
- `HoodieSpark{3_3,3_4,3_5,4_0,4_1,4_2}ExtendedSqlParser`: set
`SQL_standard_keyword_behavior` from `conf.enforceReservedKeywords`, matching
Spark's `AbstractParser`. The method exists on `SQLConf` in every supported
Spark version (3.3.0+).
- `TestBlobDataType`:
- New case `Test parse CREATE TABLE with BLOB column and TIMESTAMP under
ANSI mode`, which sets `spark.sql.ansi.enabled=true` so the regression
reproduces on every Spark profile (it fails without the fix with `no viable
alternative at input 'TIMESTAMP'`).
- Add the `TIMESTAMP` column and typed `TIMESTAMP` literal that the
primitive-type and typed-literal cases had left out because of this issue.
- The ANSI keyword-mode assertions for bare `false` / `null` now also set
`spark.sql.ansi.enforceReservedKeywords=true`, which is what selects that mode
in stock Spark.
### Impact
SQL with `BLOB`/`VECTOR` columns parses the same way as in stock Spark.
Users who set `spark.sql.ansi.enforceReservedKeywords=true` keep the ANSI
reserved-keyword behavior. No public API, config or storage format changes.
### Risk Level
Low. One line per parser, aligning keyword handling with the stock Spark
parser. Validated with `TestBlobDataType` on the Spark 3.5 profile (all 12
cases pass; the new case fails without the fix).
`SQLConf.enforceReservedKeywords` was checked to be present in the Spark 4.0.2,
4.1.1 and 4.2.0 catalyst jars.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
