sam-1112 commented on PR #5415:
URL:
https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5468588500
Thanks — confirmed against `da20a2c4`. The earlier 105-byte nesting and
992-byte capture witnesses were rejected, but this residual pattern still sat
exactly on the remaining caps: it has no captures, quantifier depth 8, and
structural size 4096 when every character class is counted as size 1. JDK 21
compiles it, while the locked `regex` 1.13.1 returns
`CompiledTooBig(10485760)`. Under the previous routing, the pattern was
therefore admitted to default native `RLike`, where the compilation error would
be propagated without a dispatcher retry.
I addressed this in `4b699afb7` using conservative JVM routing rather than
compile-then-fallback.
`parseClass` now charges for each class atom and range endpoint, including
escaped literals. The range hyphen and leading `^` negation do not add cost.
This cost feeds into the existing `MaxExpansion` accounting, so the first
`{256}` component of the reported shape already exceeds 4096 and remains off
automatic native routing. I did not lower `MaxQuantifierNesting` or add a
blanket multiplier for negated classes.
Ordinary classes remain admitted, including:
- `[^x]{256}`
- `[a-zA-Z0-9_]{256}`
- `"[^;]{256}" * 16`
The analyzer and routing regressions reconstruct the exact reported pattern
and assert `pattern.length == 1552`. The routing test uses the DataFrame
`col("s").rlike(pattern)` API so the SQL parser cannot rewrite the class
backslashes. The results match Spark, and EXPLAIN confirms that the expression
uses the JVM codegen dispatcher.
This remains a conservative structural heuristic rather than a proof against
the Rust engine's 10 MiB compiled-size limit. As before,
`allowIncompatible=true` may explicitly force a default-collation literal onto
the native path, including this pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
