Rich-T-kid opened a new issue, #26945:
URL: https://github.com/apache/datafusion/issues/26945
### Is your feature request related to a problem or challenge?
When Parquet RLE-to-dictionary conversion is enabled, the expression
optimizer rewrites two- and three-value `IN` lists on dictionary string columns
into equality chains. Each equality evaluates the dictionary values and maps
its result through the keys separately. A single `IN` expression can evaluate
membership on the dictionary values once and map one result through the keys.
TPC-H Q12 hits this with `l_shipmode IN ('MAIL', 'SHIP')` and
`o_orderpriority IN ('1-URGENT', '2-HIGH')`. In a local main-versus-patch A/B
with `DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY=true`, Q12 improved
by 3.1-3.7% across focused and full-suite runs. Q13 and Q22, used as
unchanged-plan controls, were flat.
### Describe the solution you would like
Retain short literal `IN` and `NOT IN` expressions for dictionary columns
whose values are `Utf8` or `Utf8View`. Keep the existing rewrite behavior for
non-dictionary string columns.
### Describe alternatives you have considered
The dictionary physical expression could recognize and fuse equality chains,
but preserving the original membership expression is smaller and avoids
creating the repeated work in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
