tlfua commented on issue #7585:
URL: 
https://github.com/apache/datafusion-comet/issues/7585#issuecomment-6942355561

   One more input for this consolidation, from the review of 
https://github.com/apache/datafusion-comet/pull/7457 ( 
https://github.com/apache/datafusion-comet/pull/7457#pullrequestreview-5893863590
 and 
https://github.com/apache/datafusion-comet/pull/7457#discussion_r4571919828 ).
   
   With `spark.sql.ansi.enabled=false`, #7457 now falls back a TopK only when a 
later sort key can raise in any mode (`QueryPlanSerde.raisesInAnyMode`). That 
list is intentionally best-effort. Two gaps are left for a follow-up:
   
   1. **The list is incomplete.** Spark raises regardless of ANSI for, among 
others, `regexp_extract` with an invalid group index, `slice` with a start of 0 
or a negative length, `sequence` and `array_repeat` past the array size limit, 
`str_to_map` with a duplicate key under the default `mapKeyDedupPolicy`, and 
`make_ym_interval` on overflow. Comet 1.2 ran all of these natively too, so 
this is no regression, but a single shared list would cover them for TopK as 
well as for the other callers above.
   2. **The ANSI check reads the session flag, not each expression's eval 
mode.** A plan analyzed with ANSI on, such as a view that captured 
`spark.sql.ansi.enabled=true` or a DataFrame analyzed before the session flag 
changed, keeps ANSI `Cast` and arithmetic nodes. When it runs in a session with 
ANSI off, the TopK check takes the ANSI-off branch, so such a key can still 
raise on a row Spark's ordering skips. Asking each node for its own eval mode, 
as `cannotFail` already does for the arithmetic arms, would close this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to