SemyonSinchenko commented on PR #1073:
URL:
https://github.com/apache/datafusion-comet/pull/1073#issuecomment-2489236855
Thanks for comments and suggestions!
This PR is ready for the review again.
What were changed from the last round of the review:
- I added tests for the `array_prepend` (spark 3.5+) and enabled the test
for `array_append` (spark 4.0+);
- I added tests for the corner cases:
* Index is negative;
* Index is bigger than the array length;
* Index is negative and it's abs is greater than the array length;
* Test of the fallback to spark (udf as child);
* Value to insert is actually `null`;
- I fixed the native part behavior for some of the corner cases;
These tests pointed my attention to the uncovered parts of the code that I
fixed and I also added couple of additional tests to the `native` part. I
revisited how spark do array_insert and it is more tricky than I realized at
the first glance. I added few additional comments to the native part of the
implementation and fixed the behavior.
At the moment this PR is tested in multiple ways:
- Basic tests in native part that are useful because easy to debug and very
fast to run;
- Basic tests in native part for the so called "legacy mode" in spark;
- Basic tests on the scala side;
- Logical tests on small data for corner cases (negative, positive, long,
short, null, etc.) on the scala side;
- Tests in array_prepend and array_append that are calling `array_insert`
under the hood (NULLS, different data types, etc.);
- Test of the fallback to the Spark in case when one of children is not
supported by Comet;
So, it looks to me, that all the possible cases are covered and the behavior
is the same like in spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
