andygrove commented on PR #496:
URL: https://github.com/apache/datafusion-comet/pull/496#issuecomment-2140951878
The change in this PR worked fine with q14 but caused this failure with q1:
```
py4j.protocol.Py4JJavaError: An error occurred while calling
o200.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task
161 in stage 8.0 failed 4 times, most recent failure: Lost task 161.3 in stage
8.0 (TID 195) (192.168.86.32 executor 0):
org.apache.comet.CometNativeException: General execution error with reason
org.apache.comet.CometNativeException: called `Option::unwrap()` on a `None`
value
at comet::errors::init::{{closure}}(__internal__:0)
at std::panicking::rust_panic_with_hook(__internal__:0)
at std::panicking::begin_panic_handler::{{closure}}(__internal__:0)
at
std::sys_common::backtrace::__rust_end_short_backtrace(__internal__:0)
at rust_begin_unwind(__internal__:0)
at core::panicking::panic_fmt(__internal__:0)
at core::panicking::panic(__internal__:0)
at arrow_row::row_lengths(__internal__:0)
at arrow_row::RowConverter::append(__internal__:0)
at arrow_row::RowConverter::convert_columns(__internal__:0)
at
core::iter::adapters::map::map_try_fold::{{closure}}(__internal__:0)
at arrow_row::RowConverter::append(__internal__:0)
at arrow_row::RowConverter::convert_columns(__internal__:0)
at
<datafusion_physical_plan::aggregates::group_values::row::GroupValuesRows as
datafusion_physical_plan::aggregates::group_values::GroupValues>::intern(__internal__:0)
at
<datafusion_physical_plan::aggregates::row_hash::GroupedHashAggregateStream as
futures_core::stream::Stream>::poll_next(__internal__:0)
at
comet::execution::jni_api::Java_org_apache_comet_Native_executePlan::{{closure}}(__internal__:0)
```
I am surprised that having the projection produce larger batches would cause
something like this, but I am not yet familiar with the scala/rust interactions
about this, so there is probably something that I am not understanding.
@viirya @sunchao Can you help explain why my change is causing issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
