alamb commented on issue #17857:
URL: https://github.com/apache/datafusion/issues/17857#issuecomment-3373053238
> > the key will be to figure out how to avoid slowing things down -- one
idea is to see if you can check for overflow once per batch (rather than once
per row) 🤔
>
> Do you mean checking the overflow at `left: StringViewArray combined
right: StringViewArray` instead of iterating in every single string of the view
array?
Yeah, I was thinking something like this where you could figure out how much
you could be extending the the array in one check
```
let total_lengths = array.iter().map(|v| v.len()).sum();
if i32::MAX - total_len() > total_len {
return err!("overflow")
}
```
That being said, adding a `try_append_value` in arrow-rs seems very
reasonable to me as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
