Thanks for the input. This is motivated by use-cases where suboptimal compute is Ok in order to preserve a lower memory profile. I wanted to be sure I wasn't overlooking a utility before I write my own.
-John On Tue, Mar 28, 2023 at 12:27 PM Will Jones <will.jones...@gmail.com> wrote: > Hi John, > > Arrays have a `Slice()` method that allows getting a zero-copy slices of > the array given an offset and a length. If you had a set of ranges it > wouldn't be too hard to write a function that creates a new chunked array > made up of these slices. > > Of course, there are likely cases where the overhead of creating lots of > slices costs more than materializing a whole new array. You wouldn't want > to get back a chunked array of length 1000 that is made up of 500 sliced > arrays. This is even more true if you are taking indices. I think that's > why no one has implemented such a function; it's complicated to detect when > making a zero-copy slice is better than creating a new array, so we always > just create a new array for take. But if you have a particular use case > where you know it makes sense, then I would go ahead and write a function > for that specific case. > > Best, > > Will > > On Tue, Mar 28, 2023 at 10:14 AM John Muehlhausen <j...@jgm.org> wrote: > > > Is there a way to pass a RecordBatch (or a batch wrapped as a Table) to > > Take and get back a Table composed of in-place (zero copy) slices of the > > input? I suppose this is not too hard to code, just wondered if there is > > already a utility. > > > > Result<Datum> Take(const Datum& values, const Datum& indices, > > const TakeOptions& options = TakeOptions::Defaults(), > > ExecContext* ctx = NULLPTR); > > >
