Arrow with indirection?

D and other optimizations gained by processing contiguous data. Am I missing something in my reading of the Arrow docs? Does anyone have thoughts/recommendations, or is Arrow just not a good fit for this kind of thing? Thanks, -- Andrew Bell andrew.bell...@gmail.com

Re: Arrow with indirection?

>> but it is not as memory efficient. Because you are doing a separate memory >> swap for each array. >> >> This is why arrow compute libraries will sometimes convert to a row based >> format for certain operations. >> >> On Thu, Sep 14, 2023, 8:21 AM Andrew

Re: [C++] [Parquet] Questions about batch reading byte arrays

once to get the total data size and then read again for the data) is really worth it. If you can *guess* the column size and then just reallocate if you exceed that size, you might be better off. Good luck, -- Andrew Bell andrew.bell...@gmail.com

Re: [DISCUSS][C++] Store C++ shared_ptr in arrow table

On Wed, Oct 9, 2024, 12:27 PM Yi Cao wrote: > If I place these shared ptrs in a vector, how can I make this vector saved > in Arrow table as a column? Is it possible? > What do you mean by "saved"? I don't understand the point of placing shared pointers in an arrow array. It's essentially equiv

Re: [DISCUSS][C++] Store C++ shared_ptr in arrow table

r, I can > do IntBuilder to build an array and make a record batch out of it. > > Hope this provides a bit of clarity. Thank you. > > On Wed, 9 Oct 2024 at 19:12, Andrew Bell wrote: >> >> >> On Wed, Oct 9, 2024, 12:27 PM Yi Cao wrote: >>> >>>

Re: [DISCUSS][C++] Store C++ shared_ptr in arrow table

s in a C++ container (vector/deque/whatever)? -- Andrew Bell andrew.bell...@gmail.com

Re: [DISCUSS][C++] Store C++ shared_ptr in arrow table

. This is making assumptions. I don't think the OP's request was specific enough to draw any conclusions about their system needs or design. -- Andrew Bell andrew.bell...@gmail.com

Writing Field Metadata

t metadata associated with schema fields written to a parquet file? Or do I need to use some other command to see field-level metadata? Thanks, -- Andrew Bell andrew.bell...@gmail.com

Re: Writing Field Metadata

missing something). > However, I believe the default behavior of the Arrow C++ Parquet > implementation is to not store the Arrow Schema so you'll have to opt > into that behavior to get what you want by enabling store_schema [1] > > [1] https://arrow.apache.org/docs/cpp/parque

Feather Specification?

Hi, Is there a specification for the Feather file format? All I can find is https://arrow.apache.org/docs/python/feather.html, which is insufficient to exactly describe the format. Thanks, -- Andrew Bell andrew.bell...@gmail.com