Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Felipe Aramburu
Got it. I think I am just going to use arraydata::make for this for now. Thanks a bundle! Felipe On Fri, Mar 22, 2019 at 8:43 AM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > It is not frowned upon to use the ArrayData::Make classes, you just have to > ensure the order of buffers ma

Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Francois Saint-Jacques
It is not frowned upon to use the ArrayData::Make classes, you just have to ensure the order of buffers matches what the specialized Array class expects (also matching the type you're passing to ArrayData). I'd say it is the "preferred" way if your data is already in the required layout, and the be

Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Wes McKinney
hi Felipe -- note you don't have to use the builder classes when you have the exact memory layout already, you can wrap you rmemory in arrow::Buffer and construct the arrays directly. I think it would be useful to add APIs for appending to builders with a bitmap. We don't have them now, though. Th

Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Felipe Aramburu
Is there a way to use a builder to be able to provide a bit per value LSB as is described in the documentation? I have this already and it seems silly to convert it to something else so that arrow can then make it the same format as what I had to begin with. I know there is the ArrayData class that

Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Francois Saint-Jacques
Actually, this specific method seems to use a byte per value as you questioned. I think it's worth adding documentation and an explicit warning if it confused me. I'll let bkietz chime in to comment on the usage. François On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques < fsaintjacq...@gma

Re: Creating Arrays from builders using bitmasks

2019-03-22 Thread Francois Saint-Jacques
Hello Felipe, it's a bit per value as per memory layout documentation. François On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu wrote: > In the builder base class I see this api > > > https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_b

Creating Arrays from builders using bitmasks

2019-03-22 Thread Felipe Aramburu
In the builder base class I see this api https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149 // Vector append. Treat each zero byte as a nullzero. If valid_bytes is null // assume all of length bits are valid. void Unsa