I agree this sounds like a good application for ExtensionType. At minimum, ExtensionType can be used to develop a working version of what you need to help guide further discussions.
On Mon, Jul 29, 2019 at 2:29 PM Francois Saint-Jacques <fsaintjacq...@gmail.com> wrote: > > Hello, > > if each record has a different size, then I suggest to just use a > Struct<Dim, List<T>> where Dim is a struct (or expand in the outer > struct). You can probably add your own logic with the recently > introduced ExtensionType [1]. > > François > [1] > https://github.com/apache/arrow/blob/f77c3427ca801597b572fb197b92b0133269049b/cpp/src/arrow/extension_type.h > > On Mon, Jul 29, 2019 at 3:15 PM Edward Loper <edlo...@google.com.invalid> > wrote: > > > > The intention is that each individual record could have a different size. > > This could be consistent within a given batch, but wouldn't need to be. > > For example, if I wanted to send a 3-channel image, but the image size may > > vary for each record, then I could use > > FixedSizeList<FixedSizeList<FixedSizeList<Int8>[3]>[-1]>[-1]. > > > > On Mon, Jul 29, 2019 at 1:18 PM Brian Hulette <bhule...@apache.org> wrote: > > > > > This isn't really relevant but I feel compelled to point it out - the > > > FixedSizeList type has actually been in the Arrow spec for a while, but it > > > was only implemented in JS and Java initially. It was implemented in C++ > > > just a few months ago. > > > > > > > Thanks for the clarification -- I was going based on the blame history for > > Layout.rst, but I guess it just didn't get officially documented there > > until the c++ implementation was added. > > > > -Edward > > > > > > > On Mon, Jul 29, 2019 at 7:01 AM Edward Loper <edlo...@google.com.invalid> > > > wrote: > > > > > > > The FixedSizeList type, which was added to Arrow a few months ago, is an > > > > array where each slot contains a fixed-size sequence of values. It is > > > > specified as FixedSizeList<T>[N], where T is a child type and N is a > > > signed > > > > int32 that specifies the length of each list. > > > > > > > > This is useful for encoding fixed-size tensors. E.g., if I have a > > > 100x8x10 > > > > tensor, then I can encode it as > > > > FixedSizeList<FixedSizeList<FixedSizeList<byte>[10]>[8]>[100]. > > > > > > > > But I'm also interested in encoding tensors where some dimension sizes > > > are > > > > not known in advance. It seems to me that FixedSizeList could be > > > extended > > > > to support this fairly easily, by simply defining that N=-1 means "each > > > > array slot has the same length, but that length is not known in > > > > advance." > > > > So e.g. we could encode a 100x?x10 tensor as > > > > FixedSizeList<FixedSizeList<FixedSizeList<byte>[10]>[-1]>[100]. > > > > > > > > Since these N=-1 row-lengths are not encoded in the type, we need some > > > way > > > > to determine what they are. Luckily, every Field in the schema has a > > > > corresponding FieldNode in the message; and those FieldNodes can be used > > > to > > > > deduce the row lengths. In particular, the row length must be equal to > > > the > > > > length of the child node divided by the length of the FixedSizeList. > > > E.g., > > > > if we have a FixedSizeList<byte>[-1] array with the values [[1, 2], [3, > > > 4], > > > > [5, 6]] then the message representation is: > > > > > > > > * Length: 3, Null count: 0 > > > > * Null bitmap buffer: Not required > > > > * Values array (byte array): > > > > * Length: 6, Null count: 0 > > > > * Null bitmap buffer: Not required > > > > * Value buffer: [1, 2, 3, 4, 5, 6, <unspecified padding bytes>] > > > > > > > > So we can deduce that the row length is 6/3=2. > > > > > > > > It looks to me like it would be fairly easy to add support for this. > > > E.g., > > > > in the FixedSizeListArray constructor in c++, if > > > > list_type()->list_size() > > > > is -1, then set list_size_ to values.length()/length. There would be no > > > > changes to the schema.fbs/message.fbs files -- we would just be > > > assigning a > > > > meaning to something that's currently meaningless (having > > > > FixedSizeList.listSize=-1). > > > > > > > > If there's support for adding this to Arrow, then I could put together a > > > > PR. > > > > > > > > Thanks, > > > > -Edward > > > > > > > > P.S. Apologies if this gets posted twice -- I sent it out a couple days > > > ago > > > > right before subscribing to the mailing list; but I don't see it on the > > > > archives, presumably because I wasn't subscribed yet when I sent it out. > > > > > > >
