Hey Nate, Thanks for doing this! Would you be interested in putting that commit up as a draft PR for discussion? I think we can discuss there.
I'm not sure anyone is actively working on RLE or other encoding schemes at the moment. -David On Mon, Nov 8, 2021, at 13:19, Nate Bauernfeind wrote: > I've written up the ColumnBag proposal addressing items 1 and 2 on the > list. I'm open to any and all feedback/suggestions. > > I'd be happy to add item 3 (binary metadata) to the proposed change set. > Let me know if you want me to whip up the initial suggestion for that > version (and whether or not to keep it separate from ColumnBag). > > Would RLE related efforts change the structure of RecordBatch or ColumnBag > (if accepted)? > > Here is the brief history-discussion around why ColumnBag: > https://docs.google.com/document/d/1jsmmqLTyJkU8fx0sUGIqd6yu72N4v9uHFsuGSgB_DfE/ > > Here is a brief commit doctoring up the flatbuffer to support this version > of the proposed change: > https://github.com/nbauernfeind/arrow/tree/column_bag_demo_v1 > > I don't know if it's better to comment in the document or bring comments > back to the list. If it ends up being document heavy, then I'll summarize > the main points back on the list. > > I think I'll get started on a Java impl just to learn more even if it ends > up being extra work. > > Looking forward to your feedback, > Nate > > On Mon, Aug 9, 2021 at 10:06 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > I'm still interested in RLE related effort, but not sure about my available > > bandwidth (which is why I haven't made more of an effort there). > > > > On Tue, Aug 3, 2021 at 6:00 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > Another Flatbuffers/Message.fbs project we should rekindle soon, in > > > addition to the schema evolution/replacement question which has been > > > raised with Flight, is that of sparse/compressed data (e.g. RLE). I > > > have a vacation plus some travel coming up so won't be able to devote > > > meaningful attention to this until the last part of August, but would > > > like to help it move forward. > > > > > > > > > On Tue, Jul 27, 2021 at 1:40 PM David Li <lidav...@apache.org> wrote: > > > > > > > > Hey Nate, > > > > > > > > For the first two points, semantically I'm tempted to think of it more > > > like the ability to send a "bag of columns" according to some schema (and > > > hence columns could have differing lengths or even be absent). This could > > > be a new structure alongside a record batch, which is semantically like a > > > "slice of a table" (and hence rectangular and complete), instead of > > > exposing existing users of RecordBatch to rather different behavior. > > > > > > > > For #3, a different thread was discussing some of the points there - it > > > sounds like it may be possible to relax from map<string, string> to > > > map<string, binary>. > > > > > > > > -David > > > > > > > > On Mon, Jul 26, 2021, at 11:01, Nate Bauernfeind wrote: > > > > > Wes suggested that maybe there are enough new ideas that it may make > > > sense > > > > > to evolve-past the existing structures rather than to bolt-on new > > > > > functionality. I would like to learn what requirements exist should > > new > > > > > structures be adopted, and if applicable, would like to turn this > > into > > > a > > > > > full POC proposal. > > > > > > > > > > These are the features that I feel are missing from the existing > > > design: > > > > > - the ability to notify that the columns are not consistent in length > > > (e.g. > > > > > setting RecordBatch.length to -1; and give the arrow/flight user the > > > true > > > > > FieldNode lengths). > > > > > - the ability to skip top-level field nodes that have length 0 at a > > > small > > > > > cost (such as in a bitset) > > > > > - the ability to embed binary payload in the Message flatbuffer > > wrapper > > > > > (instead of String payload only) > > > > > - the ability to concurrently use more than one schema (the most > > > likely API > > > > > will look like how one identifies a dictionary. ideally dictionaries > > > could > > > > > be shared across field nodes in a schema or across schemas in the > > same > > > > > flight) > > > > > > > > > > What other features, or improvements, could/should be considered? Any > > > > > strong opinions against the ideas above? (Remember, that a goal of > > > mine is > > > > > to be able to send a RecordBatch of rows that were modified > > intersected > > > > > only by the field-nodes that have changed (including those with only > > > inner > > > > > node changes); thus the columns are a subset of the full schema and > > > that > > > > > the length of each node is independent of the other). > > > > > > > > > > On Fri, Jul 9, 2021 at 9:26 AM Wes McKinney <wesmck...@gmail.com> > > > wrote: > > > > > > It sounds like we may want to discuss some potential evolutions of > > > the > > > > > > Arrow binary protocol (for example: new Message types). Certainly a > > > > > > can of worms but rather than trying to bolt some new functionality > > > > > > onto the existing structures, it might be better to support the new > > > > > > use cases through some new structures which will be more clear cut > > > > > > from a forward compatibility standpoint. > > > > > > > > > > Nate > > > > > > > > > > -- > > > > > > > > > > > > > -- >
