On Thu, Aug 12, 2021 at 1:03 PM Jorge Cardoso Leitão < jorgecarlei...@gmail.com> wrote:
> I agree with Antoine that we should weigh the pros and cons of flatbuffers > (or protobuf or thrift for that matter) over a more human-friendly, > simpler, format like json or MsgPack. I also struggle a bit to reason with > the complexity of using flatbuffers for this. > Ultimately I think different representations of the format will emerge if compute IR is successful, and people will implement JSON/proto/thrift/etc versions of the IR. The requirements for the compute IR as I see it are: * Implementations in IR producer and consumer languages. * Strongly typed or the ability to easily validate a payload It seems like Protobuf, Flatbuffers and JSON all meet the criteria here. Beyond that, there's precedence in the codebase for flatbuffers (which is just to say that flatbuffers is the devil we know). Can people list other concrete requirements for the format? A non-requirement might be that there be _idiomatic_ implementations for every language arrow supports, for example. I think without agreement on requirements we won't ever arrive at consensus. The compute IR spec itself doesn't really depend on the specific choice of format, but we need to get some consensus on the format. > E.g. there is no async support for thrift, flatbuffers nor protobuf in > Rust, which e.g. means that we can't read neither parquet nor arrow IPC > async atm. These problems are usually easier to work around in simpler > formats. > Can you elaborate a bit on the lack of async support here and what it would mean for a particular in-memory representation to support async, and why that prevents reading a parquet file using async? Looking at JSON as an example, most libraries in the Rust ecosystem use serde and serde_json to serialize and deserialize JSON, and any async concerns occur at the level of a client/server library like warp (or some transitive dependency thereof like Hyper). Are you referring to something like the functionality implemented in tokio-serde-json? If so, I think you could probably build something for these other formats assuming they have serde support (flatbuffers notably does _not_, partially because of its incessant need to own everything), since tokio_serde is doing most of the work in tokio-serde-json. In any case, I don't think it's a requirement for the compute IR that there be a streaming transport implementation for the format. > > Best, > Jorge > > > > On Thu, Aug 12, 2021 at 2:43 PM Antoine Pitrou <anto...@python.org> wrote: > > > > > Le 12/08/2021 à 15:05, Wes McKinney a écrit : > > > It seems that one adjacent problem here is how to make it simpler for > > > third parties (especially ones that act as front end interfaces) to > > > build and serialize/deserialize the IR structures with some kind of > > > ready-to-go middleware library, written in a language like C++. > > > > A C++ library sounds a bit complicated to deal with for Java, Rust, Go, > > etc. developers. > > > > I'm not sure which design decision and set of compromises would make the > > most sense. But this is why I'm asking the question "why not JSON?" (+ > > JSON-Schema if you want to ease validation by third parties). > > > > (note I have already mentioned MsgPack, but only in the case a binary > > encoding is really required; it doesn't have any other advantage that I > > know of over JSON, and it's less ubiquitous) > > > > Regards > > > > Antoine. > > > > > > > To do that, one would need the equivalent of arrow/type.h and related > > > Flatbuffers schema serialization code that lives in arrow/ipc. If you > > > want to be able to completely and accurately serialize Schemas, you > > > need quite a bit of code now. > > > > > > One possible approach (and not go crazy) would be to: > > > > > > * Move arrow/types.h and its dependencies into a standalone C++ > > > library that can be vendored into the main apache/arrow C++ library. I > > > don't know how onerous arrow/types.h's transitive dependencies / > > > interactions are at this point (there's a lot of stuff going on in > > > type.cc [1] now) > > > * Make the namespaces exported by this library configurable, so any > > > library can vendor the Arrow types / IR builder APIs privately into > > > their project > > > * Maintain this "Arrow types and ComputeIR library" as an always > > > zero-dependency library to facilitate vendoring > > > * Lightweight bindings in languages we care about (like Python or R or > > > GLib/Ruby) could be built to the IR builder middleware library > > > > > > This seems like what is more at issue compared with rather projects > > > are copying the Flatbuffers files out of their project from > > > apache/arrow or apache/arrow-format. > > > > > > [1]: https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.cc > > > > > > On Thu, Aug 12, 2021 at 2:05 PM Andrew Lamb <al...@influxdata.com> > > wrote: > > >> > > >> I support the idea of an independent repo that has the arrow > flatbuffers > > >> format definition files. > > >> > > >> My rationale is that the Rust implementation has a copy of the > `format` > > >> directory [1] and potential drift worries me (a bit). Having a single > > >> source of truth for the format that is not part of the large mono repo > > >> would be a good thing. > > >> > > >> Andrew > > >> > > >> [1] https://github.com/apache/arrow-rs/tree/master/format > > >> > > >> On Wed, Aug 11, 2021 at 2:40 PM Phillip Cloud <cpcl...@gmail.com> > > wrote: > > >> > > >>> Hi all, > > >>> > > >>> I'd like to bring up an idea from a recent thread ([1]) about moving > > the > > >>> `format/` directory out of the primary apache/arrow repository. > > >>> > > >>> I understand from that thread there are some concerns about using > > >>> submodules, > > >>> and I definitely sympathize with those concerns. > > >>> > > >>> In talking with David Li (disclaimer: we work together at Voltron > > Data), he > > >>> has > > >>> a great idea that I think makes everyone happy: an > > `apache/arrow-format` > > >>> repository that is the official mirror for the flatbuffers IDL, that > > >>> library > > >>> authors should use as the source of truth. > > >>> > > >>> It doesn't require a submodule, yet it also allows external projects > > the > > >>> ability to access the IDL without having to interact with the main > > arrow > > >>> repository and is backwards compatible to boot. > > >>> > > >>> In this scenario, repositories that are currently copying in the > > >>> flatbuffers > > >>> IDL can migrate to this repository at their leisure. > > >>> > > >>> My motivation for this was around sharing data structures for the > > compute > > >>> IR > > >>> proposal ([2]). > > >>> > > >>> I can think of at least two ways for IR producers and consumers of > all > > >>> languages to share the flatbuffers IDL: > > >>> > > >>> 1. A set of bindings built in some language that other languages can > > >>> integrate > > >>> with, likely C++, that allows library users to build IR using an > > API. > > >>> > > >>> The primary downside to this is that we'd have to deal with > > >>> building another library while working out any kinks in the IR design > > and > > >>> I'd > > >>> rather avoid that in the initial phases of this project. > > >>> > > >>> The benefit is that IR components don't interact much with > > `flatbuffers` or > > >>> `flatc` directly. > > >>> > > >>> 2. A single location where the format lives, that doesn't require > > depending > > >>> on > > >>> a large multi-language repository to access a handful of files. > > >>> > > >>> I think the downside to this is that there's a bit of additional > > >>> infrastructure > > >>> to automate copying in `arrow-format`. > > >>> > > >>> The benefit there is that producers and consumers can immediately > start > > >>> getting > > >>> value from compute IR without having to wait for development of a new > > API. > > >>> > > >>> One counter-proposal might be to just put the compute IR IDL in a > > separate > > >>> repo, > > >>> but that isn't tenable because the compute IR needs arrow's type > > >>> information > > >>> contained in `Schema.fbs`. > > >>> > > >>> I was hoping to avoid conflating the discussion about bindings vs > > direct > > >>> flatbuffer usage (at least initially just supporting one, I predict > > we'll > > >>> need > > >>> both ultimately) with the decision about whether to split out the > > format > > >>> directory, but it's a good example of a choice for which splitting > out > > the > > >>> format directory would be well-served. > > >>> > > >>> I'll note that this doesn't block anything on the compute IR side, > just > > >>> wanted > > >>> to surface this in a parallel thread and see what folks think. > > >>> > > >>> [1]: > > >>> > > >>> > > > https://lists.apache.org/thread.html/rcebfcb4c5d0b7752fcdda6587871c2f94661b8c4e35119f0bcfb883b%40%3Cdev.arrow.apache.org%3E > > >>> [2]: > > >>> > > >>> > > > https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit#heading=h.ie0ne0gm762l > > >>> > > >
