Reminder that today is the next Rust sync call Potential topics for discussion: * Ballista / DataFusion / etc * I remember that someone else was going to demo the use of Arrow but I can't remember exactly what that was now
On Tue, Feb 16, 2021 at 10:59 AM Dominik Moritz <domor...@cmu.edu> wrote: > Somewhat related, I tried to compile DataFusion to WASM and it didn’t work > because of some dependencies: > https://issues.apache.org/jira/projects/ARROW/issues/ARROW-11615. I wonder > whether DataFusion could have a feature flag for only shipping what is WASM > compatible? > > On Feb 15, 2021 at 12:13:04, Andrew Lamb <al...@influxdata.com> wrote: > > > Also, unrelated, is there a schedule for the sync calls? Will try and > > > > carve out some free time for the next one :) > > > > It is every other Wednesday at noon EST. Here is the original > announcement > > with more details: > > > > > https://lists.apache.org/thread.html/raa72e1a8a3ad5dbb8366e9609a041eccca87f85545c3bc3d85170cfc%40%3Cdev.arrow.apache.org%3E > > > > > > On Sun, Feb 14, 2021 at 8:29 AM Ruan Pearce-Authers < > r...@reservoirdb.com> > > wrote: > > > > I'd be interested in helping spec this out, it's especially tricky atm to > > > > track down issues when integrating DataFusion into the same binary as > other > > > > medium/large dependencies. > > > > > > Recently hit a really specific issue where DataFusion depends on Parquet, > > > > which supports various compression algs, including Brotli, and actix-web > > > > also depends on a slightly different Rust implementation of Brotli. Both > of > > > > these Brotli libs package the same underlying C lib separately, resulting > > > > in multiply-defined symbols compiling using msvc (and maybe on other > > > > platforms? didn't test in CI in the end). > > > > > > Got a quick interim hack [1] in place for my use case which doesn't > really > > > > use Parquet, so it's not pressing, but would be awesome to sort this > > > > properly upstream. > > > > > > I guess the only major tradeoff of having a comprehensive feature setup > is > > > > that it could make testing slightly harder, in terms of making sure > no-one > > > > breaks the build for specific feature combinations; this can always be > > > > mitigated with more CI though (yay, unlimited Actions minutes for public > > > > repos). > > > > > > Also, unrelated, is there a schedule for the sync calls? Will try and > > > > carve out some free time for the next one :) > > > > > > [1] > > > > > > > https://github.com/reservoirdb/arrow/commit/e63e157927a552ecf1a6f63ec401f0b6157b5468 > > > > > > -----Original Message----- > > > > From: Andrew Lamb <al...@influxdata.com> > > > > Sent: 14 February 2021 11:14 > > > > To: dev <dev@arrow.apache.org> > > > > Subject: [Rust] [DataFusion] Topic for next Rust Sync Call > > > > > > I would like to add the following item to the agenda call for the next > > > > Rust sync call: > > > > > > Dependencies > > > > > > Background: As the dependency stack gets larger, it will be harder to use > > > > DataFusion as an embedded query engine and the compile / dev times will > get > > > > higher. > > > > > > As we expand the supported functions of DataFusion this problem is likely > > > > to get worse. For example > > > > https://github.com/apache/arrow/pull/9243#discussion_r575716759 and > > > > https://github.com/apache/arrow/pull/9139 > > > > > > Proposal: Add Rust "features" to the datafusion crate and make many of > the > > > > new dependencies optional (so that we had features like regex and unicode > > > > and hash which would only pull in the dependencies / have those functions > > > > if the features were enabled.) This approach has worked well for Arrow > > > > (which has only chrono and num as required dependencies) > > > > > > >
