oh, perfect. I'll just link the JIRAs. Thanks Kou! Aldrin Montana Computer Science PhD Student UC Santa Cruz
On Mon, Jul 25, 2022 at 1:53 PM Sutou Kouhei <k...@clear-code.com> wrote: > Hi, > > https://issues.apache.org/jira/browse/ARROW-17092 may be > related. > > Thanks, > -- > kou > > In <CAC-Ect-+c-62BQwJ432UTab+n0Mr2k36Hco=6dawwt5ck7p...@mail.gmail.com> > "Re: [Rust] IPC Format / Feather support in Datafusion" on Mon, 25 Jul > 2022 13:39:54 -0700, > Aldrin <akmon...@ucsc.edu.INVALID> wrote: > > > It seems unfortunate to me that the feather file format doc page [1] > > appears to have been forgotten when those IPC file format docs were > written > > [2][3]. I will find or make a JIRA to make this info consistent in the > docs. > > > > [1]: https://arrow.apache.org/docs/python/feather.html > > [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format > > [3]: https://arrow.apache.org/docs/cpp/ipc.html > > > > Aldrin Montana > > Computer Science PhD Student > > UC Santa Cruz > > > > > > On Fri, Jul 22, 2022 at 2:46 PM Will Jones <will.jones...@gmail.com> > wrote: > > > >> FYI It looks like there is active work to change the Python [1] and R > [2] > >> implementations (built on C++) to write .arrow instead of .feather. > >> > >> [1] https://issues.apache.org/jira/browse/ARROW-17089 > >> [2] https://issues.apache.org/jira/browse/ARROW-17088 > >> > >> On Fri, Jul 22, 2022 at 11:16 AM Aldrin <akmon...@ucsc.edu.invalid> > wrote: > >> > >> > sorry, I meant "...especially *for* the rust community if they are > just > >> > using IPC directly for file formats." > >> > > >> > Aldrin Montana > >> > Computer Science PhD Student > >> > UC Santa Cruz > >> > > >> > > >> > On Fri, Jul 22, 2022 at 11:14 AM Aldrin <akmon...@ucsc.edu> wrote: > >> > > >> > > I always assumed IPC was when it was in memory, feather was when it > was > >> > > persisted as files. That distinction/indirection may be totally > moot, > >> > > though, especially if the rust community is just using IPC directly > for > >> > > file formats. > >> > > > >> > > In my mind, feather V1 came just before IPC and started using IPC > since > >> > > V2. I'm not sure if a feather V3 would ever diverge from IPC format > or > >> if > >> > > feather adds anything that's more filesystem friendly (versus other > >> > storage > >> > > system interfaces) or makes filesystem performance more predictable. > >> > > > >> > > > >> > > Aldrin Montana > >> > > Computer Science PhD Student > >> > > UC Santa Cruz > >> > > > >> > > > >> > > On Fri, Jul 22, 2022 at 10:49 AM Andrew Lamb <al...@influxdata.com> > >> > wrote: > >> > > > >> > >> For what it is worth, in the Rust Arrow community, I have only ever > >> > heard > >> > >> the format referred to as "IPC Format" , we haven't used the term > >> > Feather > >> > >> that I know of. > >> > >> > >> > >> On Fri, Jul 22, 2022 at 12:41 PM David Li <lidav...@apache.org> > >> wrote: > >> > >> > >> > >> > FeatherV2 is the same thing as the IPC file format ( > >> > >> > https://arrow.apache.org/faq/). This confusion keeps coming up, > so: > >> > how > >> > >> > do we want to brand Arrow IPC? As "Feather", or as "Arrow IPC > >> Files"? > >> > It > >> > >> > seems the latter, but even right now Arrow Dataset still writes > >> files > >> > >> with > >> > >> > .feather extension (though there are PRs to update it). I don't > >> think > >> > >> I've > >> > >> > seen a formal discussion/decision on whether to keep using > Feather > >> or > >> > >> IPC > >> > >> > as the name (though I likely missed it). > >> > >> > > >> > >> > On Fri, Jul 22, 2022, at 12:21, Andrew Lamb wrote: > >> > >> > > I am not familiar with feather -- there are some Draft PRs to > add > >> > >> support > >> > >> > > for reading [1] and writing [2] the IPC file format which may > be > >> > >> related > >> > >> > to > >> > >> > > your question > >> > >> > > > >> > >> > > [1] https://github.com/apache/arrow-datafusion/pull/1858 > >> > >> > > [2] https://github.com/apache/arrow-datafusion/pull/1893 > >> > >> > > > >> > >> > > > >> > >> > > On Fri, Jul 22, 2022 at 12:10 PM Ian Joiner < > >> iajoiner...@gmail.com> > >> > >> > wrote: > >> > >> > > > >> > >> > >> Hi, > >> > >> > >> > >> > >> > >> I'd like to ask about the level of Feather support in > Datafusion. > >> > >> Can we > >> > >> > >> read and write Feather files? I don't really find any > information > >> > >> while > >> > >> > >> searching through the repo or checking the docs. > >> > >> > >> > >> > >> > >> Thanks, > >> > >> > >> Ian > >> > >> > >> > >> > >> > > >> > >> > >> > > > >> > > >> >
