Hi, https://issues.apache.org/jira/browse/ARROW-17092 may be related.
Thanks, -- kou In <CAC-Ect-+c-62BQwJ432UTab+n0Mr2k36Hco=6dawwt5ck7p...@mail.gmail.com> "Re: [Rust] IPC Format / Feather support in Datafusion" on Mon, 25 Jul 2022 13:39:54 -0700, Aldrin <akmon...@ucsc.edu.INVALID> wrote: > It seems unfortunate to me that the feather file format doc page [1] > appears to have been forgotten when those IPC file format docs were written > [2][3]. I will find or make a JIRA to make this info consistent in the docs. > > [1]: https://arrow.apache.org/docs/python/feather.html > [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format > [3]: https://arrow.apache.org/docs/cpp/ipc.html > > Aldrin Montana > Computer Science PhD Student > UC Santa Cruz > > > On Fri, Jul 22, 2022 at 2:46 PM Will Jones <will.jones...@gmail.com> wrote: > >> FYI It looks like there is active work to change the Python [1] and R [2] >> implementations (built on C++) to write .arrow instead of .feather. >> >> [1] https://issues.apache.org/jira/browse/ARROW-17089 >> [2] https://issues.apache.org/jira/browse/ARROW-17088 >> >> On Fri, Jul 22, 2022 at 11:16 AM Aldrin <akmon...@ucsc.edu.invalid> wrote: >> >> > sorry, I meant "...especially *for* the rust community if they are just >> > using IPC directly for file formats." >> > >> > Aldrin Montana >> > Computer Science PhD Student >> > UC Santa Cruz >> > >> > >> > On Fri, Jul 22, 2022 at 11:14 AM Aldrin <akmon...@ucsc.edu> wrote: >> > >> > > I always assumed IPC was when it was in memory, feather was when it was >> > > persisted as files. That distinction/indirection may be totally moot, >> > > though, especially if the rust community is just using IPC directly for >> > > file formats. >> > > >> > > In my mind, feather V1 came just before IPC and started using IPC since >> > > V2. I'm not sure if a feather V3 would ever diverge from IPC format or >> if >> > > feather adds anything that's more filesystem friendly (versus other >> > storage >> > > system interfaces) or makes filesystem performance more predictable. >> > > >> > > >> > > Aldrin Montana >> > > Computer Science PhD Student >> > > UC Santa Cruz >> > > >> > > >> > > On Fri, Jul 22, 2022 at 10:49 AM Andrew Lamb <al...@influxdata.com> >> > wrote: >> > > >> > >> For what it is worth, in the Rust Arrow community, I have only ever >> > heard >> > >> the format referred to as "IPC Format" , we haven't used the term >> > Feather >> > >> that I know of. >> > >> >> > >> On Fri, Jul 22, 2022 at 12:41 PM David Li <lidav...@apache.org> >> wrote: >> > >> >> > >> > FeatherV2 is the same thing as the IPC file format ( >> > >> > https://arrow.apache.org/faq/). This confusion keeps coming up, so: >> > how >> > >> > do we want to brand Arrow IPC? As "Feather", or as "Arrow IPC >> Files"? >> > It >> > >> > seems the latter, but even right now Arrow Dataset still writes >> files >> > >> with >> > >> > .feather extension (though there are PRs to update it). I don't >> think >> > >> I've >> > >> > seen a formal discussion/decision on whether to keep using Feather >> or >> > >> IPC >> > >> > as the name (though I likely missed it). >> > >> > >> > >> > On Fri, Jul 22, 2022, at 12:21, Andrew Lamb wrote: >> > >> > > I am not familiar with feather -- there are some Draft PRs to add >> > >> support >> > >> > > for reading [1] and writing [2] the IPC file format which may be >> > >> related >> > >> > to >> > >> > > your question >> > >> > > >> > >> > > [1] https://github.com/apache/arrow-datafusion/pull/1858 >> > >> > > [2] https://github.com/apache/arrow-datafusion/pull/1893 >> > >> > > >> > >> > > >> > >> > > On Fri, Jul 22, 2022 at 12:10 PM Ian Joiner < >> iajoiner...@gmail.com> >> > >> > wrote: >> > >> > > >> > >> > >> Hi, >> > >> > >> >> > >> > >> I'd like to ask about the level of Feather support in Datafusion. >> > >> Can we >> > >> > >> read and write Feather files? I don't really find any information >> > >> while >> > >> > >> searching through the repo or checking the docs. >> > >> > >> >> > >> > >> Thanks, >> > >> > >> Ian >> > >> > >> >> > >> > >> > >> >> > > >> > >>
