It seems unfortunate to me that the feather file format doc page [1] appears to have been forgotten when those IPC file format docs were written [2][3]. I will find or make a JIRA to make this info consistent in the docs.
[1]: https://arrow.apache.org/docs/python/feather.html [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format [3]: https://arrow.apache.org/docs/cpp/ipc.html Aldrin Montana Computer Science PhD Student UC Santa Cruz On Fri, Jul 22, 2022 at 2:46 PM Will Jones <will.jones...@gmail.com> wrote: > FYI It looks like there is active work to change the Python [1] and R [2] > implementations (built on C++) to write .arrow instead of .feather. > > [1] https://issues.apache.org/jira/browse/ARROW-17089 > [2] https://issues.apache.org/jira/browse/ARROW-17088 > > On Fri, Jul 22, 2022 at 11:16 AM Aldrin <akmon...@ucsc.edu.invalid> wrote: > > > sorry, I meant "...especially *for* the rust community if they are just > > using IPC directly for file formats." > > > > Aldrin Montana > > Computer Science PhD Student > > UC Santa Cruz > > > > > > On Fri, Jul 22, 2022 at 11:14 AM Aldrin <akmon...@ucsc.edu> wrote: > > > > > I always assumed IPC was when it was in memory, feather was when it was > > > persisted as files. That distinction/indirection may be totally moot, > > > though, especially if the rust community is just using IPC directly for > > > file formats. > > > > > > In my mind, feather V1 came just before IPC and started using IPC since > > > V2. I'm not sure if a feather V3 would ever diverge from IPC format or > if > > > feather adds anything that's more filesystem friendly (versus other > > storage > > > system interfaces) or makes filesystem performance more predictable. > > > > > > > > > Aldrin Montana > > > Computer Science PhD Student > > > UC Santa Cruz > > > > > > > > > On Fri, Jul 22, 2022 at 10:49 AM Andrew Lamb <al...@influxdata.com> > > wrote: > > > > > >> For what it is worth, in the Rust Arrow community, I have only ever > > heard > > >> the format referred to as "IPC Format" , we haven't used the term > > Feather > > >> that I know of. > > >> > > >> On Fri, Jul 22, 2022 at 12:41 PM David Li <lidav...@apache.org> > wrote: > > >> > > >> > FeatherV2 is the same thing as the IPC file format ( > > >> > https://arrow.apache.org/faq/). This confusion keeps coming up, so: > > how > > >> > do we want to brand Arrow IPC? As "Feather", or as "Arrow IPC > Files"? > > It > > >> > seems the latter, but even right now Arrow Dataset still writes > files > > >> with > > >> > .feather extension (though there are PRs to update it). I don't > think > > >> I've > > >> > seen a formal discussion/decision on whether to keep using Feather > or > > >> IPC > > >> > as the name (though I likely missed it). > > >> > > > >> > On Fri, Jul 22, 2022, at 12:21, Andrew Lamb wrote: > > >> > > I am not familiar with feather -- there are some Draft PRs to add > > >> support > > >> > > for reading [1] and writing [2] the IPC file format which may be > > >> related > > >> > to > > >> > > your question > > >> > > > > >> > > [1] https://github.com/apache/arrow-datafusion/pull/1858 > > >> > > [2] https://github.com/apache/arrow-datafusion/pull/1893 > > >> > > > > >> > > > > >> > > On Fri, Jul 22, 2022 at 12:10 PM Ian Joiner < > iajoiner...@gmail.com> > > >> > wrote: > > >> > > > > >> > >> Hi, > > >> > >> > > >> > >> I'd like to ask about the level of Feather support in Datafusion. > > >> Can we > > >> > >> read and write Feather files? I don't really find any information > > >> while > > >> > >> searching through the repo or checking the docs. > > >> > >> > > >> > >> Thanks, > > >> > >> Ian > > >> > >> > > >> > > > >> > > > > > >
