Thank you both. I hadn't read the IPC documentation closely enough to
understand that it supported metadata at the message level. It seems like
the best approach in my case is then probably to flush the dataset to
separate files as a large number of IPC message batches, and send the
schema and the
I can't speak to whether Pandas conversion will ever change. Some one else
can potentially chime in I don't recall any JIRAs recently changing this
type of conversion, however currently for library functionality there
aren't any hard guarantees for backwards compatibility (generally we try to
do o
>
> But I'm also attaching table
> metadata to each feather, which I'd hate to lose.
>
Note the arrow format allows attaching custom metadata at the column
(field), schema, and message level, so it should be possible to retain any
metadata this way.
-Jacob
On Wed, Oct 7, 2020 at 11:38 AM Benjami
Hello,
Exciting project, thanks for all your work. I gather it's appropriate to
ask a use question here? Assuming so:
I have a web application that serves portions of a dataset I've broken into
a few thousand featherV2 files structured as a quadtree. The structure
makes heavy use of text dictiona
