Hm, I didn't think it was possible, but it looks like there may be some things you can try?
My understanding was that you create a writer for an IPC stream or file and you pass a schema on construction which is used as "the schema" for the IPC stream/file. So, RecordBatches written using that writer should/need to match the given schema. This doesn't check the metadata, I don't think, but it only writes an "IPC payload" if the equality check passes. That being said, I did some checking, and some things seem like it's more flexible now (but I could be wrong). I'm not sure what the dictionary deltas are (maybe it's for dictionary arrays rather than metadata), but the "emit_dictionary_deltas" IpcOption may be relevant [1]. Otherwise, the `WriteRecordBatch` function appears to take a metadata length [2] and the `WriteRecordBatchStream` function [3] seems to only check that a vector of RecordBatches have matching schemas. Also, the `WritePayload` function (from a RecordBatchWriter via MakeFileWriter) seems to be relevant for how to write metadata that can be leveraged for a seek-based interface [4]. But, ultimately, I am not sure these things are exposed at a higher level (e.g. pyarrow), even though they're available for use. They're also not exposed via the feather interface, as far as I know. [1]: https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4N5arrow3ipc15IpcWriteOptions22emit_dictionary_deltasE [2]: https://github.com/apache/arrow/blob/apache-arrow-7.0.0/cpp/src/arrow/ipc/writer.cc#L644 [3]: https://github.com/apache/arrow/blob/apache-arrow-7.0.0/cpp/src/arrow/ipc/writer.cc#L665 [4]: https://github.com/apache/arrow/blob/apache-arrow-7.0.0/cpp/src/arrow/ipc/writer.cc#L1253 Aldrin Montana Computer Science PhD Student UC Santa Cruz On Tue, Apr 5, 2022 at 1:55 AM Yue Ni <niyue....@gmail.com> wrote: > Hi there, > > I am investigating analyzing time series data using apache arrow. I would > like to store some record batch specific metadata, for example, some > statistics/tags about data in a particular record batch. More specifically, > I may use a single record batch to store metric samples for a certain time > range, and would like to store the min/max time and some dimensional data > like `host` and `aws_region` as metadata for a particular record batch so > that when loading multiple record batches from IPC file, the metadata may > vary from batch to batch in an IPC file, and I can filter these batches > quickly simply using metadata without looking into data in the arrays. And > I would like to know if it is possible to store such per record batch > metadata in an arrow IPC file. > > There is a similar effort I can find on the web [1], but it stores all the > record batches metadata in the IPC file footer's schema. I think the footer > will be fully loaded for every access, which will introduce some > unnecessary IO if only a few of the record batches are read each time. > > I read some docs/source code [2] [3], and if my understanding is correct, > it is technically possible to store different metadata in different record > batches since in the streaming format, each message has a `custom_metadata` > associated with it. But I don't find any API (at least in pyarrow) allowing > me to do this. APIs like `pyarrow.record_batch` does allow users to specify > metadata when constructing a record batch, but it doesn't seem to be used > if `RecordBatchFileWriter` has a schema provided (which of course doesn't > have such record batch specific metadata). > > I haven't looked into the lower level C++ API yet, and it seems the > assumption is that all the batches in the IPC file should share the same > schema, but do we allow them to have different metadata if the schema > (field names and their types) is the same? If we don't allow such usage > currently, do you think it is a valid use case to support this kind of > usage? Thanks. > > [1] > > https://github.com/heterodb/pg-strom/wiki/806%3A-Apache-Arrow-Min-Max-Statistics-Hint > [2] > > https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc > [3] > > https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_table.py >
