Hi All,
I was looking at the Iceberg metadata layer of Manifest Lists and
Manifest Files and they are all JSON formats. Thinking out loud, what if
that layer was changed to Arrow file format?
Since Arrow's in-memory representation is the same as the on-disk
representation. A lot of overhead of serde, reading and writing could be
saved. Also because Arrow has RecordBatches we could parallelize the read
of that to multiple threads. This is just thinking out loud. Please let me
know what your thoughts are.
Regards,
Taher Koitawala
