Oh interesting, so we could use this to annotate pretty much arbitrary data?
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Thu, Nov 12, 2026 at 1:21 PM Neelesh Salian <[email protected]> wrote: > Hi all, > > I've started a discussion on dev@parquet about a new logical type, > SERIALIZED_VALUE [1][2], and I'd like > the Spark community's feedback on whether Spark would support it. > > SERIALIZED_VALUE would annotate a BYTE_ARRAY column of serialized > protobuf, FlatBuffers, or thrift > messages. It would record the format of the bytes and store the schema in > the file, along with the name of > the root message type. For protobuf the schema would be a > FileDescriptorSet. Parquet would never decode > the bytes, and a reader that does not know the type would read the column > as plain BYTE_ARRAY. > > Spark already decodes protobuf with *from_protobuf*, which takes a > message name and a binary > FileDescriptorSet, but the user has to supply both on every call. With > both stored in the file, Spark > could read them from there. > > Parquet would define the annotation and would store the bytes and the > schema without parsing them, so it > would take on no dependency on any serialization format. Spark would own > the decoding and how fields are > exposed to queries. A first step in Spark would be something small, like > mapping the annotation to > BinaryType and carrying the format and schema through on write. If there > is agreement on the direction, > I'd like to work with the community on an SPIP for reading message fields > directly and for projected > columns, which Parquet would take up once an engine wants them. > > Feedback on the direction is very welcome. > > Thanks, > Neelesh > > [1] *Proposal:* > https://docs.google.com/document/d/1RWH8P2PfMWMXvJuwtGXLe1HQbDjmL_gqR6wmhC7x3Ac/edit?usp=sharing > [2] *Parquet discussion thread:* > https://lists.apache.org/thread.html/5xn8o9rvrv1rsjs811t2gn0tkjq0zl02 >
