Hi all,

I've started a discussion on dev@parquet about a new logical type,
SERIALIZED_VALUE [1][2], and I'd like
the Spark community's feedback on whether Spark would support it.

SERIALIZED_VALUE would annotate a BYTE_ARRAY column of serialized protobuf,
FlatBuffers, or thrift
messages. It would record the format of the bytes and store the schema in
the file, along with the name of
the root message type. For protobuf the schema would be a
FileDescriptorSet. Parquet would never decode
the bytes, and a reader that does not know the type would read the column
as plain BYTE_ARRAY.

Spark already decodes protobuf with *from_protobuf*, which takes a message
name and a binary
FileDescriptorSet, but the user has to supply both on every call. With both
stored in the file, Spark
could read them from there.

Parquet would define the annotation and would store the bytes and the
schema without parsing them, so it
would take on no dependency on any serialization format. Spark would own
the decoding and how fields are
exposed to queries. A first step in Spark would be something small, like
mapping the annotation to
BinaryType and carrying the format and schema through on write. If there is
agreement on the direction,
I'd like to work with the community on an SPIP for reading message fields
directly and for projected
columns, which Parquet would take up once an engine wants them.

Feedback on the direction is very welcome.

Thanks,
Neelesh

[1] *Proposal:*
https://docs.google.com/document/d/1RWH8P2PfMWMXvJuwtGXLe1HQbDjmL_gqR6wmhC7x3Ac/edit?usp=sharing
[2] *Parquet discussion thread:*
https://lists.apache.org/thread.html/5xn8o9rvrv1rsjs811t2gn0tkjq0zl02

Reply via email to