Hi Gyula,
Sergey and I spent a significant amount of time in researching different
formats.
When we introduced CompiledPlan, the question whether we want to use a
binary format for performance and efficiency reasons immediately came
up, but we decided to postpone this discussion.
Back then we thought about BSON, but looking at the most popular formats
that support all JSON types natively and offer a lossless conversion
between JSON and the binary format we ended up with Smile. Also given
that we have a large Jackson-based code base that is able to
serialize/deserialize all RexNode, StreamExecNode, DataTypes etc. Smile
seems to be the best fit.
For clarification: We won't change the default serialization. All
methods writeToFile/readFromFile() and APIs (e.g. EXECUTE COMPILED PLAN)
still operate primarily on JSON. The binary format is mostly intended
for advanced use cases. The given Flink API can be used to convert to
JSON at any time.
Regards,
Timo
On 07.02.25 16:14, Gyula Fóra wrote:
Hey!
Do we have some examples of other frameworks/projects etc using the Smile
format?
This seems to be a somewhat arbitrary change with regard to the selected
format, my concern is that this will make the compiled plan less useful in
general as it's harder to parse with standard tools.
What is the main problem with the current json format?
Thanks
Gyula
On Fri, Feb 7, 2025 at 3:31 PM Sergey Nuyanzin <snuyan...@gmail.com> wrote:
Hi everyone,
I would like to initiate a discussion for the FLIP-508[1] below, which adds
support for Smile[2] format for Compiled plans
Looking forward to hearing from you.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-508%3A+Add+support+for+Smile+format+for+Compiled+plans
[2] https://github.com/FasterXML/smile-format-specification
--
Best regards,
Sergey
