Thanks Timo for the extra info on the context of this change. +1
Gyula On Fri, Feb 7, 2025 at 5:05 PM Timo Walther <twal...@apache.org> wrote: > Hi Gyula, > > Sergey and I spent a significant amount of time in researching different > formats. > > When we introduced CompiledPlan, the question whether we want to use a > binary format for performance and efficiency reasons immediately came > up, but we decided to postpone this discussion. > > Back then we thought about BSON, but looking at the most popular formats > that support all JSON types natively and offer a lossless conversion > between JSON and the binary format we ended up with Smile. Also given > that we have a large Jackson-based code base that is able to > serialize/deserialize all RexNode, StreamExecNode, DataTypes etc. Smile > seems to be the best fit. > > For clarification: We won't change the default serialization. All > methods writeToFile/readFromFile() and APIs (e.g. EXECUTE COMPILED PLAN) > still operate primarily on JSON. The binary format is mostly intended > for advanced use cases. The given Flink API can be used to convert to > JSON at any time. > > Regards, > Timo > > > On 07.02.25 16:14, Gyula Fóra wrote: > > Hey! > > Do we have some examples of other frameworks/projects etc using the Smile > > format? > > > > This seems to be a somewhat arbitrary change with regard to the selected > > format, my concern is that this will make the compiled plan less useful > in > > general as it's harder to parse with standard tools. > > > > What is the main problem with the current json format? > > > > Thanks > > Gyula > > > > On Fri, Feb 7, 2025 at 3:31 PM Sergey Nuyanzin <snuyan...@gmail.com> > wrote: > > > >> Hi everyone, > >> > >> I would like to initiate a discussion for the FLIP-508[1] below, which > adds > >> support for Smile[2] format for Compiled plans > >> > >> Looking forward to hearing from you. > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-508%3A+Add+support+for+Smile+format+for+Compiled+plans > >> [2] https://github.com/FasterXML/smile-format-specification > >> > >> -- > >> Best regards, > >> Sergey > >> > > > >
