Thanks for clarifying this Timo!
If there is no other feedback, I'm going to start voting in a couple of days On Fri, Feb 7, 2025 at 5:13 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > Thanks Timo for the extra info on the context of this change. > > +1 > > Gyula > > On Fri, Feb 7, 2025 at 5:05 PM Timo Walther <twal...@apache.org> wrote: > > > Hi Gyula, > > > > Sergey and I spent a significant amount of time in researching different > > formats. > > > > When we introduced CompiledPlan, the question whether we want to use a > > binary format for performance and efficiency reasons immediately came > > up, but we decided to postpone this discussion. > > > > Back then we thought about BSON, but looking at the most popular formats > > that support all JSON types natively and offer a lossless conversion > > between JSON and the binary format we ended up with Smile. Also given > > that we have a large Jackson-based code base that is able to > > serialize/deserialize all RexNode, StreamExecNode, DataTypes etc. Smile > > seems to be the best fit. > > > > For clarification: We won't change the default serialization. All > > methods writeToFile/readFromFile() and APIs (e.g. EXECUTE COMPILED PLAN) > > still operate primarily on JSON. The binary format is mostly intended > > for advanced use cases. The given Flink API can be used to convert to > > JSON at any time. > > > > Regards, > > Timo > > > > > > On 07.02.25 16:14, Gyula Fóra wrote: > > > Hey! > > > Do we have some examples of other frameworks/projects etc using the > Smile > > > format? > > > > > > This seems to be a somewhat arbitrary change with regard to the > selected > > > format, my concern is that this will make the compiled plan less useful > > in > > > general as it's harder to parse with standard tools. > > > > > > What is the main problem with the current json format? > > > > > > Thanks > > > Gyula > > > > > > On Fri, Feb 7, 2025 at 3:31 PM Sergey Nuyanzin <snuyan...@gmail.com> > > wrote: > > > > > >> Hi everyone, > > >> > > >> I would like to initiate a discussion for the FLIP-508[1] below, which > > adds > > >> support for Smile[2] format for Compiled plans > > >> > > >> Looking forward to hearing from you. > > >> > > >> [1] > > >> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-508%3A+Add+support+for+Smile+format+for+Compiled+plans > > >> [2] https://github.com/FasterXML/smile-format-specification > > >> > > >> -- > > >> Best regards, > > >> Sergey > > >> > > > > > > > > -- Best regards, Sergey
