Hi Ferenc, Making SQL Gateway to be an easy-to-use platform infrastructure of Flink SQL is one of the important roadmaps [1].
The persistence ability of the SQL Gateway is a major work in 1.18 release. One of the persistence demand is that the registered catalogs are currently kept in memory and lost when Gateway restarts. There is an accepted FLIP (FLIP-295)[2] target to resolve this issue and make Gateway can persist the registered catalogs information into files or databases. I'm not sure whether this is something you are looking for? Best, Jark [1]: https://flink.apache.org/roadmap/#a-unified-sql-platform [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations On Fri, 23 Jun 2023 at 00:25, Ferenc Csaky <ferenc.cs...@pm.me.invalid> wrote: > Hello devs, > > I would like to open a discussion about persistence possibilitis for the > SQL Gateway. At Cloudera, we are happy to see the work already done on this > project and looking for ways to utilize it on our platform as well, but > currently it lacks some features that would be essential in our case, where > we could help out. > > I am not sure if any thought went into gateway persistence specifics > already, and this feature could be implemented in fundamentally differnt > ways, so I think the frist step could be to agree on the basics. > > First, in my opinion, persistence should be an optional feature of the > gateway, that can be enabled if desired. There can be a lot of > implementation details, but there can be some major directions to follow: > > - Utilize Hive catalog: The Hive catalog can already be used to have > persistenct meta-objects, so the crucial thing that would be missing in > this case is other catalogs. Personally, I would not pursue this option, > because in my opinion it would limit the usability of this feature too much. > - Serialize the session as is: Saving the whole session (or its context) > [1] as is to durable storage, so it can be kept and picked up again. > - Serialize the required elements (catalogs, tables, functions, etc.), not > necessarily as a whole: The main point here would be to serialize a > different object, so the persistent data will not be that sensitive to > changes of the session (or its context). There can be numerous factors > here, like try to keep the model close to the session itself, so the > boilerplate required for the mapping can be kept to minimal, or focus on > saving what is actually necessary, making the persistent storage more > portable. > > WDYT? > > Cheers, > F > > [1] > https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/session/Session.java
