Hi all, First of all, I think having a Connection resource is something that will be beneficial for Apache Flink. I could see that being extended in the future to allow for easier secret handling [1]. In my mental mind, I'm comparing this proposal against SQL/MED from the ISO standard [2]. I do think that SQL/MED isn't a very user friendly syntax though, looking at Postgres for example [3].
I think it's a valid question if Connection should be considered with a catalog or database-level scope. @Ryan can you share something more, since you've mentioned "Note: I much prefer catalogs for this case. Which is what we use internally to manage connection properties". It looks like there isn't a strong favourable approach looking at other vendors (like, Databricks does scopes it on a Unity catalog, Snowflake on a database level). Also looking forward to Leonard's input. Best regards, Martijn [1] https://issues.apache.org/jira/browse/FLINK-36818 [2] https://www.iso.org/standard/84804.html [3] https://www.postgresql.org/docs/current/sql-createserver.html On Fri, May 30, 2025 at 5:07 AM Leonard Xu <xbjt...@gmail.com> wrote: > Hey Mayank. > > Thanks for the FLIP, I went through this FLIP quickly and found some > issues which I think we > need to deep discuss later. As we’re on a short Dragon boat Festival, > could you kindly hold > on this thread? and we will back to continue the FLIP discuss. > > Best, > Leonard > > > > 2025 4月 29 23:07，Mayank Juneja <mayankjunej...@gmail.com> 写道： > > > > Hi all, > > > > I would like to open up for discussion a new FLIP-529 [1]. > > > > Motivation: > > Currently, Flink SQL handles external connectivity by defining endpoints > > and credentials in table configuration. This approach prevents > reusability > > of these connections and makes table definition less secure by exposing > > sensitive information. > > We propose the introduction of a new "connection" resource in Flink. This > > will be a pluggable resource configured with a remote endpoint and > > associated access key. Once defined, connections can be reused across > table > > definitions, and eventually for model definition (as discussed in > FLIP-437) > > for inference, enabling seamless and secure integration with external > > systems. > > The connection resource will provide a new, optional way to manage > external > > connectivity in Flink. Existing methods for table definitions will remain > > unchanged. > > > > [1] https://cwiki.apache.org/confluence/x/cYroF > > > > Best Regards, > > Mayank Juneja > >
