Hi Mayank, Overall I think the idea of having shared properties (secrets or otherwise) at the database layer is really interesting/useful. Even outside of the ML scope.
However, I have a concern: When I describe a table, I would not like it if all properties did not appear in the returned results of the describe. It adds a hidden layer of properties. CONNECTION is not a standard SQL syntax (at least that I am aware of, please correct me if I am wrong) and it adds a FlinkSQL specific layer of complexity. I like the idea of Grants to hide secrets rather than nesting them in a Connection object. The Connection object could still be described. Something akin to Snowflake's Secrets ( https://docs.snowflake.com/en/sql-reference/sql/show-secrets) that can be referenced within table properties and could be obfuscated during a describe. Then if we hide secrets, we can figure out a different name than Connection for common properties and they can be shown as part of the table spec. Thoughts? Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, May 16, 2025 at 1:35 PM Yash Anand <yan...@confluent.io.invalid> wrote: > Hi Mayank, > > Thanks for the initiative! I think this functionality would be a really > great addition to Flink. > > +1 for the proposal. > > Thanks, > Yash > > On Tue, May 6, 2025 at 9:36 AM Mayank Juneja <mayankjunej...@gmail.com> > wrote: > > > Hi Ferenc, > > > > Thanks for your question! We agree that a CONNECTION could logically be a > > catalog-level resource, especially since it’s intended to be reused > across > > multiple tables. However, we think there’s value in defining it at the > > database level to introduce a layer of isolation and scoping. > > > > This is particularly useful in shared or multi-tenant environments where > > different databases within the same catalog might need to connect to > > different external systems or use distinct credentials. > > > > That said, we’re open to revisiting this if there's strong consensus > around > > catalog-level scoping. Appreciate the feedback! > > > > Regards, > > Mayank > > > > On Fri, May 2, 2025 at 12:12 AM Ferenc Csaky <ferenc.cs...@pm.me.invalid > > > > wrote: > > > > > Hi Mayank, > > > > > > Thank you for starting the discussion! In general, I think such > > > functionality > > > would be a really great addition to Flink. > > > > > > Could you pls. elaborate a bit more one what is the reason of defining > a > > > `connection` resource on the database level instead of the catalog > level? > > > If I think about `JdbcCatalog`, or `HiveCatalog`, the catalog is in > > 1-to-1 > > > mapping with an RDBMS, or a HiveMetastore, so my initial thinking is > > that a > > > `connection` seems more like a catalog level resource. > > > > > > WDYT? > > > > > > Thanks, > > > Ferenc > > > > > > > > > > > > On Tuesday, April 29th, 2025 at 17:08, Mayank Juneja < > > > mayankjunej...@gmail.com> wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > I would like to open up for discussion a new FLIP-529 [1]. > > > > > > > > Motivation: > > > > Currently, Flink SQL handles external connectivity by defining > > endpoints > > > > and credentials in table configuration. This approach prevents > > > reusability > > > > of these connections and makes table definition less secure by > exposing > > > > sensitive information. > > > > We propose the introduction of a new "connection" resource in Flink. > > This > > > > will be a pluggable resource configured with a remote endpoint and > > > > associated access key. Once defined, connections can be reused across > > > table > > > > definitions, and eventually for model definition (as discussed in > > > FLIP-437) > > > > for inference, enabling seamless and secure integration with external > > > > systems. > > > > The connection resource will provide a new, optional way to manage > > > external > > > > connectivity in Flink. Existing methods for table definitions will > > remain > > > > unchanged. > > > > > > > > [1] https://cwiki.apache.org/confluence/x/cYroF > > > > > > > > Best Regards, > > > > Mayank Juneja > > > > > > > > > -- > > *Mayank Juneja* > > Product Manager | Data Streaming and AI > > >
