Hi Ryan, Thanks for the feedback.
Just a quick clarification—DESCRIBE TABLE in Flink does *not* show table properties; I believe you might be thinking of SHOW CREATE TABLE, which does include properties as part of the DDL output. The goal of SHOW CREATE TABLE is to provide a fully reproducible statement, so if secrets were hidden or omitted, that reproducibility would be lost. By referencing a named CONNECTION, we avoid this issue—users can reconstruct the table DDL without exposing secrets directly, since the sensitive information lives in the reusable connection object. Regarding the naming: CONNECTION is not part of the SQL standard, but neither are alternatives like SECRET or INTEGRATION. We opted for CONNECTION because it conveys intent clearly and aligns with common terminology in data platforms, even if it's not a standard keyword. That said, we're definitely open to refining the naming based on broader community input. Appreciate you raising these points and helping move the conversation forward. Best, Mayank On Fri, May 16, 2025 at 7:15 PM Ryan van Huuksloot <ryan.vanhuuksl...@shopify.com.invalid> wrote: > Hi Mayank, > > Overall I think the idea of having shared properties (secrets or otherwise) > at the database layer is really interesting/useful. Even outside of the ML > scope. > > However, I have a concern: > When I describe a table, I would not like it if all properties did not > appear in the returned results of the describe. It adds a hidden layer of > properties. > CONNECTION is not a standard SQL syntax (at least that I am aware of, > please correct me if I am wrong) and it adds a FlinkSQL specific layer of > complexity. > > I like the idea of Grants to hide secrets rather than nesting them in a > Connection object. The Connection object could still be described. > Something akin to Snowflake's Secrets ( > https://docs.snowflake.com/en/sql-reference/sql/show-secrets) that can be > referenced within table properties and could be obfuscated during a > describe. > > Then if we hide secrets, we can figure out a different name than Connection > for common properties and they can be shown as part of the table spec. > > Thoughts? > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Fri, May 16, 2025 at 1:35 PM Yash Anand <yan...@confluent.io.invalid> > wrote: > > > Hi Mayank, > > > > Thanks for the initiative! I think this functionality would be a really > > great addition to Flink. > > > > +1 for the proposal. > > > > Thanks, > > Yash > > > > On Tue, May 6, 2025 at 9:36 AM Mayank Juneja <mayankjunej...@gmail.com> > > wrote: > > > > > Hi Ferenc, > > > > > > Thanks for your question! We agree that a CONNECTION could logically > be a > > > catalog-level resource, especially since it’s intended to be reused > > across > > > multiple tables. However, we think there’s value in defining it at the > > > database level to introduce a layer of isolation and scoping. > > > > > > This is particularly useful in shared or multi-tenant environments > where > > > different databases within the same catalog might need to connect to > > > different external systems or use distinct credentials. > > > > > > That said, we’re open to revisiting this if there's strong consensus > > around > > > catalog-level scoping. Appreciate the feedback! > > > > > > Regards, > > > Mayank > > > > > > On Fri, May 2, 2025 at 12:12 AM Ferenc Csaky > <ferenc.cs...@pm.me.invalid > > > > > > wrote: > > > > > > > Hi Mayank, > > > > > > > > Thank you for starting the discussion! In general, I think such > > > > functionality > > > > would be a really great addition to Flink. > > > > > > > > Could you pls. elaborate a bit more one what is the reason of > defining > > a > > > > `connection` resource on the database level instead of the catalog > > level? > > > > If I think about `JdbcCatalog`, or `HiveCatalog`, the catalog is in > > > 1-to-1 > > > > mapping with an RDBMS, or a HiveMetastore, so my initial thinking is > > > that a > > > > `connection` seems more like a catalog level resource. > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Ferenc > > > > > > > > > > > > > > > > On Tuesday, April 29th, 2025 at 17:08, Mayank Juneja < > > > > mayankjunej...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > I would like to open up for discussion a new FLIP-529 [1]. > > > > > > > > > > Motivation: > > > > > Currently, Flink SQL handles external connectivity by defining > > > endpoints > > > > > and credentials in table configuration. This approach prevents > > > > reusability > > > > > of these connections and makes table definition less secure by > > exposing > > > > > sensitive information. > > > > > We propose the introduction of a new "connection" resource in > Flink. > > > This > > > > > will be a pluggable resource configured with a remote endpoint and > > > > > associated access key. Once defined, connections can be reused > across > > > > table > > > > > definitions, and eventually for model definition (as discussed in > > > > FLIP-437) > > > > > for inference, enabling seamless and secure integration with > external > > > > > systems. > > > > > The connection resource will provide a new, optional way to manage > > > > external > > > > > connectivity in Flink. Existing methods for table definitions will > > > remain > > > > > unchanged. > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/cYroF > > > > > > > > > > Best Regards, > > > > > Mayank Juneja > > > > > > > > > > > > > -- > > > *Mayank Juneja* > > > Product Manager | Data Streaming and AI > > > > > > -- *Mayank Juneja* Product Manager | Data Streaming and AI
