Hi Mayank, Thanks for the response. I have a few more questions.
1. Why not use a catalog that has default properties? That way you could re-use properties if the object exists in that catalog/database. --- If we are insistent on Connections. 1. I don't understand the distinction between a "CONNECTION" and the rest of the properties that may exist on a table. In your flip you have some properties on the Connection object and some on the table object. How would someone decide which object the property should exist under? 2. We're referencing a SQL object through a properties string. In SQL, you usually interact with SQL objects with SQL syntax, not a string property. Why not support something like this? Connectors already enforce what properties are required. This provides broader utility while solving for your use case - from my understanding. *Note: I much prefer catalogs for this case. Which is what we use internally to manage connection properties.* CREATE OR REPLACE CONNECTION mysql_customer_db WITH ( 'connector' = 'jdbc', 'jdbc.url' = 'jdbc:mysql://customer-db.example.com:3306/customerdb', 'jdbc.auth.type' = 'basic', 'jdbc.username' = 'readonly_user', 'jdbc.password' = 'secret123', 'jdbc.connection.ssl.enabled' = 'true' ); CREATE TABLE customers ( customer_id INT, name STRING, email STRING, country STRING, vip_status BOOLEAN, PRIMARY KEY (customer_id) NOT ENFORCED ) WITH CONNECTION mysql_customer_db OVERRIDE ( 'jdbc.table-name' = 'customers', 'jdbc.lookup.cache' = 'PARTIAL', 'jdbc.lookup.partial-cache.max-rows' = '10000', 'jdbc.lookup.partial-cache.expire-after-write' = '1h', 'jdbc.lookup.partial-cache.expire-after-access' = '10m' ); >From my perspective, this FLIP has a SQL smell. I would be interested to hear what folks like Timo think. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Wed, May 28, 2025 at 10:23 AM Mayank Juneja <mayankjunej...@gmail.com> wrote: > Thanks all for the feedback on the FLIP. If there are no more discussions, > I will start voting on this tomorrow. > > On Wed, May 21, 2025 at 6:00 AM Mayank Juneja <mayankjunej...@gmail.com> > wrote: > > > Hi Ryan, > > > > Thanks for the feedback. > > > > Just a quick clarification—DESCRIBE TABLE in Flink does *not* show table > > properties; I believe you might be thinking of SHOW CREATE TABLE, which > > does include properties as part of the DDL output. The goal of SHOW > > CREATE TABLE is to provide a fully reproducible statement, so if secrets > > were hidden or omitted, that reproducibility would be lost. By > referencing > > a named CONNECTION, we avoid this issue—users can reconstruct the table > > DDL without exposing secrets directly, since the sensitive information > > lives in the reusable connection object. > > > > Regarding the naming: CONNECTION is not part of the SQL standard, but > > neither are alternatives like SECRET or INTEGRATION. We opted for > > CONNECTION because it conveys intent clearly and aligns with common > > terminology in data platforms, even if it's not a standard keyword. > > > > That said, we're definitely open to refining the naming based on broader > > community input. Appreciate you raising these points and helping move the > > conversation forward. > > > > Best, > > Mayank > > > > On Fri, May 16, 2025 at 7:15 PM Ryan van Huuksloot > > <ryan.vanhuuksl...@shopify.com.invalid> wrote: > > > >> Hi Mayank, > >> > >> Overall I think the idea of having shared properties (secrets or > >> otherwise) > >> at the database layer is really interesting/useful. Even outside of the > ML > >> scope. > >> > >> However, I have a concern: > >> When I describe a table, I would not like it if all properties did not > >> appear in the returned results of the describe. It adds a hidden layer > of > >> properties. > >> CONNECTION is not a standard SQL syntax (at least that I am aware of, > >> please correct me if I am wrong) and it adds a FlinkSQL specific layer > of > >> complexity. > >> > >> I like the idea of Grants to hide secrets rather than nesting them in a > >> Connection object. The Connection object could still be described. > >> Something akin to Snowflake's Secrets ( > >> https://docs.snowflake.com/en/sql-reference/sql/show-secrets) that can > be > >> referenced within table properties and could be obfuscated during a > >> describe. > >> > >> Then if we hide secrets, we can figure out a different name than > >> Connection > >> for common properties and they can be shown as part of the table spec. > >> > >> Thoughts? > >> > >> Ryan van Huuksloot > >> Sr. Production Engineer | Streaming Platform > >> [image: Shopify] > >> < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > >> > >> > >> On Fri, May 16, 2025 at 1:35 PM Yash Anand <yan...@confluent.io.invalid > > > >> wrote: > >> > >> > Hi Mayank, > >> > > >> > Thanks for the initiative! I think this functionality would be a > really > >> > great addition to Flink. > >> > > >> > +1 for the proposal. > >> > > >> > Thanks, > >> > Yash > >> > > >> > On Tue, May 6, 2025 at 9:36 AM Mayank Juneja < > mayankjunej...@gmail.com> > >> > wrote: > >> > > >> > > Hi Ferenc, > >> > > > >> > > Thanks for your question! We agree that a CONNECTION could logically > >> be a > >> > > catalog-level resource, especially since it’s intended to be reused > >> > across > >> > > multiple tables. However, we think there’s value in defining it at > the > >> > > database level to introduce a layer of isolation and scoping. > >> > > > >> > > This is particularly useful in shared or multi-tenant environments > >> where > >> > > different databases within the same catalog might need to connect to > >> > > different external systems or use distinct credentials. > >> > > > >> > > That said, we’re open to revisiting this if there's strong consensus > >> > around > >> > > catalog-level scoping. Appreciate the feedback! > >> > > > >> > > Regards, > >> > > Mayank > >> > > > >> > > On Fri, May 2, 2025 at 12:12 AM Ferenc Csaky > >> <ferenc.cs...@pm.me.invalid > >> > > > >> > > wrote: > >> > > > >> > > > Hi Mayank, > >> > > > > >> > > > Thank you for starting the discussion! In general, I think such > >> > > > functionality > >> > > > would be a really great addition to Flink. > >> > > > > >> > > > Could you pls. elaborate a bit more one what is the reason of > >> defining > >> > a > >> > > > `connection` resource on the database level instead of the catalog > >> > level? > >> > > > If I think about `JdbcCatalog`, or `HiveCatalog`, the catalog is > in > >> > > 1-to-1 > >> > > > mapping with an RDBMS, or a HiveMetastore, so my initial thinking > is > >> > > that a > >> > > > `connection` seems more like a catalog level resource. > >> > > > > >> > > > WDYT? > >> > > > > >> > > > Thanks, > >> > > > Ferenc > >> > > > > >> > > > > >> > > > > >> > > > On Tuesday, April 29th, 2025 at 17:08, Mayank Juneja < > >> > > > mayankjunej...@gmail.com> wrote: > >> > > > > >> > > > > > >> > > > > > >> > > > > Hi all, > >> > > > > > >> > > > > I would like to open up for discussion a new FLIP-529 [1]. > >> > > > > > >> > > > > Motivation: > >> > > > > Currently, Flink SQL handles external connectivity by defining > >> > > endpoints > >> > > > > and credentials in table configuration. This approach prevents > >> > > > reusability > >> > > > > of these connections and makes table definition less secure by > >> > exposing > >> > > > > sensitive information. > >> > > > > We propose the introduction of a new "connection" resource in > >> Flink. > >> > > This > >> > > > > will be a pluggable resource configured with a remote endpoint > and > >> > > > > associated access key. Once defined, connections can be reused > >> across > >> > > > table > >> > > > > definitions, and eventually for model definition (as discussed > in > >> > > > FLIP-437) > >> > > > > for inference, enabling seamless and secure integration with > >> external > >> > > > > systems. > >> > > > > The connection resource will provide a new, optional way to > manage > >> > > > external > >> > > > > connectivity in Flink. Existing methods for table definitions > will > >> > > remain > >> > > > > unchanged. > >> > > > > > >> > > > > [1] https://cwiki.apache.org/confluence/x/cYroF > >> > > > > > >> > > > > Best Regards, > >> > > > > Mayank Juneja > >> > > > > >> > > > >> > > > >> > > -- > >> > > *Mayank Juneja* > >> > > Product Manager | Data Streaming and AI > >> > > > >> > > >> > > > > > > -- > > *Mayank Juneja* > > Product Manager | Data Streaming and AI > > > > > -- > *Mayank Juneja* > Product Manager | Data Streaming and AI >
