Yeah, definitely since this a forming feature.
BR,
G
On Mon, Mar 31, 2025 at 9:41 AM Zakelly Lan wrote:
> Hi Gabor,
>
> Only one minor suggestion, is it possible to make this \@Experimental? I
> could expect some new metadata columns added or changed in the near future,
> just want to maintai
Hi Gabor,
Only one minor suggestion, is it possible to make this \@Experimental? I
could expect some new metadata columns added or changed in the near future,
just want to maintain some flexibility until the feature reaches maturity.
Best,
Zakelly
On Fri, Mar 28, 2025 at 8:47 PM Gabor Somogyi
Just to avoid jumping from one thread to another here's Timo's suggestion:
"if I understand the discussion correctly, you want to use a PTF without
table arguments to return a table (read from savepoint metadata)? If
this is the case, you don't need a PTF for it. A regular table function
can also
Hi all,
Given the simplicity, I also +1 for PTF or any other function
implementation if PTF is not applicable for this.
I would like to raise a consideration regarding the usage implementation:
> Would it be necessary to allow users to utilize the CREATE FUNCTION
> statement for registering the P
One more question about the FLIP.
I think the output schema is definitely a public API to users. If users use
the `CREATE FUNCTION` statement, is it means the class path is also a
public API to users. Alternatively, this is merely an experimental feature
and we don't have any promise about this fu
+1 to use PTF.
I would like to raise a consideration regarding the usage implementation:
Would it be necessary to allow users to utilize the CREATE FUNCTION
statement for registering the PTF?
Currently, Flink SQL supports letting external systems register modules and
leverage these modules to cen
Just found out that PTF in batch mode is not supported, plz see the dev
mailing about it [1].
[1] https://lists.apache.org/thread/ytm9m1qt4pq2q2gjngfktrn8vrlvkf07
BR,
G
On Thu, Mar 27, 2025 at 3:38 PM Gabor Somogyi
wrote:
> In the meantime I've just updated the FLIP according to this to be
>
In the meantime I've just updated the FLIP according to this to be
optimistic 🙂
BR,
G
On Thu, Mar 27, 2025 at 2:15 PM Gabor Somogyi
wrote:
> Considering all the facts I also +1 on PTF. Even if something is missing
> we can add later.
>
> @Zakelly Lan @Shengkai Fang are you also on the
> same p
Considering all the facts I also +1 on PTF. Even if something is missing we
can add later.
@Zakelly Lan @Shengkai Fang are you also on the
same page or have something to add?
BR,
G
On Thu, Mar 27, 2025 at 1:50 PM Lincoln Lee wrote:
> +1 for PTF
>
> > Is it possible to describe such function
+1 for PTF
> Is it possible to describe such function to see the column names/types?
Although Flink SQL does not directly support this feature, users can
achieve
similar results with the help of `explain` syntax, e.g.
'explain select * from read_state_metadata(...)'
Best,
Lincoln Lee
Gyula Fó
Hey!
I think the PTF approach strikes a great balance in simplicity and the
capabilities that we get out of it.
I think this could be a completely viable alternative to the dedicated
connector, +1.
Cheers,
Gyula
On Thu, Mar 27, 2025 at 10:37 AM Shengkai Fang wrote:
> Hi, Gabor.
>
> > Do I und
Hi, Gabor.
> Do I understand correctly that this is 2.x only feature and we can't
backport it to 1.x line
Yes. PTF is only supported in 2.x verison.
> Is it possible to describe such function to see the column names/types?
Flink SQL doesn't support this feature, but postgres[2] or mysql[1] has
Hi Shengkai,
Thanks for your effort with the example, this looks promising.
I like the fact that users wouldn't need to sweat with complex create table
statements.
Couple of questions:
* Do I understand correctly that this is 2.x only feature and we can't
backport it to 1.x line?
I'm not intended
Many thanks for your reminder, Leonard. Here's the link I mentioned[1].
Best,
Shengkai
[1] https://github.com/apache/flink/pull/26358
Leonard Xu 于2025年3月27日周四 10:05写道：
> Your link is broken, Shengkai
>
> Best,
> Leonard
>
> > 2025年3月27日 10:01，Shengkai Fang 写道：
> >
> > Hi, All.
> >
> > I write
Your link is broken, Shengkai
Best,
Leonard
> 2025年3月27日 10:01，Shengkai Fang 写道：
>
> Hi, All.
>
> I write a simple demo to illustrate my idea. Hope this helps.
>
> Best,
> Shengkai
>
> https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1
>
> Gabor Somogyi 于2025年3
Hi, All.
I write a simple demo to illustrate my idea. Hope this helps.
Best,
Shengkai
https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1
Gabor Somogyi 于2025年3月26日周三 15:54写道：
> > I'm fine with a seperate SQL connector for metadata, so maybe we could
> update the FLI
> I'm fine with a seperate SQL connector for metadata, so maybe we could
update the FLIP about our discussion?
Sorry, I've forgotten this part. Yeah, no matter we choose I'm going to
update the FLIP.
G
On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi
wrote:
> Hi All,
>
> I've also lack of the kn
Hi All,
I've also lack of the knowledge of PTF so I've read just the motivation
part:
"The SQL 2016 standard introduced a way of defining custom SQL operators
defined by ISO/IEC 19075-7:2021 (Part 7: Polymorphic table functions).
~200 pages define how this new kind of function can consume and pro
Hi Zakelly , Shengkai!
I don't know too much about PTFs, it would be interesting to see how the
usage would look in practice.
Do you have some mockup/example in mind how the PTF would look for example
when want to:
- Simply display/aggregate whats in the metadata
- Join keyed state with some me
Hi everyone,
I'm fine with a seperate SQL connector for metadata, so maybe we could
update the FLIP about our discussion? And Shengkai provides a PTF
implementation, does that also meet the requirement?
Best,
Zakelly
On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi
wrote:
> Hi All,
>
> @Zakelly:
Hi All,
@Zakelly: Gyula summarised it correctly what I meant so please treat the
content as mine.
As an addition I'm not against to add CLI at all, I'm just stating that in
some cases like this, users would like to have
a self-serving solution where they can provide SQL statements which can
trigge
Hi!
@Zakelly Lan
I think what Gabor means is that users want to have predefined SQL scripts
to perform state analysis tasks to debug/identify problems.
Such as write a SQL script that joins the metadata table with the state and
do some analytics on it.
If we have a meta table then the SQL script
Hi Zaklely, Shengkai
Several topics are going on so adding gist answers to them. When some topic
is not touched please highlight it.
@Shengkai: I've read through all the previous FLIPs related catalogs and if
we would like to keep the concepts there
then one-to-one mapping relationship between sa
Hi all,
Thanks for your answers! Getting everyone aligned on this topic is
challenging, but it’s definitely worth the effort since it will help
streamline things moving forward.
@Gabor are you saying that users are using some scripts to define the SQL
metadata connector and get the information, r
Hi.
After reading the doc[1], I think Spark provides a function for users to
consume the metadata from the savepoint. In Flink SQL, similar
functionality is implemented through Polymorphic Table Functions (PTF) as
proposed in FLIP-440[2]. Below is a code example[3] illustrating this
concept:
```
Hi All!
Thank you for the answers and concerns from everyone.
On the CLI vs State Metadata Connector/Table question I would also like to
step back a little and look at the bigger picture.
I think the overall vision in Flink SQL is to provide a SQL native
environment where we can serve complex us
Hi, All.
About State Catalog, I want to share more thoughts about this.
In the initial design concept, I understood that a savepoint and a state
catalog have a one-to-one mapping relationship. Each operator corresponds
to a database, and the state of each operator is represented as individual
tab
Hi Gabor and Gyula,
Thanks for your valuable input and investigation.
Here's my thoughts:
The metadata is in very limited size. The small amount of data weakens the
need to analyze it **via SQL**. The main needs of users are reading and
showing that, instead of processing that. If users can quic
Hi All!
Without going into too much detail here are my 2 cents regarding the
virtual column / catalog metadata / table (connector) discussion for the
State metadata.
State metadata such as the types of states, their properties, names, sizes
etc are all valuable information that can be used to enr
1. State TTL for Value Columns
> I’m planning on adding this, and we may collaborate on it in the future.
+1 on this, just ping me.
2. Metadata Table vs. Metadata Column
After some code digging and POC all I can say that with heavy effort we can
maybe add such changes that we're able to show me
Hi, Gabor. Thanks for your response.
> 1. State TTL for Value Columns
Thank you for addressing the limitations here. However, I believe it would
be beneficial to further clarify the API in this FLIP regarding how users
can specify the TTL column.
One potential approach that comes to mind is usin
Looking forward to hearing the good news!
Best,
Shengkai
Gabor Somogyi 于2025年3月12日周三 22:24写道：
> Thanks for both the valuable input!
>
> Let me take a closer look at the suggestions, like the Catalog capabilities
> and possibility of embedding TypeInformation or
> StateDescriptor metadata direct
Thanks for both the valuable input!
Let me take a closer look at the suggestions, like the Catalog capabilities
and possibility of embedding TypeInformation or
StateDescriptor metadata directly into the raw state files...
BR,
G
On Wed, Mar 12, 2025 at 8:17 AM Shengkai Fang wrote:
> Thanks for
Thanks for Zakelly's clarification.
1. State TTL for Value Columns
+1 to delay the discussion about this.
2. Metadata Table vs. Metadata Column
I’d like to share my perspective on the State Catalog proposal. While
introducing this capability is beneficial, there is a blocker: the current
StateB
Hi Gabor and Shengkai,
Thanks for sharing your thoughts! This is a long discussion and sorry for
the late reply (I'm busy catching up with release 2.0 these days).
1. State TTL for Value Columns
Let me first clarify your thoughts to ensure I understand correctly. IIUC,
there is no persistent co
Hi Gabor,
> 2. Adding a new connector with `savepoint-metadata`
I would argue against introducing a new connector type named
savepoint-metadata, as the existing Catalog mechanism can inherently
provide the necessary connector factory capabilities. I’ve detailed this
proposal in branch[1]. Please
Hi Shengkai,
> 1. State TTL for Value Columns
>From directional perspective I agree your idea how it can be implemented.
Previously I've mentioned that TTL information is not exposed on the state
processor API (which the SQL state connector uses to read data)
and unless somebody show me the oppos
Hi Shengkai,
Please see my comments inline.
BR,
G
On Mon, Mar 3, 2025 at 7:07 AM Shengkai Fang wrote:
> Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP:
>
> 1. State TTL for Value Columns
> How can users retrieve the state TTL (Time-to-Live) for each value column?
>
Hi Shengkai,
Thanks for the questions. This week I'm off but will answer them when I'm
back.
G
On Mon, Mar 3, 2025, 07:07 Shengkai Fang wrote:
> Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP:
>
> 1. State TTL for Value Columns
> How can users retrieve the state TTL
Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP:
1. State TTL for Value Columns
How can users retrieve the state TTL (Time-to-Live) for each value column?
>From my understanding of the current design, it seems that this
functionality is not supported. Could you clarify if
Hi Zakelly,
In order to shoot for simplicity `METADATA VIRTUAL` as key words for
definition is the target.
When it's not super complex the latter can be added too.
BR,
G
On Sun, Mar 2, 2025 at 3:37 PM Zakelly Lan wrote:
> Hi Gabor,
>
> +1 for this.
>
> Will the metadata column use `METADATA V
Hi Gabor,
+1 for this.
Will the metadata column use `METADATA VIRTUAL` as key words for
definition, or `METADATA FROM xxx VIRTUAL` for renaming, just like the
Kafka table?
Best,
Zakelly
On Sat, Mar 1, 2025 at 1:31 PM Gabor Somogyi
wrote:
> Hi All,
>
> I'd like to start a discussion of FLIP-5
Hi All,
I'd like to start a discussion of FLIP-512: Add meta information to SQL
state connector [1].
Feel free to add your thoughts to make this feature better.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector
BR,
G
43 matches
Mail list logo
