Re: [VOTE] FLIP-529: Connections in Flink SQL and TableAPI

2025-07-31 Thread Shengkai Fang
+1(binding) Best, Shengkai Hao Li 于2025年7月30日周三 23:29写道: > Hi everyone, > > I'd like to start a vote on FLIP-529: Connections in Flink SQL and TableAPI > [1], which has been discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is an objection > or not enoug

Re: [DISCUSS] FLIP-539: Support WITH Clause in CREATE FUNCTION Statement in Flink SQL

2025-07-29 Thread Shengkai Fang
Hi. I have some questions about this FLIP: 1. What's the difference between the FunctionDescriptor and CatalogFunction? In my opinion, they are almost same here. If possible, can we reuse the same pojo class here? 2. The FLIP writes "The WITH clause will accept a list of key-value pairs similar

[DISCUSS] FLIP-540: Support VECTOR_SEARCH in Flink SQL

2025-07-29 Thread Shengkai Fang
Hi, I'd like to start a discussion of FLIP-540: Support VECTOR_SEARCH in Flink SQL[1]. In FLIP-437/FLIP-525, Apache Flink has initially integrated Large Language Model (LLM) capabilities, enabling semantic understanding and real-time processing of streaming data pipelines. This integration has bee

Re: Re: [DISCUSS] FLIP-535:Introduce RateLimiter to Source

2025-07-27 Thread Shengkai Fang
Hi, Zexian. Thanks for your FLIP. I think rate litmit is very important feature for our users. But I have some questions about the FLIP: 1. How do I determine the input parameter `requestSize` for `RateLimiter#acquire`? If the rate limiter indicates there are 2 remaining requests that can be emit

Re: [DISCUSS] FLIP-529 Connections in Flink SQL and Table API

2025-07-23 Thread Shengkai Fang
hi. Sorry for the late reply. I just have some questions: 1. Why SecretStoreFactory#open throws a CatalogException? I think the exteranl system can not handle this exception. 2. I think we can also modify the create catalog ddl syntax. ``` CREATE CATALOG cat USING CONNECTION mycat.mydb.mysql_pr

Re: [DISCUSS] State Schema Evolution for RowData

2025-07-23 Thread Shengkai Fang
; Please let me know if you have any further suggestions or questions. > > Thanks again for your valuable input. > > Best, > Weiqing > > On Sun, Jul 20, 2025 at 10:14 PM Shengkai Fang wrote: > > > Hi Weiqing. > > > > +1 for the FLIP. I have some suggestions

Re: [DISCUSS] State Schema Evolution for RowData

2025-07-20 Thread Shengkai Fang
Hi Weiqing. +1 for the FLIP. I have some suggestions about the FLIP: 1. Compared to adding a field named originalRowType in `RowDataSerializer`, I prefer to add a field named fieldNames with type String[] . WDYT? I think this field is used for name-based field mapping, so we just add the required

Re: [DISCUSS] Add UDF Metrics

2025-07-09 Thread Shengkai Fang
I just have some questions: 1. The current metrics hierarchy shows that the UDF metric group belongs to the TaskMetricGroup. I think it would be better for the UDF metric group to belong to the OperatorMetricGroup instead, because a UDF might be used by multiple operators. 2. What are the naming c

[jira] [Created] (FLINK-38028) Rename MLPredictUtils and FunctionCallUtils

2025-06-29 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-38028: - Summary: Rename MLPredictUtils and FunctionCallUtils Key: FLINK-38028 URL: https://issues.apache.org/jira/browse/FLINK-38028 Project: Flink Issue Type

[jira] [Created] (FLINK-37940) Support retry for ML_PREDICT

2025-06-10 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37940: - Summary: Support retry for ML_PREDICT Key: FLINK-37940 URL: https://issues.apache.org/jira/browse/FLINK-37940 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-37920) Reuse same utils for AsyncScalar, AsyncLookupJoin and AsyncMLPredict

2025-06-08 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37920: - Summary: Reuse same utils for AsyncScalar, AsyncLookupJoin and AsyncMLPredict Key: FLINK-37920 URL: https://issues.apache.org/jira/browse/FLINK-37920 Project

[jira] [Created] (FLINK-37904) Support state ttl for async ML_PREDICT

2025-06-05 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37904: - Summary: Support state ttl for async ML_PREDICT Key: FLINK-37904 URL: https://issues.apache.org/jira/browse/FLINK-37904 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-37903) Support runtime config for ML_PREDICT

2025-06-04 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37903: - Summary: Support runtime config for ML_PREDICT Key: FLINK-37903 URL: https://issues.apache.org/jira/browse/FLINK-37903 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-37901) Support to serde ExecNode for ML_PREDICT

2025-06-04 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37901: - Summary: Support to serde ExecNode for ML_PREDICT Key: FLINK-37901 URL: https://issues.apache.org/jira/browse/FLINK-37901 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-37902) Support batch execution for ML_PREDICT

2025-06-04 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37902: - Summary: Support batch execution for ML_PREDICT Key: FLINK-37902 URL: https://issues.apache.org/jira/browse/FLINK-37902 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-37899) Unstable JoinRestoreTest

2025-06-04 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37899: - Summary: Unstable JoinRestoreTest Key: FLINK-37899 URL: https://issues.apache.org/jira/browse/FLINK-37899 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-529 Connections in Flink SQL and Table API

2025-06-03 Thread Shengkai Fang
Thanks for Mayank's proposal! My bad! I had the wrong person listed as the initiator. Best, Shengkai Shengkai Fang 于2025年6月4日周三 12:07写道: > Hi, LiHao. > > Thanks for the proposal. The idea of connection is very attractive and > really helps Flink users to manage their secret

Re: [DISCUSS] FLIP-529 Connections in Flink SQL and Table API

2025-06-03 Thread Shengkai Fang
Hi, LiHao. Thanks for the proposal. The idea of connection is very attractive and really helps Flink users to manage their secret. But I have some concerns about this FLIP: *Proposal for a Top-Level Connection Concept in Flink SQL* I believe connection should be elevated as a core, top-level conc

Re: [VOTE] FLIP-526: Model ML_PREDICT, ML_EVALUATE Table API

2025-05-28 Thread Shengkai Fang
+1(binding) Best, Shengkai Timo Walther 于2025年5月29日周四 00:00写道: > +1 (binding) > > We should still check whether changes to QueryOperationVisitor are > necessary but this is internal API and should not the FLIP. The public > API looks correct to me. > > Cheers, > Timo > > > On 12.05.25 18:50, Ha

Re: [VOTE] FLIP-525: Model ML_PREDICT, ML_EVALUATE Implementation Design

2025-05-08 Thread Shengkai Fang
+1 (binding) Best, Shengkai Mayank Juneja 于2025年5月9日周五 03:18写道: > +1 (non-binding) > > On Thu, May 8, 2025 at 11:02 AM Hao Li wrote: > > > Hi everyone, > > > > I'd like to start a vote on FLIP-525: Model ML_PREDICT, ML_EVALUATE > > Implementation Design [1], which has been discussed in this th

Re: [DISCUSS] FLIP-526: Model ML_PREDICT, ML_EVALUATE Table API

2025-04-28 Thread Shengkai Fang
Hi, Hao. +1 for the proposal. Best, Shengkai Hao Li 于2025年4月29日周二 07:27写道: > Hi All, > > I would like to start a discussion about FLIP-526 [1]: Model ML_PREDICT, > ML_EVALUATE Table API. > > This FLIP is a follow up of FLIP-507 [2] to propose the table api for model > related functions. This F

Re: [DISCUSS] FLIP-525: Model ML_PREDICT, ML_EVALUATE Implementation Design

2025-04-28 Thread Shengkai Fang
L_PREDICT, > ML_EVALUATE Implementation Design. This FLIP is co-authored with Shengkai > Fang. > > This FLIP is a follow up of FLIP-437 [2] to propose the implementation > design for ML_PREDICT and ML_EVALUATE function which were introduced in > FLIP-437. > > For more details, see FLIP

Re: [DISCUSS] FLIP-517: Better Handling of Dynamic Table Primitives with PTFs

2025-04-23 Thread Shengkai Fang
Hi, Timo I believe I may have misapplied the query hints and table hints. The current proposal only applies to query hints. For table hints, I think we would need to create a view, apply the table hints to the view, and use it as the SEARCH_KEY function's input. Best, Shengkai Shengkai Fa

Re: [DISCUSS] FLIP-517: Better Handling of Dynamic Table Primitives with PTFs

2025-04-17 Thread Shengkai Fang
n about the naming of the SEARCH_KEY, considering > > that it is a simplified alternative to lookup join, and the word 'search' > > may be more likely to remind users of keyword searching which is a bit > > different from joins in sql, would it be better to consider follo

Re: [ANNOUNCE] New Apache Flink Committer - Xiqian Yu

2025-04-16 Thread Shengkai Fang
Congratulations! Best Shengkai Hongshun Wang 于2025年4月17日周四 10:39写道: > Congratulations! > > Best > Hongshun > > > 2025年4月16日 15:25,Leonard Xu 写道: > > > > becoming > >

Re: [ANNOUNCE] New Apache Flink Committer - Yanquan Lv

2025-04-16 Thread Shengkai Fang
Congratulations! Best Shengkai Samrat Deb 于2025年4月17日周四 11:12写道: > congratulations Yanquan Lv! > > bests, > Samrat > > On Thu, 17 Apr 2025 at 8:10 AM, Hongshun Wang > wrote: > > > Congrats Yanquan ! > > > > Best, > > Hongshun > > > > On Wed, Apr 16, 2025 at 9:04 PM gongzhongqiang < > gongzhong

Re: [DISCUSS] FLIP-517: Better Handling of Dynamic Table Primitives with PTFs

2025-04-14 Thread Shengkai Fang
Thanks for the FLIP, it helps a lot for us to develop features like VECTOR_SEARCH. But I have some questions about the FLIP: 1. Timing of Option Consumption for SEARCH_KEY Parameters For the FOR SYSTEM_TIME AS OF syntax, the planner leverages hints and catalog tables to load the table scan during

Re: Re: [VOTE] FLIP-512: Add meta information to SQL state connector

2025-04-05 Thread Shengkai Fang
+1(binding) Best, Shengkai Ferenc Csaky 于2025年3月31日周一 20:09写道: > +1 (binding) > > Best, > Ferenc > > > > > On Monday, March 31st, 2025 at 13:06, Rui Fan <1996fan...@gmail.com> > wrote: > > > > > > > +1(binding) > > > > Best, > > Rui > > > > On Mon, Mar 31, 2025 at 5:42 PM Andrei Kaigorodov < >

Re: Re: [ANNOUNCE] New Apache Flink PMC Member - Sergey Nuyanzin

2025-04-03 Thread Shengkai Fang
Congratulations, Sergey! Best, Shengkai Xuyang 于2025年4月3日周四 16:19写道: > Cheers, Sergey! > > > -- > > Best! > Xuyang > > > > > > 在 2025-04-03 15:58:48,"Zakelly Lan" 写道: > >Congratulations, Sergey! > > > > > >Best, > >Zakelly > > > >On Thu, Apr 3, 2025 at 3:10 PM Dawid Wysakowicz < > wysa

Re: [ANNOUNCE] New Apache Flink PMC Member - Zakelly Lan

2025-04-02 Thread Shengkai Fang
Congratulations, Zakelly! Best, Shengkai Yash Anand 于2025年4月2日周三 13:07写道: > Congratulations, Zakelly! > > On Wed, Apr 2, 2025 at 12:03 AM Sergey Nuyanzin > wrote: > > > Congratulations, Zakelly! > > > > On Wed, Apr 2, 2025, 05:59 Yunfeng Zhou > > wrote: > > > > > Congratulations! > > > > > >

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Shengkai Fang
; > well. > > > > > > > > Cheers, > > > > Gyula > > > > > > > > On Fri, Mar 28, 2025 at 7:39 AM Timo Walther > > wrote: > > > > > > > >> Hi Gabor, > > > >> > > > >>

Re: Process table function scan operation in batch mode is not supported

2025-03-27 Thread Shengkai Fang
I think it is by design. You can read the FLIP, it says: *Time Semantics*: - PTFs support event-time semantics only. - Processing-time doesn’t go well with batch mode and thus a unified API should built on event-time. The proposed onWatermark timers allow for making processing

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Shengkai Fang
this function. Best, Shengkai Shengkai Fang 于2025年3月28日周五 10:20写道: > +1 to use PTF. > > I would like to raise a consideration regarding the usage implementation: > Would it be necessary to allow users to utilize the CREATE FUNCTION > statement for registering the PTF? > > Cur

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Shengkai Fang
> In the meantime I've just updated the FLIP according to this to be > > optimistic 🙂 > > > > BR, > > G > > > > On Thu, Mar 27, 2025 at 2:15 PM Gabor Somogyi > > > wrote: > > > >> Considering all the facts I also +1 on PTF. Eve

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Shengkai Fang
it to 1.x line? > I'm not intended to do any backport, just would like to know the technical > constraints. > * Is it possible to describe such function to see the column names/types? > > BR, > G > > > On Thu, Mar 27, 2025 at 3:17 AM Shengkai Fang wrote: >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-26 Thread Shengkai Fang
Many thanks for your reminder, Leonard. Here's the link I mentioned[1]. Best, Shengkai [1] https://github.com/apache/flink/pull/26358 Leonard Xu 于2025年3月27日周四 10:05写道: > Your link is broken, Shengkai > > Best, > Leonard > > > 2025年3月27日 10:01,Shengkai Fang 写道:

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-26 Thread Shengkai Fang
SQL > >> > >> > metadata connector and get the information, right? If so, would a > >> CLI > >> > >> tool > >> > >> > be more convenient? It's easy to invoke and can get the result > >> > swiftly. > >> > &

Re: Question on usage of SQL Gateway with a remote Flink cluster

2025-03-23 Thread Shengkai Fang
Hi, Ammu. Thanks for the improvement, I have assigned it to you. Best, Shengkai Ammu P 于2025年3月24日周一 12:57写道: > Hi Shengkai, > > Thanks for the detailing. That clarified the setup related questions. > However it would be nice to have this captured somewhere in the gateway > documentation so

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-19 Thread Shengkai Fang
; to have some sort of checkpoint/savepoint lineage or however we call it. > > Since we've not yet reached this point there are no technical details, > it's > > more like a vision. It's a common pattern that > > jobs are physically running but somehow the state

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-18 Thread Shengkai Fang
he time, so a single connector seems to be an overkill. > > > > State data is also static within a savepoint and that's the reason why > the > > state processor API is working in batch mode. > > When we handle multiple checkpoints in a streaming fashion then this can &

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-15 Thread Shengkai Fang
e but not the per key state itself. > But again, this is a good feature as-is and can be handled in a separate > jira. +1 for a separate jira. Best, Shengkai Gabor Somogyi 于2025年3月10日周一 19:05写道: > Hi Shengkai, > > Please see my comments inline. > > BR, > G > > > O

Re: Question on usage of SQL Gateway with a remote Flink cluster

2025-03-12 Thread Shengkai Fang
Hi Ammu, The SQL Gateway enables clients to specify which cluster to submit jobs to. Here’s my understanding of the workflow: 1. The SQL client reads the configuration files (flink-conf.yaml or config.yaml) located in $FLINK_HOME/conf. The default configuration typically includes the rest.address

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-12 Thread Shengkai Fang
riptor metadata directly into the raw state files... > > BR, > G > > > On Wed, Mar 12, 2025 at 8:17 AM Shengkai Fang wrote: > > > Thanks for Zakelly's clarification. > > > > 1. State TTL for Value Columns > > > > +1 to delay the discussion about

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-12 Thread Shengkai Fang
without > specifying complex columns, as well as describe the savepoint catalog to > get the metadata. That's a good solution in my mind. > > > Best, > Zakelly > > On Wed, Mar 12, 2025 at 10:35 AM Shengkai Fang wrote: > > > Hi Gabor, > > > > > 2.

Re: Integrating LLMs into apache flink pipelines

2025-03-11 Thread Shengkai Fang
Thanks for your sharing! Could you confirm whether the pipeline is capable of generating structured summaries for product-specific comments or customer feedback? Best, Shengkai Ganesh 于2025年2月23日周日 13:38写道: > Hi Everyone, I've been experimenting with integrating LLMs into ETL and > data pipeli

Re: Question on usage of SQL Gateway with a remote Flink cluster

2025-03-11 Thread Shengkai Fang
Sql client uses the REST API to submit flink jobs. So you should use the option `rest.address` and `rest.port` to expose jobmanager address. Best, Shengkai Ammu P 于2025年3月5日周三 15:53写道: > Hi everyone, > > I am trying to run a SQL script through SQL Client in gateway mode. The > gateway is runnin

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-11 Thread Shengkai Fang
state itself. > > But again, this is a good feature as-is and can be handled in a separate > > jira. > > I've just created https://issues.apache.org/jira/browse/FLINK-37456. > > [1] https://www.postgresql.org/docs/current/view-pg-tables.html > [2] > > https://www.data

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Shengkai Fang
Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP: 1. State TTL for Value Columns How can users retrieve the state TTL (Time-to-Live) for each value column? >From my understanding of the current design, it seems that this functionality is not supported. Could you clarify if

Re: [ANNOUNCE] New Apache Flink Committer - Feng Jin

2025-02-20 Thread Shengkai Fang
Congratulations! Best, Shengkai Ron Liu 于2025年2月21日周五 11:00写道: > Congratulations > > Best, > Ron > > Yuepeng Pan 于2025年2月21日周五 09:57写道: > > > Congratulations ! > > > > > > Best, > > Yuepeng. > > > > > > > > > > > > At 2025-02-21 09:52:46, "Lincoln Lee" wrote: > > >Hi everyone, > > > > > >On b

Re: [ANNOUNCE] New Apache Flink Committer - Xuyang

2025-02-19 Thread Shengkai Fang
Congratulations! Best, Shengkai Lincoln Lee 于2025年2月20日周四 09:50写道: > Hi everyone, > > On behalf of the PMC, I'm happy to announce that Xuyang has become a > new Flink Committer! > > Xuyang has been contributing to the Flink community since Sep 15, 2021, he > has > driven and contributed to 5 FL

[jira] [Created] (FLINK-37186) Cross-team verification for "FLIP-480: Support to deploy SQL script in application mode"

2025-01-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-37186: - Summary: Cross-team verification for "FLIP-480: Support to deploy SQL script in application mode" Key: FLINK-37186 URL: https://issues.apache.org/jira/browse/F

Re: [VOTE] FLIP-492: Support Query Modifications for Materialized Tables

2025-01-01 Thread Shengkai Fang
+1(binding) Best, Shengkai Fang Yuepeng Pan 于2024年12月31日周二 15:58写道: > +1 (non-binding) > > > Best, > Yuepeng Pan > > > > > > > > > > > > > > > 在 2024-12-30 16:51:36,"Ron Liu" 写道: > >+1(binding) > > > >B

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-23 Thread Shengkai Fang
for the SQL connector > - Add non-keyed/operator state data support for the SQL connector > - Add state catalog > > BR, > G > > > On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang wrote: > > > Hi. > > > > Thanks Gabor's great FLIP. It is very useful for many ca

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-22 Thread Shengkai Fang
Hi. Thanks Gabor's great FLIP. It is very useful for many cases. But I have some suggestions about this FLIP: 1. It's better to introduce a state Catalog that helps users to translate the state information to a table schema. The state catalog uses ``.`` to find required state and uses state descr

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

2024-12-04 Thread Shengkai Fang
way to dynamically provide dependencies in > Python - similar to Java? > > (I haven't use the jar submission in Java) > > Thanks, > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=sales

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

2024-12-03 Thread Shengkai Fang
workloads on the same > session cluster. Making the session cluster essentially useless. > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > &g

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

2024-12-03 Thread Shengkai Fang
Hi. Why needs different image for taskmanager? Do you mean different operators require different resources? As far as I know, JM supports to manage taskmanager with different profiles. For example, a cluster may consists of two taskmanagers with following profiles: * TM1 contains 4 slots, every s

[jira] [Created] (FLINK-36761) Add document about how to deploy script with SQL Client and SQL Gateway

2024-11-21 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36761: - Summary: Add document about how to deploy script with SQL Client and SQL Gateway Key: FLINK-36761 URL: https://issues.apache.org/jira/browse/FLINK-36761 Project

[jira] [Created] (FLINK-36756) Bump up the sql-gateway rest api version

2024-11-21 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36756: - Summary: Bump up the sql-gateway rest api version Key: FLINK-36756 URL: https://issues.apache.org/jira/browse/FLINK-36756 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-36757) Introduce splitter to split scripts

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36757: - Summary: Introduce splitter to split scripts Key: FLINK-36757 URL: https://issues.apache.org/jira/browse/FLINK-36757 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-36760) Support to deploy script via sql client

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36760: - Summary: Support to deploy script via sql client Key: FLINK-36760 URL: https://issues.apache.org/jira/browse/FLINK-36760 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-36762) Add E2ECase to run sql client in application mode

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36762: - Summary: Add E2ECase to run sql client in application mode Key: FLINK-36762 URL: https://issues.apache.org/jira/browse/FLINK-36762 Project: Flink Issue

[jira] [Created] (FLINK-36759) Intrdouce REST API to deploy script

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36759: - Summary: Intrdouce REST API to deploy script Key: FLINK-36759 URL: https://issues.apache.org/jira/browse/FLINK-36759 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-36758) Introduce driver to load the application in application mode

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36758: - Summary: Introduce driver to load the application in application mode Key: FLINK-36758 URL: https://issues.apache.org/jira/browse/FLINK-36758 Project: Flink

[jira] [Created] (FLINK-36755) Relax the limit about the pipeline.jar must have one in application mode

2024-11-20 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36755: - Summary: Relax the limit about the pipeline.jar must have one in application mode Key: FLINK-36755 URL: https://issues.apache.org/jira/browse/FLINK-36755 Project

[jira] [Created] (FLINK-36702) FLIP-480: Support to deploy SQL script in application mode

2024-11-13 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36702: - Summary: FLIP-480: Support to deploy SQL script in application mode Key: FLINK-36702 URL: https://issues.apache.org/jira/browse/FLINK-36702 Project: Flink

[RESULT][VOTE] FLIP-480: Support to deploy script in application mode

2024-11-13 Thread Shengkai Fang
g(non-binding) - Ferenc Csaky(binding) - Zhu Zhu(binding) - shuai xu(non-binding) - Yuepeng Pan(non-binding) - Shengkai Fang(binding) There are no disapproving votes. Thanks everyone! Best, Shengkai [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-480%3A+Support+to+deploy+SQL+scri

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-13 Thread Shengkai Fang
+1(binding) I think I have received enough votes. Thanks for all the help and suggestions! Best, Shengkai

Re: Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Shengkai Fang
+1(binding) Best, Shengkai Xuyang 于2024年11月12日周二 10:36写道: > +1 (non-binding) > > > -- > > Best! > Xuyang > > > > > > 在 2024-11-12 03:42:33,"Jim Hughes" 写道: > >Hi Timo, > > > >I'm +1 (non-binding) on the FLIP. > > > >As a point of order, is the vote supposed to happen in a new thread? >

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Shengkai Fang
the outcome of this > discussion also back to FLIP-316 because as you said we will probably need > both gateway execution styles in the long run. > > Cheers, > Gyula > > > > On Mon, Nov 11, 2024 at 11:55 AM Shengkai Fang wrote: > > > Hi. > > > >

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Shengkai Fang
plement FLIP-480 first and then a > completely separate logic in FLIP-316 for the same thing. > > I think this FLIP FLIP-480 should not cover the SQL gateway and probably > should not change it in any way. The gateway related changes should be > consolidated in FLIP-316 and implemen

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-10 Thread Shengkai Fang
n other words, I think we need 2 different runners/designs. The direct app > submission should be able to run the script as is, but in the gateway I > think we need to have a runner based on the compiled sql plan instead. > > Cheers, > Gyula > > > On Thu, Nov 7, 2024 at 2:49 AM S

[VOTE]FLIP-480: Support to deploy script in application mode

2024-11-07 Thread Shengkai Fang
Hi everyone, I'd like to start a vote on FLIP-480: Support to deploy script in application mode[1]. The discussion can be found here[2]. The vote will be open for at least 72 hours unless there are any objections or insufficient votes. Best, Shengkai [1] https://cwiki.apache.org/confluence/disp

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-06 Thread Shengkai Fang
Hi, eveyone. The discussion doesn't receive any response for a while. I will close the discussion and start the vote tomorrow if the discussion doesn't receive any response today. Thanks all yours response! Best, Shengkai Shengkai Fang 于2024年11月5日周二 14:00写道: > Hi, Lincoln. Th

Re: [ANNOUNCE] New Apache Flink Committer - Junrui Li

2024-11-05 Thread Shengkai Fang
Congratulations! Best, Shengkai Lincoln Lee 于2024年11月6日周三 00:05写道: > Congratulations Junrui! > > > Best, > Lincoln Lee > > > Zhanghao Chen 于2024年11月5日周二 23:32写道: > > > Congrats, Junrui! > > > > > > Best, > > Zhanghao Chen > > > > From: Zhu Zhu > > Sent: Tuesda

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-04 Thread Shengkai Fang
e need to clarify the behavior when both are > empty, such as throwing an error? > Also use a unified name for script vs statements, like 'script'? > > 2. Regarding Python UDFs, should we change it to a > description of "Additional Python resources," corresponding >

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-04 Thread Shengkai Fang
i, Shegnkai. > > > > Thank you for your answer. I have no further questions. > > > > > > > > > > -- > > > > Best! > > Xuyang > > > > > > > > > > > > At 2024-11-04 10:00:32, "Shengkai Fang" fskm...@gmail

Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-03 Thread Shengkai Fang
r/docs/deployment/overview/#application-mode > > > > > -- > > Best! > Xuyang > > > > > > 在 2024-10-31 10:18:13,"Ron Liu" 写道: > >Hi, Shengkai > > > >Thanks for your quick response. It looks good to me. > > > &g

Re: [DISCUSS] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-02 Thread Shengkai Fang
Hi, Timo. Thanks for your proposal. This FLIP greatly extends the ease of use of SQL! But I have some questions about this FLIP: 1. How do users register PTF to Flink? It looks like users can use CREATE FUNCTION to register the PTF. 2. Can the input parameter of PTF be a view? If the input param

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-30 Thread Shengkai Fang
ad the future work for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-artifacts-local-upload-enabled Shengkai Fang 于2024年10月31日周四 09:30写道: > Hi, Feng! > > > if only clusterID is available, it may not be very convenient to

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-30 Thread Shengkai Fang
Hi, Feng! > if only clusterID is available, it may not be very convenient to connect to this application later on. If FLIP-479 is accepted, I think we can just adapt the sql-gateway behaviour to the behaviour that FLIP-479 mentioned. Best, Shengkai

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-28 Thread Shengkai Fang
rs/udf that are located at the $FLINK_HOME/lib directory. > Is the FLIP-316 still in need later? Yes. I think FLIP-316 is a great idea to use json plan to run the SQL Job and it brings great convenience to users to submit job in application mode in interactive mode. Best, Shengkai Shengkai

Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-28 Thread Shengkai Fang
to be made to the client here, > when deploying scripts in application mode, we need to call the newly added > interface of the gateway service. > > > Best, > Feng > > > On Thu, Oct 24, 2024 at 4:27 PM Shengkai Fang wrote: > > > Hi, everyone. > > > &g

Re: Consequences of removing per-job mode in Flink 2.0

2024-10-25 Thread Shengkai Fang
SQL gateway in our own platform as well to do the heavy > lifting regarding job deployment. Will watch those tickets. > > Thanks, > Ferenc > > > > > On Thursday, 10 October 2024 at 04:10, Shengkai Fang > wrote: > > > > > > > Hello Ferenc. Sorry for

[DISCUSS] FLIP-480: Support to deploy script in application mode

2024-10-24 Thread Shengkai Fang
Hi, everyone. I'd like to initiate a discussion about FLIP-480: Support to deploy script in application mode[1]. FLIP-480 supports to solve the problem that table program can not run in application mode. Comparing to FLIP-316[2], FLIP-480 tries to compile the script in the JM side, which is free

Re: [VOTE] FLIP-478: Introduce Config Option Scope

2024-10-16 Thread Shengkai Fang
+1(binding) Best, Shengkai Rui Fan <1996fan...@gmail.com> 于2024年10月17日周四 10:01写道: > +1(binding) > > Best, > Rui > > On Thu, Oct 17, 2024 at 10:00 AM Junrui Lee wrote: > > > Hi devs, > > > > I'd like to start a vote on the FLIP-478: Introduce Config Option > Scope[1]. > > The discussion thread i

Re: Consequences of removing per-job mode in Flink 2.0

2024-10-09 Thread Shengkai Fang
Hello Ferenc. Sorry for the late response. I think you are right and we should revert the FLINK-36310. We have already supported the SQL job submission based on application mode several months ago. I will write the design about this and reopen the FLIP-316 discussion later. We hope Flink-2.0 suppo

Re: PR review request for sql-gateway feature

2024-07-22 Thread Shengkai Fang
Dongwoo > > [1] > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/plugins/#custom-flink-resource-validators > > > > 2024년 7월 19일 (금) 오후 3:50, Shengkai Fang 님이 작성: > > > Hi Dongwoo, > > > > Thanks for your proposal

Re: PR review request for sql-gateway feature

2024-07-18 Thread Shengkai Fang
Hi Dongwoo, Thanks for your proposal! I think the idea about OperationValidator is similar to the plan advice[1], which can give advice if the statement has problems. I wonder if it would be more beneficial to develop the concept within the existing infrastructure, rather than introducing a new c

[jira] [Created] (FLINK-35735) Don't list catalogs when closing session

2024-07-01 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-35735: - Summary: Don't list catalogs when closing session Key: FLINK-35735 URL: https://issues.apache.org/jira/browse/FLINK-35735 Project: Flink Issue

Re: Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-10 Thread Shengkai Fang
+1 (binding) Best, Shengkai Ron Liu 于2024年5月10日周五 12:07写道: > +1(binding) > > Best, > Ron > > Jark Wu 于2024年5月10日周五 09:51写道: > > > +1 (binding) > > > > Best, > > Jark > > > > On Thu, 9 May 2024 at 21:27, Lincoln Lee wrote: > > > > > +1 (binding) > > > > > > Best, > > > Lincoln Lee > > > > > >

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-04-28 Thread Shengkai Fang
Hi, Liu. Thanks for your proposal. I have some questions about the FLIP: *WorkflowScheduler* 1. How to get the exception details if `modifyRefreshWorkflow` fails? 2. Could you give us an example about how to configure the scheduler? *SQL Gateway* 1. SqlGatewayService requires Session as the in

Re: [DISCUSS][FLINK-31830] Align the Nullability Handling of ROW between SQL and TableAPI

2023-12-24 Thread Shengkai Fang
Thanks for Jane and Sergey's proposal! +1 to correct the Table API behavior. I have one question: Is the influence only limited to the RowType? Does the Map or Array type have the same problems? Best, Shengkai [DISCUSS][FLINK-31830] Align the Nullability Handling of ROW between SQL and TableA J

Re: Re: Re: Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-08 Thread Shengkai Fang
GROUP BY f0, window_start, > window_end(window_start and window_end is produced by WINDOW TVF). We can > use field index here and use a new Calc node to alias them to avoid field > name conflict in WindowAggregate node. > What about this idea, cc @Shengkai Fang ? > > >I woul

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Shengkai Fang
Thanks for all Flink CDC maintainers great work! Big +1. Best, Shengkai tison 于2023年12月7日周四 12:01写道: > This is very cool! +1 from my side. > > Best, > tison. > > Benchao Li 于2023年12月7日周四 11:56写道: > > > > Thank you, Leonard and all the Flink CDC maintainers. > > > > Big big +1 from me. As a hea

Re: Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-04 Thread Shengkai Fang
Hi, Xuyang. Thanks for your great work. Big +1! After reading the FLIP, I have some questions. Please read the content below. 1. Support SESSION Window TVF Aggregation a. Could you give an example about the pass-through column? A session window may contain multiple rows, which value is select

[jira] [Created] (FLINK-33490) Validate the name conflicts when creating view

2023-11-08 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-33490: - Summary: Validate the name conflicts when creating view Key: FLINK-33490 URL: https://issues.apache.org/jira/browse/FLINK-33490 Project: Flink Issue Type

[jira] [Created] (FLINK-33446) SubQueryAntiJoinTest#testMultiNotExistsWithCorrelatedOnWhere_NestedCorrelation doesn't produce the correct plan

2023-11-02 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-33446: - Summary: SubQueryAntiJoinTest#testMultiNotExistsWithCorrelatedOnWhere_NestedCorrelation doesn't produce the correct plan Key: FLINK-33446 URL: https://issues.apache.org

[jira] [Created] (FLINK-33226) Forbid to drop current database

2023-10-09 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-33226: - Summary: Forbid to drop current database Key: FLINK-33226 URL: https://issues.apache.org/jira/browse/FLINK-33226 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-32618) Remove the dependency of the flink-core in the flink-sql-jdbc-driver-bundle

2023-07-18 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-32618: - Summary: Remove the dependency of the flink-core in the flink-sql-jdbc-driver-bundle Key: FLINK-32618 URL: https://issues.apache.org/jira/browse/FLINK-32618

  1   2   3   4   >