Re: [DISCUSS] addition of Flink Script Operators

2023-08-07 Thread Kevin Tseng
Hi Jing, Thanks for the input, that's exactly how I have imagined the standard deployment process. With regards to the Exactly-Once guarantee as per my understanding: Current Exactly-Once semantics are performed via Two Phase Commit that is done at both Source and Sink (plus any checkpoint ena

Re: [DISCUSS] addition of Flink Script Operators

2023-08-04 Thread Jing Ge
Hi Kevin, Thanks for your proposal. It is quite interesting. We had a similar solution back to the Apache Storm era many years ago. The job deployment with operators itself could be considered as physical distributed task containers. There will be a management tool, like a control plane, to deploy

[DISCUSS] addition of Flink Script Operators

2023-08-01 Thread Kevin Tseng
Hello Dev Mailing List, I have a feature proposition that I am interested to hear more and to gather feedback from everyone on the mailing list. (please view this email in monospace font when possible) _*Proposal*_* * Currently, when Flink DataStream API is to be utilized, every data processin