Hello David, First of all thanks for the reply .Now answering your question I would like to state that Yes, I am aware that there is an existing Flink runner based on the DataStream and Operator API in the Beam codebase. However, the runner I propose to develop will differ from the existing one in a few significant ways.
Firstly, the existing runner uses Flink's lower-level Operator API, which provides more control over the execution of the pipeline but requires more manual configuration. In contrast, the runner I propose to develop will use Flink's higher-level DataStream API, which provides a more user-friendly and streamlined way to define streaming data processing pipelines. This will make it easier for Beam users to write and execute streaming pipelines using Flink. Secondly, the existing runner does not support Beam's windowing and triggering semantics in the Flink DataStream API, which can be a significant limitation for users who want to take advantage of these features. In contrast, the runner I propose to develop will provide support for these features, enabling Beam users to write streaming pipelines that use Beam's windowing and triggering semantics and execute them on the Flink streaming runtime. Overall, the runner I propose to develop will provide a more user-friendly and feature-rich integration between Beam and Flink, making it easier for Beam users to take advantage of Flink's high-performance capabilities for data processing. Hope you have found all the answers in the above lines. Best Regards, Siddharth Aryan. On Thu, Apr 27, 2023, 1:39 PM Jeff Zhang <zjf...@gmail.com> wrote: > Same question as David, one idea in my mind is to integrate the beam sql > api with flink table api, this does not exist in the current flink runner. > > On Thu, Apr 27, 2023 at 3:46 PM David Morávek <d...@apache.org> wrote: > >> Hi Siddharth, >> >> Thanks for your interest in the Flink Runner for Beam. Reading through >> the project, one thing that immediately strikes me is that there already is >> a Flink runner based on DataStream and Operator (one level below >> DataStream) API in the code base. Are you aware of this? If yes, how does >> the runner you want to introduce differ from the existing one? >> >> Best, >> D. >> >> On Sun, Apr 2, 2023 at 9:41 PM Svetak Sundhar via dev < >> dev@beam.apache.org> wrote: >> >>> Hi Siddharth, >>> I left some comments as well on the sentiment analysis proposal. >>> >>> Thanks, >>> >>> >>> Svetak Sundhar >>> >>> Technical Solutions Engineer, Data >>> s <nellywil...@google.com>vetaksund...@google.com >>> >>> >>> >>> On Sun, Apr 2, 2023 at 1:58 PM Anand Inguva via dev <dev@beam.apache.org> >>> wrote: >>> >>>> I left some comments on the sentiment analysis proposal. >>>> >>>> Thanks, >>>> Anand >>>> >>>> On Thu, Mar 30, 2023 at 9:59 AM Danny McCormick via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> Thanks Siddharth! I left some comments on the sentiment analysis >>>>> proposal, I am probably not the best person to comment on the flink >>>>> datastream api one though. >>>>> >>>>> Thanks, >>>>> Danny >>>>> >>>>> On Fri, Mar 24, 2023 at 11:53 PM Siddharth Aryan < >>>>> siddhartharyan...@gmail.com> wrote: >>>>> >>>>>> Hello , >>>>>> I am Siddharth Aryan a undergrad and I am looking forward to someone >>>>>> who can help me reviewing my proposal and give me a feedback on the them >>>>>> which help me to create a good proposal. >>>>>> Here ,I am attaching my both the project proposals: >>>>>> >Sentimental Analysis Pipeline with the help of Machine Learnig: >>>>>> >>>>>> https://docs.google.com/document/d/1U6zcXAWsDCrWlbf14f5VlLqPZFucwXR48tD7mrERW-g/edit?usp=sharing >>>>>> >>>>>> >Integrating Apache Beam with Flink Datastream API: >>>>>> >>>>>> https://docs.google.com/document/d/1sQEe9eVuoHX9QWS9Zj5wVl7MLmfk7QO09pjZOsk-TFY/edit?usp=sharing >>>>>> >>>>>> Best Regards >>>>>> Siddharth Aryan >>>>>> >>>>>> Github :https://github.com/nervoussidd >>>>>> >>>>> > > -- > Best Regards > > Jeff Zhang >
