Hi Jincheng, Appreciated for the kind tips and offering of help. Definitely need it! Could you grant me write permission for confluence? My Id: Dian Fu
Thanks, Dian > 在 2019年8月26日，上午9:53，jincheng sun <sunjincheng...@gmail.com> 写道： > > Thanks for your feedback Hequn & Dian. > > Dian, I am glad to see that you want help to create the FLIP! > Everyone will have first time, and I am very willing to help you complete > your first FLIP creation. Here some tips: > > - First I'll give your account write permission for confluence. > - Before create the FLIP, please have look at the FLIP Template [1], (It's > better to know more about FLIP by reading [2]) > - Create Flink Python UDFs related JIRAs after completing the VOTE of > FLIP.(I think you also can bring up the VOTE thread, if you want! ) > > Any problems you encounter during this period，feel free to tell me that we > can solve them together. :) > > Best, > Jincheng > > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template > [2] > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > Hequn Cheng <chenghe...@gmail.com> 于2019年8月23日周五 上午11:54写道： > >> +1 for starting the vote. >> >> Thanks Jincheng a lot for the discussion. >> >> Best, Hequn >> >> On Fri, Aug 23, 2019 at 10:06 AM Dian Fu <dian0511...@gmail.com> wrote: >> >>> Hi Jincheng, >>> >>> +1 to start the FLIP create and VOTE on this feature. I'm willing to help >>> on the FLIP create if you don't mind. As I haven't created a FLIP before, >>> it will be great if you could help on this. :) >>> >>> Regards, >>> Dian >>> >>>> 在 2019年8月22日，下午11:41，jincheng sun <sunjincheng...@gmail.com> 写道： >>>> >>>> Hi all, >>>> >>>> Thanks a lot for your feedback. If there are no more suggestions and >>>> comments, I think it's better to initiate a vote to create a FLIP for >>>> Apache Flink Python UDFs. >>>> What do you think? >>>> >>>> Best, Jincheng >>>> >>>> jincheng sun <sunjincheng...@gmail.com> 于2019年8月15日周四 上午12:54写道： >>>> >>>>> Hi Thomas, >>>>> >>>>> Thanks for your confirmation and the very important reminder about >>> bundle >>>>> processing. >>>>> >>>>> I have had add the description about how to perform bundle processing >>> from >>>>> the perspective of checkpoint and watermark. Feel free to leave >>> comments if >>>>> there are anything not describe clearly. >>>>> >>>>> Best, >>>>> Jincheng >>>>> >>>>> >>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月14日周三 上午10:08写道： >>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thanks a lot the suggestions. >>>>>> >>>>>> Regarding to bundle processing, there is a section "Checkpoint"[1] in >>> the >>>>>> design doc which talks about how to handle the checkpoint. >>>>>> However, I think you are right that we should talk more about it, >> such >>> as >>>>>> what's bundle processing, how it affects the checkpoint and >> watermark, >>> how >>>>>> to handle the checkpoint and watermark, etc. >>>>>> >>>>>> [1] >>>>>> >>> >> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 >>>>>> < >>>>>> >>> >> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 >>>>>>> >>>>>> >>>>>> Regards, >>>>>> Dian >>>>>> >>>>>>> 在 2019年8月14日，上午1:01，Thomas Weise <t...@apache.org> 写道： >>>>>>> >>>>>>> Hi Jincheng, >>>>>>> >>>>>>> Thanks for putting this together. The proposal is very detailed, >>>>>> thorough >>>>>>> and for me as a Beam Flink runner contributor easy to understand :) >>>>>>> >>>>>>> One thing that you should probably detail more is the bundle >>>>>> processing. It >>>>>>> is critically important for performance that multiple elements are >>>>>>> processed in a bundle. The default bundle size in the Flink runner >> is >>>>>> 1s or >>>>>>> 1000 elements, whichever comes first. And for streaming, you can >> find >>>>>> the >>>>>>> logic necessary to align the bundle processing with watermarks and >>>>>>> checkpointing here: >>>>>>> >>>>>> >>> >> https://github.com/apache/beam/blob/release-2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java >>>>>>> >>>>>>> Thomas >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 13, 2019 at 7:05 AM jincheng sun < >>> sunjincheng...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> The Python Table API(without Python UDF support) has already been >>>>>> supported >>>>>>>> and will be available in the coming release 1.9. >>>>>>>> As Python UDF is very important for Python users, we'd like to >> start >>>>>> the >>>>>>>> discussion about the Python UDF support in the Python Table API. >>>>>>>> Aljoscha Krettek, Dian Fu and I have discussed offline and have >>>>>> drafted a >>>>>>>> design doc[1]. It includes the following items: >>>>>>>> >>>>>>>> - The user-defined function interfaces. >>>>>>>> - The user-defined function execution architecture. >>>>>>>> >>>>>>>> As mentioned by many guys in the previous discussion thread[2], a >>>>>>>> portability framework was introduced in Apache Beam in latest >>>>>> releases. It >>>>>>>> provides well-defined, language-neutral data structures and >> protocols >>>>>> for >>>>>>>> language-neutral user-defined function execution. This design is >>> based >>>>>> on >>>>>>>> Beam's portability framework. We will introduce how to make use of >>>>>> Beam's >>>>>>>> portability framework for user-defined function execution: data >>>>>>>> transmission, state access, checkpoint, metrics, logging, etc. >>>>>>>> >>>>>>>> Considering that the design relies on Beam's portability framework >>> for >>>>>>>> Python user-defined function execution and not all the contributors >>> in >>>>>>>> Flink community are familiar with Beam's portability framework, we >>> have >>>>>>>> done a prototype[3] for proof of concept and also ease of >>>>>> understanding of >>>>>>>> the design. >>>>>>>> >>>>>>>> Welcome any feedback. >>>>>>>> >>>>>>>> Best, >>>>>>>> Jincheng >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> >>>>>> >>> >> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit?usp=sharing >>>>>>>> [2] >>>>>>>> >>>>>>>> >>>>>> >>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html >>>>>>>> [3] https://github.com/dianfu/flink/commits/udf_poc >>>>>>>> >>>>>> >>>>>> >>> >>> >>
