Thanks everyone for the votes! I’ll summarize the voting result in a separate email.
Best, Wei > 在 2019年10月28日，11:38，jincheng sun <sunjincheng...@gmail.com> 写道： > > Hi Max, > > Thanks for your feedback. You are right, we really need a more generic > solution, I volunteer to draft an init solution design doc, and bring up > the discussion in Beam @dev ASAP. (Maybe after release of Flink 1.10). > > Thank you for the voting. > > Best, > Jincheng > > Maximilian Michels <m...@apache.org> 于2019年10月26日周六 上午1:05写道： > >> Hi Wei, hi Jincheng, >> >> +1 on the current approach. >> >> I agree it would be nice to allow for the Beam artifact staging to use >> Flink's BlobServer. However, the current implementation which uses the >> distributed file system is more generic, since the BlobServer is only >> available on the TaskManager and not necessarily inside Harness >> containers (Stage 3). >> >> So for stage 1 (Client <=> JobServer) we could certainly use the >> BlobServer. Stage 2 (Flink job submission) already uses it, and Stage 3 >> (container setup) probably has to have some form of distributed file >> system or directory which has been populated with the dependencies. >> >> Thanks, >> Max >> >> On 25.10.19 03:45, Wei Zhong wrote: >>> Hi Max, >>> >>> Is there any other concerns from your side? I appreciate if you can give >> some feedback and vote on this. >>> >>> Best, >>> Wei >>> >>>> 在 2019年10月25日，09:33，jincheng sun <sunjincheng...@gmail.com> 写道： >>>> >>>> Hi Thomas, >>>> >>>> Thanks for your explanation. I understand your original intention. I >> will >>>> seriously consider this issue. After I have the initial solution, I will >>>> bring up a further discussion in Beam ML. >>>> >>>> Thanks for your voting. :) >>>> >>>> Best, >>>> Jincheng >>>> >>>> >>>> Thomas Weise <t...@apache.org> 于2019年10月25日周五 上午7:32写道： >>>> >>>>> Hi Jincheng, >>>>> >>>>> Yes, this topic can be further discussed on the Beam ML. The only >> reason I >>>>> brought it up here is that it would be desirable from Beam Flink runner >>>>> perspective for the artifact staging mechanism that you work on to be >>>>> reusable. >>>>> >>>>> Stage 1 in Beam is also up to the runner, artifact staging is a service >>>>> discovered from the job server and that the Flink job server currently >> uses >>>>> DFS is not set in stone. My interest was more regarding assumptions >>>>> regarding the artifact structure, which may or may not allow for >> reusable >>>>> implementation. >>>>> >>>>> +1 for the proposal otherwise >>>>> >>>>> Thomas >>>>> >>>>> >>>>> On Mon, Oct 21, 2019 at 8:40 PM jincheng sun <sunjincheng...@gmail.com >>> >>>>> wrote: >>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thanks for sharing your thoughts. I think improve and solve the >>>>> limitations >>>>>> of the Beam artifact staging is good topic(For beam). >>>>>> >>>>>> As I understand it as follows: >>>>>> >>>>>> For Beam(data): >>>>>> Stage1: BeamClient ------> JobService (data will be upload to >> DFS). >>>>>> Stage2: JobService(FlinkClient) ------> FlinkJob (operator >> download >>>>>> the data from DFS) >>>>>> Stage3: Operator ------> Harness(artifact staging service) >>>>>> >>>>>> For Flink(data): >>>>>> Stage1: FlinkClient(data(local) upload to BlobServer using >> distribute >>>>>> cache) ------> Operator (data will be download from BlobServer). Do >> not >>>>>> have to depend on DFS. >>>>>> Stage2: Operator ------> Harness(for docker we using artifact >> staging >>>>>> service) >>>>>> >>>>>> So, I think Beam have to depend on DFS in Stage1. and Stage2 can using >>>>>> distribute cache if we remove the dependency of DFS for Beam in >>>>> Stage1.(Of >>>>>> course we need more detail here), we can bring up the discussion in a >>>>>> separate Beam dev@ ML, the current discussion focuses on Flink 1.10 >>>>>> version >>>>>> of UDF Environment and Dependency Management for python, so I >> recommend >>>>>> voting in the current ML for Flink 1.10, Beam artifact staging >>>>> improvements >>>>>> are discussed in a separate Beam dev@. >>>>>> >>>>>> What do you think? >>>>>> >>>>>> Best, >>>>>> Jincheng >>>>>> >>>>>> Thomas Weise <t...@apache.org> 于2019年10月21日周一 下午10:25写道： >>>>>> >>>>>>> Beam artifact staging currently relies on shared file system and >> there >>>>>> are >>>>>>> limitations, for example when running locally with Docker and local >> FS. >>>>>> It >>>>>>> sounds like a distributed cache based implementation might be a good >>>>>>> (better?) option for artifact staging even for the Beam Flink runner? >>>>>>> >>>>>>> If so, can the implementation you propose be compatible with the Beam >>>>>>> artifact staging service so that it can be plugged into the Beam >> Flink >>>>>>> runner? >>>>>>> >>>>>>> Thanks, >>>>>>> Thomas >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 21, 2019 at 2:34 AM jincheng sun < >> sunjincheng...@gmail.com >>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Max, >>>>>>>> >>>>>>>> Sorry for the late reply. Regarding the issue you mentioned above, >>>>> I'm >>>>>>> glad >>>>>>>> to share my thoughts: >>>>>>>> >>>>>>>>> For process-based execution we use Flink's cache distribution >>>>> instead >>>>>>> of >>>>>>>> Beam's artifact staging. >>>>>>>> >>>>>>>> In current design, we use Flink's cache distribution to upload >> users' >>>>>>> files >>>>>>>> from client to cluster in both docker mode and process mode. That >> is, >>>>>>>> Flink's cache distribution and Beam's artifact staging service work >>>>>>>> together in docker mode. >>>>>>>> >>>>>>>> >>>>>>>>> Do we want to implement two different ways of staging artifacts? It >>>>>>> seems >>>>>>>> sensible to use the same artifact staging functionality also for the >>>>>>>> process-based execution. >>>>>>>> >>>>>>>> I agree that the implementation will be simple if we use the same >>>>>>> artifact >>>>>>>> staging functionality also for the process-based execution. However, >>>>>> it's >>>>>>>> not the best for performance as it will introduce an additional >>>>> network >>>>>>>> transmission, as in process mode TaskManager and python worker share >>>>>> the >>>>>>>> same environment, in which case the user files in Flink Distribute >>>>>> Cache >>>>>>>> can be accessed by python worker directly. We do not need the >> staging >>>>>>>> service in this case. >>>>>>>> >>>>>>>>> Apart from being simpler, this would also allow the process-based >>>>>>>> execution to run in other environments than the Flink TaskManager >>>>>>>> environment. >>>>>>>> >>>>>>>> IMHO, this case is more like docker mode, and we can share or reuse >>>>> the >>>>>>>> code of Beam docker mode. Furthermore, in this case python worker is >>>>>>>> launched by the operator, so it is always in the same environment as >>>>>> the >>>>>>>> operator. >>>>>>>> >>>>>>>> Thanks again for your feedback, and it is valuable for find out the >>>>>> final >>>>>>>> best architecture. >>>>>>>> >>>>>>>> Feel free to correct me if there is anything incorrect. >>>>>>>> >>>>>>>> Best, >>>>>>>> Jincheng >>>>>>>> >>>>>>>> Maximilian Michels <m...@apache.org> 于2019年10月16日周三 下午4:23写道： >>>>>>>> >>>>>>>>> I'm also late to the party here :) When I saw the first draft, I >>>>> was >>>>>>>>> thinking how exactly the design doc would tie in with Beam. Thanks >>>>>> for >>>>>>>>> the update. >>>>>>>>> >>>>>>>>> A couple of comments with this regard: >>>>>>>>> >>>>>>>>>> Flink has provided a distributed cache mechanism and allows users >>>>>> to >>>>>>>>> upload their files using "registerCachedFile" method in >>>>>>>>> ExecutionEnvironment/StreamExecutionEnvironment. The python files >>>>>> users >>>>>>>>> specified through "add_python_file", "set_python_requirements" and >>>>>>>>> "add_python_archive" are also uploaded through this method >>>>>> eventually. >>>>>>>>> >>>>>>>>> For process-based execution we use Flink's cache distribution >>>>> instead >>>>>>> of >>>>>>>>> Beam's artifact staging. >>>>>>>>> >>>>>>>>>> Apache Beam Portability Framework already supports artifact >>>>> staging >>>>>>>> that >>>>>>>>> works out of the box with the Docker environment. We can use the >>>>>>> artifact >>>>>>>>> staging service defined in Apache Beam to transfer the dependencies >>>>>>> from >>>>>>>>> the operator to Python SDK harness running in the docker container. >>>>>>>>> >>>>>>>>> Do we want to implement two different ways of staging artifacts? It >>>>>>>>> seems sensible to use the same artifact staging functionality also >>>>>> for >>>>>>>>> the process-based execution. Apart from being simpler, this would >>>>>> also >>>>>>>>> allow the process-based execution to run in other environments than >>>>>> the >>>>>>>>> Flink TaskManager environment. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Max >>>>>>>>> >>>>>>>>> On 15.10.19 11:13, Wei Zhong wrote: >>>>>>>>>> Hi Thomas, >>>>>>>>>> >>>>>>>>>> Thanks a lot for your suggestion! >>>>>>>>>> >>>>>>>>>> As you can see from the section "Goals" that this FLIP focuses on >>>>>> the >>>>>>>>> dependency management in process mode. However, the APIs and design >>>>>>>>> proposed in this FLIP also applies for the docker mode. So it makes >>>>>>> sense >>>>>>>>> to me to also describe how this design is integated to the artifact >>>>>>>> staging >>>>>>>>> service of Apache Beam in docker mode. I have updated the design >>>>> doc >>>>>>> and >>>>>>>>> looking forward to your feedback. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Wei >>>>>>>>>> >>>>>>>>>>> 在 2019年10月15日，01:54，Thomas Weise <t...@apache.org> 写道： >>>>>>>>>>> >>>>>>>>>>> Sorry for joining the discussion late. >>>>>>>>>>> >>>>>>>>>>> The Beam environment already supports artifact staging, it works >>>>>> out >>>>>>>> of >>>>>>>>> the >>>>>>>>>>> box with the Docker environment. I think it would be helpful to >>>>>>>> explain >>>>>>>>> in >>>>>>>>>>> the FLIP how this proposal relates to what Beam offers / how it >>>>>>> would >>>>>>>> be >>>>>>>>>>> integrated. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Thomas >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 14, 2019 at 8:09 AM Jeff Zhang <zjf...@gmail.com> >>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> Hequn Cheng <chenghe...@gmail.com> 于2019年10月14日周一 下午10:55写道： >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> Good job, Wei! >>>>>>>>>>>>> >>>>>>>>>>>>> Best, Hequn >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:54 PM Dian Fu < >>>>> dian0511...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Wei, >>>>>>>>>>>>>> >>>>>>>>>>>>>> +1 (non-binding). Thanks for driving this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Dian >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 在 2019年10月14日，下午1:40，jincheng sun <sunjincheng...@gmail.com >>>>>> >>>>>>> 写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Wei Zhong <weizhong0...@gmail.com> 于2019年10月12日周六 下午8:41写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would like to start the vote for FLIP-78[1] which is >>>>>>> discussed >>>>>>>>> and >>>>>>>>>>>>>>>> reached consensus in the discussion thread[2]. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The vote will be open for at least 72 hours. I'll try to >>>>>> close >>>>>>> it >>>>>>>>> by >>>>>>>>>>>>>>>> 2019-10-16 18:00 UTC, unless there is an objection or not >>>>>>> enough >>>>>>>>>>>>> votes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Wei >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management >>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-78:+Flink+Python+UDF+Environment+and+Dependency+Management >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-UDF-Environment-and-Dependency-Management-td33514.html >>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-UDF-Environment-and-Dependency-Management-td33514.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best Regards >>>>>>>>>>>> >>>>>>>>>>>> Jeff Zhang >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >>
