Thanks for updating the FLIP, Yangze. I agree with Till that we probably want to separate the K8s/Yarn decorator calls. Users can still configure one driver class, and we can use `instanceof` to check whether the driver implemented K8s/Yarn specific interfaces.
Moreover, I'm not sure about exposing entire `ContainerRequest` / `Pod` (`AbstractKubernetesStepDecorator` directly manipulates on `Pod`) to user codes. It gives more access to user codes than needed for defining external resource, which might cause problems. Instead, I would suggest to have interface like `Map<String key, String value> getYarn/KubernetesExternalResource()` and assemble them into `ContainerRequest` / `Pod` in Yarn/KubernetesResourceManager. Thank you~ Xintong Song On Fri, Mar 27, 2020 at 1:10 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi everyone, > > I'm a bit late to the party. I think the current proposal looks good. > > Concerning the ExternalResourceDriver interface defined in the FLIP [1], I > would suggest to not include the decorator calls for Kubernetes and Yarn in > the base interface. Instead I would suggest to segregate the deployment > specific decorator calls into separate interfaces. That way an > ExternalResourceDriver does not have to support all deployments from the > very beginning. Moreover, some resources might not be supported by a > specific deployment target and the natural way to express this would be to > not implement the respective deployment specific interface. > > Moreover, having void > addExternalResourceToRequest(AMRMClient.ContainerRequest containerRequest) > in the ExternalResourceDriver interface would require Hadoop on Flink's > classpath whenever the external resource driver is being used. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink > > Cheers, > Till > > On Thu, Mar 26, 2020 at 12:45 PM Stephan Ewen <se...@apache.org> wrote: > > > Nice, thanks a lot! > > > > On Thu, Mar 26, 2020 at 10:21 AM Yangze Guo <karma...@gmail.com> wrote: > > > > > Thanks for the suggestion, @Stephan, @Becket and @Xintong. > > > > > > I've updated the FLIP accordingly. I do not add a > > > ResourceInfoProvider. Instead, I introduce the ExternalResourceDriver, > > > which takes the responsibility of all relevant operations on both RM > > > and TM sides. > > > After a rethink about decoupling the management of external resources > > > from TaskExecutor, I think we could do the same thing on the > > > ResourceManager side. We do not need to add a specific allocation > > > logic to the ResourceManager each time we add a specific external > > > resource. > > > - For Yarn, we need the ExternalResourceDriver to edit the > > > containerRequest. > > > - For Kubenetes, ExternalResourceDriver could provide a decorator for > > > the TM pod. > > > > > > In this way, just like MetricReporter, we allow users to define their > > > custom ExternalResourceDriver. It is more extensible and fits the > > > separation of concerns. For more details, please take a look at [1]. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink > > > > > > Best, > > > Yangze Guo > > > > > > On Wed, Mar 25, 2020 at 7:32 PM Stephan Ewen <se...@apache.org> wrote: > > > > > > > > This sounds good to go ahead from my side. > > > > > > > > I like the approach that Becket suggested - in that case the core > > > > abstraction that everyone would need to understand would be "external > > > > resource allocation" and the "ResourceInfoProvider", and the GPU > > specific > > > > code would be a specific implementation only known to that component > > that > > > > allocates the external resource. That fits the separation of concerns > > > well. > > > > > > > > I also understand that it should not be over-engineered in the first > > > > version, so some simplification makes sense, and then gradually > expand > > > from > > > > there. > > > > > > > > So +1 to go ahead with what was suggested above (Xintong / Becket) > from > > > my > > > > side. > > > > > > > > On Mon, Mar 23, 2020 at 6:55 AM Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > Thanks for the comments, Stephan & Becket. > > > > > > > > > > @Stephan > > > > > > > > > > I see your concern, and I completely agree with you that we should > > > first > > > > > think about the "library" / "plugin" / "extension" style if > possible. > > > > > > > > > > If GPUs are sliced and assigned during scheduling, there may be > > reason, > > > > > > although it looks that it would belong to the slot then. Is that > > > what we > > > > > > are doing here? > > > > > > > > > > > > > > > In the current proposal, we do not have the GPUs sliced and > assigned > > to > > > > > slots, because it could be problematic without dynamic slot > > allocation. > > > > > E.g., the number of GPUs might not be evenly divisible by the > number > > of > > > > > slots. > > > > > > > > > > I think it makes sense to eventually have the GPUs assigned to > slots. > > > Even > > > > > then, we might still need a TM level GPUManager (or > ResourceProvider > > > like > > > > > Becket suggested). For memory, in each slot we can simply request > the > > > > > amount of memory, leaving it to JVM / OS to decide which memory > > > (address) > > > > > should be assigned. For GPU, and potentially other resources like > > > FPGA, we > > > > > need to explicitly specify which GPU (index) should be used. > > > Therefore, we > > > > > need some component at the TM level to coordinate which slot uses > > which > > > > > GPU. > > > > > > > > > > IMO, unless we say Flink will not support slot-level GPU slicing at > > > least > > > > > in the foreseeable future, I don't see a good way to avoid touching > > > the TM > > > > > core. To that end, I think Becket's suggestion points to a good > > > direction, > > > > > that supports more features (GPU, FPGA, etc.) with less coupling to > > > the TM > > > > > core (only needs to understand the general interfaces). The > detailed > > > > > implementation for specific resource types can even be encapsulated > > as > > > a > > > > > library. > > > > > > > > > > @Becket > > > > > > > > > > Thanks for sharing your thought on the final state. Despite the > > > details how > > > > > the interfaces should look like, I think this is a really good > > > abstraction > > > > > for supporting general resource types. > > > > > > > > > > I'd like to further clarify that, the following three things are > all > > > that > > > > > the "Flink core" needs to understand. > > > > > > > > > > - The *amount* of resource, for scheduling. Actually, we already > > > have > > > > > the Resource class in ResourceProfile and ResourceSpec for > > extended > > > > > resource. It's just not really used. > > > > > - The *info*, that Flink provides to the operators / user codes. > > > > > - The *provider*, which generates the info based on the amount. > > > > > > > > > > The "core" does not need to understand the specific implementation > > > details > > > > > of the above three. They can even be implemented in a 3rd-party > > > library. > > > > > Similar to how we allow users to define their custom > MetricReporter. > > > > > > > > > > Thank you~ > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > On Mon, Mar 23, 2020 at 8:45 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > > > Thanks for the comment, Stephan. > > > > > > > > > > > > - If everything becomes a "core feature", it will make the > > project > > > hard > > > > > > > to develop in the future. Thinking "library" / "plugin" / > > > "extension" > > > > > > style > > > > > > > where possible helps. > > > > > > > > > > > > > > > > > > Completely agree. It is much more important to design a mechanism > > > than > > > > > > focusing on a specific case. Here is what I am thinking to fully > > > support > > > > > > custom resource management: > > > > > > 1. On the JM / RM side, use ResourceProfile and ResourceSpec to > > > define > > > > > the > > > > > > resource and the amount required. They will be used to find > > suitable > > > TMs > > > > > > slots to run the tasks. At this point, the resources are only > > > measured by > > > > > > amount, i.e. they do not have individual ID. > > > > > > > > > > > > 2. On the TM side, have something like *"ResourceInfoProvider"* > to > > > > > identify > > > > > > and provides the detail information of the individual resource, > > e.g. > > > GPU > > > > > > ID.. It is important because the operator may have to explicitly > > > interact > > > > > > with the physical resource it uses. The ResourceInfoProvider > might > > > look > > > > > > like something below. > > > > > > interface ResourceInfoProvider<INFO> { > > > > > > Map<AbstractID, INFO> retrieveResourceInfo(OperatorId opId, > > > > > > ResourceProfile resourceProfile); > > > > > > } > > > > > > > > > > > > - There could be several "*ResourceInfoProvider*" configured on > the > > > TM to > > > > > > retrieve the information for different resources. > > > > > > - The TM will be responsible to assign those individual resources > > to > > > each > > > > > > operator according to their requested amount. > > > > > > - The operators will be able to get the ResourceInfo from their > > > > > > RuntimeContext. > > > > > > > > > > > > If we agree this is a reasonable final state. We can adapt the > > > current > > > > > FLIP > > > > > > to it. In fact it does not sound a big change to me. All the > > proposed > > > > > > configuration can be as is, it is just that Flink itself won't > care > > > about > > > > > > them, instead a GPUInfoProviver implementing the > > ResourceInfoProvider > > > > > will > > > > > > use them. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Mon, Mar 23, 2020 at 1:47 AM Stephan Ewen <se...@apache.org> > > > wrote: > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > The main point I wanted to throw into the discussion is the > > > following: > > > > > > > - With more and more use cases, more and more tools go into > > Flink > > > > > > > - If everything becomes a "core feature", it will make the > > > project > > > > > hard > > > > > > > to develop in the future. Thinking "library" / "plugin" / > > > "extension" > > > > > > style > > > > > > > where possible helps. > > > > > > > > > > > > > > - A good thought experiment is always: How many future > > developers > > > > > have > > > > > > to > > > > > > > interact with this code (and possibly understand it partially), > > > even if > > > > > > the > > > > > > > features they touch have nothing to do with GPU support. If > many > > > > > > > contributors to unrelated features will have to touch it and > > > understand > > > > > > it, > > > > > > > then let's think if there is a different solution. Maybe there > is > > > not, > > > > > > but > > > > > > > then we should be sure why. > > > > > > > > > > > > > > - That led me to raising this issue: If the GPU manager > > becomes a > > > > > core > > > > > > > service in the TaskManager, Environment, RuntimeContext, etc. > > then > > > > > > everyone > > > > > > > developing TM and streaming tasks need to understand the GPU > > > manager. > > > > > > That > > > > > > > seems oddly specific, is my impression. > > > > > > > > > > > > > > Access to configuration seems not the right reason to do that. > We > > > > > should > > > > > > > expose the Flink configuration from the RuntimeContext anyways. > > > > > > > > > > > > > > If GPUs are sliced and assigned during scheduling, there may be > > > reason, > > > > > > > although it looks that it would belong to the slot then. Is > that > > > what > > > > > we > > > > > > > are doing here? > > > > > > > > > > > > > > Best, > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 20, 2020 at 2:58 AM Xintong Song < > > > tonysong...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for the feedback, Becket. > > > > > > > > > > > > > > > > IMO, eventually an operator should only see info of GPUs that > > are > > > > > > > dedicated > > > > > > > > for it, instead of all GPUs on the machine/container in the > > > current > > > > > > > design. > > > > > > > > It does not make sense to let the user who writes a UDF to > > worry > > > > > about > > > > > > > > coordination among multiple operators running on the same > > > machine. > > > > > And > > > > > > if > > > > > > > > we want to limit the GPU info an operator sees, we should not > > > let the > > > > > > > > operator to instantiate GPUManager, which means we have to > > expose > > > > > > > something > > > > > > > > through runtime context, either GPU info or some kind of > > limited > > > > > access > > > > > > > to > > > > > > > > the GPUManager. > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 19, 2020 at 5:48 PM Becket Qin < > > becket....@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > It probably make sense for us to first agree on the final > > > state. > > > > > More > > > > > > > > > specifically, will the resource info be exposed through > > runtime > > > > > > context > > > > > > > > > eventually? > > > > > > > > > > > > > > > > > > If that is the final state and we have a seamless migration > > > story > > > > > > from > > > > > > > > this > > > > > > > > > FLIP to that final state, Personally I think it is OK to > > > expose the > > > > > > GPU > > > > > > > > > info in the runtime context. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2020 at 11:21 AM Xintong Song < > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > @Yangze, > > > > > > > > > > I think what Stephan means (@Stephan, please correct me > if > > > I'm > > > > > > wrong) > > > > > > > > is > > > > > > > > > > that, we might not need to hold and maintain the > GPUManager > > > as a > > > > > > > > service > > > > > > > > > in > > > > > > > > > > TaskManagerServices or RuntimeContext. An alternative is > to > > > > > create > > > > > > / > > > > > > > > > > retrieve the GPUManager only in the operators that need > it, > > > e.g., > > > > > > > with > > > > > > > > a > > > > > > > > > > static method `GPUManager.get()`. > > > > > > > > > > > > > > > > > > > > @Stephan, > > > > > > > > > > I agree with you on excluding GPUManager from > > > > > TaskManagerServices. > > > > > > > > > > > > > > > > > > > > - For the first step, where we provide unified > TM-level > > > GPU > > > > > > > > > information > > > > > > > > > > to all operators, it should be fine to have operators > > > access / > > > > > > > > > > lazy-initiate GPUManager by themselves. > > > > > > > > > > - In future, we might have some more fine-grained GPU > > > > > > management, > > > > > > > > > where > > > > > > > > > > we need to maintain GPUManager as a service and put > GPU > > > info > > > > > in > > > > > > > slot > > > > > > > > > > profiles. But at least for now it's not necessary to > > > introduce > > > > > > > such > > > > > > > > > > complexity. > > > > > > > > > > > > > > > > > > > > However, I have some concerns on excluding GPUManager > from > > > > > > > > RuntimeContext > > > > > > > > > > and let operators access it directly. > > > > > > > > > > > > > > > > > > > > - Configurations needed for creating the GPUManager is > > not > > > > > > always > > > > > > > > > > available for operators. > > > > > > > > > > - If later we want to have fine-grained control over > GPU > > > > > (e.g., > > > > > > > > > > operators in each slot can only see GPUs reserved for > > that > > > > > > slot), > > > > > > > > the > > > > > > > > > > approach cannot be easily extended. > > > > > > > > > > > > > > > > > > > > I would suggest to wrap the GPUManager behind > > RuntimeContext > > > and > > > > > > only > > > > > > > > > > expose the GPUInfo to users. For now, we can declare a > > method > > > > > > > > > > `getGPUInfo()` in RuntimeContext, with a default > definition > > > that > > > > > > > calls > > > > > > > > > > `GPUManager.get()` to get the lazily-created GPUManager. > If > > > later > > > > > > we > > > > > > > > want > > > > > > > > > > to create / retrieve GPUManager in a different way, we > can > > > simply > > > > > > > > change > > > > > > > > > > how `getGPUInfo` is implemented, without needing to > change > > > any > > > > > > public > > > > > > > > > > interfaces. > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 14, 2020 at 10:09 AM Yangze Guo < > > > karma...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > @Shephan > > > > > > > > > > > Do you mean Minicluster? Yes, it makes sense to share > the > > > GPU > > > > > > > Manager > > > > > > > > > > > in such scenario. > > > > > > > > > > > If that's what you worry about, I'm +1 for holding > > > > > > > > > > > GPUManager(ExternalResourceManagers) in TaskExecutor > > > instead of > > > > > > > > > > > TaskManagerServices. > > > > > > > > > > > > > > > > > > > > > > Regarding the RuntimeContext/FunctionContext, it just > > > holds the > > > > > > GPU > > > > > > > > > > > info instead of the GPU Manager. AFAIK, it's the only > > > place we > > > > > > > could > > > > > > > > > > > pass GPU info to the RichFunction/UserDefinedFunction. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 14, 2020 at 4:06 AM Isaac Godfried < > > > > > > > is...@paddlesoft.net > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---- On Fri, 13 Mar 2020 15:58:20 +0000 > > se...@apache.org > > > > > wrote > > > > > > > > ---- > > > > > > > > > > > > > > > > > > > > > > > > > > Can we somehow keep this out of the TaskManager > > > services > > > > > > > > > > > > > I fear that we could not. IMO, the GPUManager(or > > > > > > > > > > > > > ExternalServicesManagers in future) is conceptually > > > one of > > > > > > the > > > > > > > > task > > > > > > > > > > > > > manager services, just like MemoryManager before > > 1.10. > > > > > > > > > > > > > - It maintains/holds the GPU resource at TM level > and > > > all > > > > > of > > > > > > > the > > > > > > > > > > > > > operators allocate the GPU resources from it. So, > it > > > should > > > > > > be > > > > > > > > > > > > > exclusive to a single TaskExecutor. > > > > > > > > > > > > > - We could add a collection called > > > ExternalResourceManagers > > > > > > to > > > > > > > > hold > > > > > > > > > > > > > all managers of other external resources in the > > future. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you help me understand why this needs the > addition > > in > > > > > > > > > > > TaskMagerServices > > > > > > > > > > > > or in the RuntimeContext? > > > > > > > > > > > > Are you worried about the case when multiple Task > > > Executors > > > > > run > > > > > > > in > > > > > > > > > the > > > > > > > > > > > same > > > > > > > > > > > > JVM? That's not common, but wouldn't it actually be > > good > > > in > > > > > > that > > > > > > > > case > > > > > > > > > > to > > > > > > > > > > > > share the GPU Manager, given that the GPU is shared? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > --------------------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What parts need information about this? > > > > > > > > > > > > > In this FLIP, operators need the information. Thus, > > we > > > > > expose > > > > > > > GPU > > > > > > > > > > > > > information to the RuntimeContext/FunctionContext. > > The > > > slot > > > > > > > > profile > > > > > > > > > > is > > > > > > > > > > > > > not aware of GPU resources as GPU is TM level > > resource > > > now. > > > > > > > > > > > > > > > > > > > > > > > > > > > Can the GPU Manager be a "self contained" thing > > that > > > > > simply > > > > > > > > takes > > > > > > > > > > the > > > > > > > > > > > > > configuration, and then abstracts everything > > > internally? > > > > > > > > > > > > > Yes, we just pass the path/args of the discover > > script > > > and > > > > > > how > > > > > > > > many > > > > > > > > > > > > > GPUs per TM to it. It takes the responsibility to > get > > > the > > > > > GPU > > > > > > > > > > > > > information and expose them to the > > > > > > > RuntimeContext/FunctionContext > > > > > > > > > of > > > > > > > > > > > > > Operators. Meanwhile, we'd better not allow > operators > > > to > > > > > > > directly > > > > > > > > > > > > > access GPUManager, it should get what they want > from > > > > > Context. > > > > > > > We > > > > > > > > > > could > > > > > > > > > > > > > then decouple the interface/implementation of > > > GPUManager > > > > > and > > > > > > > > Public > > > > > > > > > > > > > API. > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 13, 2020 at 7:26 PM Stephan Ewen < > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds fine to initially start with GPU > specific > > > > > support > > > > > > > and > > > > > > > > > > think > > > > > > > > > > > > > about > > > > > > > > > > > > > > generalizing this once we better understand the > > > space. > > > > > > > > > > > > > > > > > > > > > > > > > > > > About the implementation suggested in FLIP-108: > > > > > > > > > > > > > > - Can we somehow keep this out of the TaskManager > > > > > services? > > > > > > > > > > Anything > > > > > > > > > > > we > > > > > > > > > > > > > > have to pull through all layers of the TM makes > the > > > TM > > > > > > > > components > > > > > > > > > > yet > > > > > > > > > > > > > more > > > > > > > > > > > > > > complex and harder to maintain. > > > > > > > > > > > > > > > > > > > > > > > > > > > > - What parts need information about this? > > > > > > > > > > > > > > -> do the slot profiles need information about > the > > > GPU? > > > > > > > > > > > > > > -> Can the GPU Manager be a "self contained" > thing > > > that > > > > > > > simply > > > > > > > > > > takes > > > > > > > > > > > > > > the configuration, and then abstracts everything > > > > > > internally? > > > > > > > > > > > Operators > > > > > > > > > > > > > can > > > > > > > > > > > > > > access it via "GPUManager.get()" or so? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 4:19 AM Yangze Guo < > > > > > > > karma...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedbacks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Becket > > > > > > > > > > > > > > > Regarding the WebUI and GPUInfo, you're right, > > > I'll add > > > > > > > them > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > > > > Public API section. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Stephan @Becket > > > > > > > > > > > > > > > Regarding the general extended resource > > mechanism, > > > I > > > > > > second > > > > > > > > > > > Xintong's > > > > > > > > > > > > > > > suggestion. > > > > > > > > > > > > > > > - It's better to leverage ResourceProfile and > > > > > > ResourceSpec > > > > > > > > > after > > > > > > > > > > we > > > > > > > > > > > > > > > supporting fine-grained GPU scheduling. As a > > first > > > step > > > > > > > > > > proposal, I > > > > > > > > > > > > > > > prefer to not include it in the scope of this > > FLIP. > > > > > > > > > > > > > > > - Regarding the "Extended Resource Manager", > if I > > > > > > > understand > > > > > > > > > > > > > > > correctly, it just a code refactoring atm, we > > could > > > > > > extract > > > > > > > > the > > > > > > > > > > > > > > > open/close/allocateExtendResources of > GPUManager > > to > > > > > that > > > > > > > > > > > interface. If > > > > > > > > > > > > > > > that is the case, +1 to do it during > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Xingbo > > > > > > > > > > > > > > > As Xintong said, we looked into how Spark > > supports > > > a > > > > > > > general > > > > > > > > > > > "Custom > > > > > > > > > > > > > > > Resource Scheduling" before and decided to > > > introduce a > > > > > > > common > > > > > > > > > > > resource > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > schema(taskmanager.resource.{resourceName}.amount/discovery-script) > > > > > > > > > > > > > > > to make it more extensible. I think the > > "resource" > > > is a > > > > > > > > proper > > > > > > > > > > > level > > > > > > > > > > > > > > > to contain all the configs of extended > resources. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 10:48 AM Xingbo Huang < > > > > > > > > > hxbks...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot for the FLIP, Yangze. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is no doubt that GPU resource > management > > > > > support > > > > > > > will > > > > > > > > > > > greatly > > > > > > > > > > > > > > > > facilitate the development of AI-related > > > applications > > > > > > by > > > > > > > > > > PyFlink > > > > > > > > > > > > > users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have only one comment about this wiki: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the names of several GPU > > > configurations, I > > > > > > > think > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > > better > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > delete the resource field makes it consistent > > > with > > > > > the > > > > > > > > names > > > > > > > > > of > > > > > > > > > > > other > > > > > > > > > > > > > > > > resource-related configurations in > > > TaskManagerOption. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > e.g. > > > taskmanager.resource.gpu.discovery-script.path > > > > > -> > > > > > > > > > > > > > > > > taskmanager.gpu.discovery-script.path > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xingbo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com> > > > 于2020年3月4日周三 > > > > > > > > 上午10:39写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Stephan, @Becket, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Actually, Yangze, Yang and I also had an > > > offline > > > > > > > > discussion > > > > > > > > > > > about > > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > the "GPU Support" as some general "Extended > > > > > Resource > > > > > > > > > > Support". > > > > > > > > > > > We > > > > > > > > > > > > > > > believe > > > > > > > > > > > > > > > > > supporting extended resources in a general > > > > > mechanism > > > > > > is > > > > > > > > > > > definitely > > > > > > > > > > > > > a > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > and extensible way. The reason we propose > > this > > > FLIP > > > > > > > > > narrowing > > > > > > > > > > > its > > > > > > > > > > > > > scope > > > > > > > > > > > > > > > > > down to GPU alone, is mainly for the > concern > > on > > > > > extra > > > > > > > > > efforts > > > > > > > > > > > and > > > > > > > > > > > > > > > review > > > > > > > > > > > > > > > > > capacity needed for a general mechanism. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To come up with a well design on a general > > > extended > > > > > > > > > resource > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > mechanism, we would need to investigate > more > > > on how > > > > > > > > people > > > > > > > > > > use > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > kind of resources in practice. For GPU, we > > > learnt > > > > > > such > > > > > > > > > > > knowledge > > > > > > > > > > > > > from > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > experts, Becket and his team members. But > for > > > FPGA, > > > > > > or > > > > > > > > > other > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > extended resources, we don't have such > > > convenient > > > > > > > > > information > > > > > > > > > > > > > sources, > > > > > > > > > > > > > > > > > making the investigation requires more > > efforts, > > > > > > which I > > > > > > > > > tend > > > > > > > > > > to > > > > > > > > > > > > > think > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > not necessary atm. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the other hand, we also looked into how > > > Spark > > > > > > > > supports a > > > > > > > > > > > general > > > > > > > > > > > > > > > "Custom > > > > > > > > > > > > > > > > > Resource Scheduling". Assuming we want to > > have > > > a > > > > > > > similar > > > > > > > > > > > general > > > > > > > > > > > > > > > extended > > > > > > > > > > > > > > > > > resource mechanism in the future, we > believe > > > that > > > > > the > > > > > > > > > current > > > > > > > > > > > GPU > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > design can be easily extended, in an > > > incremental > > > > > way > > > > > > > > > without > > > > > > > > > > > too > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > reworks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - The most important part is probably user > > > > > > interfaces. > > > > > > > > > Spark > > > > > > > > > > > > > offers > > > > > > > > > > > > > > > > > configuration options to define the amount, > > > > > discovery > > > > > > > > > script > > > > > > > > > > > and > > > > > > > > > > > > > > > vendor > > > > > > > > > > > > > > > > > (on > > > > > > > > > > > > > > > > > k8s) in a per resource type bias [1], which > > is > > > very > > > > > > > > similar > > > > > > > > > > to > > > > > > > > > > > > > what > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > proposed in this FLIP. I think it's not > > > necessary > > > > > to > > > > > > > > expose > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > options > > > > > > > > > > > > > > > > > in the general way atm, since we do not > have > > > > > supports > > > > > > > for > > > > > > > > > > other > > > > > > > > > > > > > > > resource > > > > > > > > > > > > > > > > > types now. If later we decided to have per > > > resource > > > > > > > type > > > > > > > > > > config > > > > > > > > > > > > > > > > > options, we > > > > > > > > > > > > > > > > > can have backwards compatibility on the > > current > > > > > > > proposed > > > > > > > > > > > options > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > simple key mapping. > > > > > > > > > > > > > > > > > - For the GPU Manager, if later needed we > can > > > > > change > > > > > > it > > > > > > > > to > > > > > > > > > a > > > > > > > > > > > > > > > "Extended > > > > > > > > > > > > > > > > > Resource Manager" (or whatever it is > called). > > > That > > > > > > > should > > > > > > > > > be > > > > > > > > > > a > > > > > > > > > > > > > pure > > > > > > > > > > > > > > > > > component-internal refactoring. > > > > > > > > > > > > > > > > > - For ResourceProfile and ResourceSpec, > there > > > are > > > > > > > already > > > > > > > > > > > > > fields for > > > > > > > > > > > > > > > > > general extended resource. We can of course > > > > > leverage > > > > > > > them > > > > > > > > > > when > > > > > > > > > > > > > > > > > supporting > > > > > > > > > > > > > > > > > fine grained GPU scheduling. That is also > not > > > in > > > > > the > > > > > > > > scope > > > > > > > > > of > > > > > > > > > > > > > this > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > step proposal, and would require FLIP-56 to > > be > > > > > > finished > > > > > > > > > > first. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To summary up, I agree with Becket that > have > > a > > > > > > separate > > > > > > > > > FLIP > > > > > > > > > > > for > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > general extended resource mechanism, and > keep > > > it in > > > > > > > mind > > > > > > > > > when > > > > > > > > > > > > > > > discussing > > > > > > > > > > > > > > > > > and implementing the current one. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://spark.apache.org/docs/3.0.0-preview/configuration.html#custom-resource-scheduling-and-configuration-overview > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 9:18 AM Becket Qin < > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That's a good point, Stephan. It makes > > total > > > > > sense > > > > > > to > > > > > > > > > > > generalize > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > resource management to support custom > > > resources. > > > > > > > Having > > > > > > > > > > that > > > > > > > > > > > > > allows > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > to add new resources by themselves. The > > > general > > > > > > > > resource > > > > > > > > > > > > > management > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > > involve two different aspects: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. The custom resource type definition. > It > > is > > > > > > > supported > > > > > > > > > by > > > > > > > > > > > the > > > > > > > > > > > > > > > extended > > > > > > > > > > > > > > > > > > resources in ResourceProfile and > > > ResourceSpec. > > > > > This > > > > > > > > will > > > > > > > > > > > likely > > > > > > > > > > > > > cover > > > > > > > > > > > > > > > > > > majority of the cases. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. The custom resource allocation logic, > > > i.e. how > > > > > > to > > > > > > > > > assign > > > > > > > > > > > the > > > > > > > > > > > > > > > resources > > > > > > > > > > > > > > > > > > to different tasks, operators, and so on. > > > This > > > > > may > > > > > > > > > require > > > > > > > > > > > two > > > > > > > > > > > > > > > levels / > > > > > > > > > > > > > > > > > > steps: > > > > > > > > > > > > > > > > > > a. Subtask level - make sure the subtasks > > > are put > > > > > > > into > > > > > > > > > > > > > suitable > > > > > > > > > > > > > > > > > slots. > > > > > > > > > > > > > > > > > > It is done by the global RM and is not > > > > > customizable > > > > > > > > right > > > > > > > > > > > now. > > > > > > > > > > > > > > > > > > b. Operator level - map the exact > resource > > > to the > > > > > > > > > operators > > > > > > > > > > > > > in > > > > > > > > > > > > > > > TM. > > > > > > > > > > > > > > > > > e.g. > > > > > > > > > > > > > > > > > > GPU 1 for operator A, GPU 2 for operator > B. > > > This > > > > > > step > > > > > > > > is > > > > > > > > > > > needed > > > > > > > > > > > > > > > assuming > > > > > > > > > > > > > > > > > > the global RM does not distinguish > > individual > > > > > > > resources > > > > > > > > > of > > > > > > > > > > > the > > > > > > > > > > > > > same > > > > > > > > > > > > > > > type. > > > > > > > > > > > > > > > > > > It is true for memory, but not for GPU. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The GPU manager is designed to do 2.b > here. > > > So it > > > > > > > > should > > > > > > > > > > > > > discover the > > > > > > > > > > > > > > > > > > physical GPU information and bind/match > > them > > > to > > > > > > each > > > > > > > > > > > operators. > > > > > > > > > > > > > > > Making > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > general will fill in the missing piece to > > > support > > > > > > > > custom > > > > > > > > > > > resource > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > definition. But I'd avoid calling it a > > > "External > > > > > > > > Resource > > > > > > > > > > > > > Manager" to > > > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > confusion with RM, maybe something like > > > "Operator > > > > > > > > > Resource > > > > > > > > > > > > > Assigner" > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > be more accurate. So for each resource > type > > > users > > > > > > can > > > > > > > > > have > > > > > > > > > > an > > > > > > > > > > > > > > > optional > > > > > > > > > > > > > > > > > > "Operator Resource Assigner" in the TM. > For > > > > > memory, > > > > > > > > users > > > > > > > > > > > don't > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > this, > > > > > > > > > > > > > > > > > > but for other extended resources, users > may > > > need > > > > > > > that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Personally I think a pluggable "Operator > > > Resource > > > > > > > > > Assigner" > > > > > > > > > > > is > > > > > > > > > > > > > > > achievable > > > > > > > > > > > > > > > > > > in this FLIP. But I am also OK with > having > > > that > > > > > in > > > > > > a > > > > > > > > > > separate > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > because > > > > > > > > > > > > > > > > > > the interface between the "Operator > > Resource > > > > > > > Assigner" > > > > > > > > > and > > > > > > > > > > > > > operator > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > > take a while to settle down if we want to > > > make it > > > > > > > > > generic. > > > > > > > > > > > But I > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > implementation should take this future > work > > > into > > > > > > > > > > > consideration so > > > > > > > > > > > > > > > that we > > > > > > > > > > > > > > > > > > don't need to break backwards > compatibility > > > once > > > > > we > > > > > > > > have > > > > > > > > > > > that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 12:27 AM Stephan > > Ewen > > > < > > > > > > > > > > > se...@apache.org> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for writing this FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I cannot really give much input into > the > > > > > > mechanics > > > > > > > of > > > > > > > > > > > GPU-aware > > > > > > > > > > > > > > > > > > scheduling > > > > > > > > > > > > > > > > > > > and GPU allocation, as I have no > > experience > > > > > with > > > > > > > > that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One thought I had when reading the > > > proposal is > > > > > if > > > > > > > it > > > > > > > > > > makes > > > > > > > > > > > > > sense to > > > > > > > > > > > > > > > > > look > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > the "GPU Manager" as an "External > > Resource > > > > > > > Manager", > > > > > > > > > and > > > > > > > > > > > GPU > > > > > > > > > > > > > is one > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > resource. > > > > > > > > > > > > > > > > > > > The way I understand the > ResourceProfile > > > and > > > > > > > > > > ResourceSpec, > > > > > > > > > > > > > that is > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > is done there. > > > > > > > > > > > > > > > > > > > It has the advantage that it looks more > > > > > > extensible. > > > > > > > > > Maybe > > > > > > > > > > > > > there is > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > GPU > > > > > > > > > > > > > > > > > > > Resource, a specialized NVIDIA GPU > > > Resource, > > > > > and > > > > > > > FPGA > > > > > > > > > > > > > Resource, a > > > > > > > > > > > > > > > > > Alibaba > > > > > > > > > > > > > > > > > > > TPU Resource, etc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 3, 2020 at 7:57 AM Becket > > Qin < > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the FLIP Yangze. GPU > > resource > > > > > > > management > > > > > > > > > > > support > > > > > > > > > > > > > is a > > > > > > > > > > > > > > > > > > > must-have > > > > > > > > > > > > > > > > > > > > for machine learning use cases. > > Actually > > > it > > > > > is > > > > > > > one > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > > mostly > > > > > > > > > > > > > > > > > asked > > > > > > > > > > > > > > > > > > > > question from the users who are > > > interested in > > > > > > > using > > > > > > > > > > Flink > > > > > > > > > > > > > for ML. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some quick comments / questions to > the > > > wiki. > > > > > > > > > > > > > > > > > > > > 1. The WebUI / REST API should > probably > > > also > > > > > be > > > > > > > > > > > mentioned in > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > public > > > > > > > > > > > > > > > > > > > > interface section. > > > > > > > > > > > > > > > > > > > > 2. Is the data structure that holds > GPU > > > info > > > > > > > also a > > > > > > > > > > > public > > > > > > > > > > > > > API? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 3, 2020 at 10:15 AM > Xintong > > > Song > > > > > < > > > > > > > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for drafting the FLIP and > > > kicking > > > > > off > > > > > > > the > > > > > > > > > > > > > discussion, > > > > > > > > > > > > > > > > > Yangze. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Big +1 for this feature. Supporting > > > using > > > > > of > > > > > > > GPU > > > > > > > > in > > > > > > > > > > > Flink > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > significant, > > > > > > > > > > > > > > > > > > > > > especially for the ML scenarios. > > > > > > > > > > > > > > > > > > > > > I've reviewed the FLIP wiki doc and > > it > > > > > looks > > > > > > > good > > > > > > > > > to > > > > > > > > > > > me. I > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > it's a > > > > > > > > > > > > > > > > > > > > > very good first step for Flink's > GPU > > > > > > supports. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 2, 2020 at 12:06 PM > > Yangze > > > Guo > > > > > < > > > > > > > > > > > > > karma...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We would like to start a > discussion > > > > > thread > > > > > > on > > > > > > > > > > > "FLIP-108: > > > > > > > > > > > > > Add > > > > > > > > > > > > > > > GPU > > > > > > > > > > > > > > > > > > > > > > support in Flink"[1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This FLIP mainly discusses the > > > following > > > > > > > > issues: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Enable user to configure how > many > > > GPUs > > > > > > in a > > > > > > > > > task > > > > > > > > > > > > > executor > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > forward such requirements to the > > > external > > > > > > > > > resource > > > > > > > > > > > > > managers > > > > > > > > > > > > > > > (for > > > > > > > > > > > > > > > > > > > > > > Kubernetes/Yarn/Mesos setups). > > > > > > > > > > > > > > > > > > > > > > - Provide information of > available > > > GPU > > > > > > > > resources > > > > > > > > > to > > > > > > > > > > > > > > > operators. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Key changes proposed in the FLIP > > are > > > as > > > > > > > > follows: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Forward GPU resource > requirements > > > to > > > > > > > > > > > Yarn/Kubernetes. > > > > > > > > > > > > > > > > > > > > > > - Introduce GPUManager as one of > > the > > > task > > > > > > > > manager > > > > > > > > > > > > > services to > > > > > > > > > > > > > > > > > > > discover > > > > > > > > > > > > > > > > > > > > > > and expose GPU resource > information > > > to > > > > > the > > > > > > > > > context > > > > > > > > > > of > > > > > > > > > > > > > > > functions. > > > > > > > > > > > > > > > > > > > > > > - Introduce the default script > for > > > GPU > > > > > > > > discovery, > > > > > > > > > > in > > > > > > > > > > > > > which we > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > the privilege mode to help user > to > > > > > achieve > > > > > > > > > > > worker-level > > > > > > > > > > > > > > > isolation > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > standalone mode. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please find more details in the > > FLIP > > > wiki > > > > > > > > > document > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > Looking > > > > > > > > > > > > > > > > > > > forward > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > your feedbacks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
