UCX allow the user to use various transportation protocols without having to be aware of which protocol is being used under the hood. The image at the top of this page gives a good explanation of how this happens https://openucx.org/introduction/ . So the application can be agnostic if the transport is happening over the network interface card, pcie, or RDMA.
On 2021/09/09 11:48:16, Antoine Pitrou <anto...@python.org> wrote: > > Le 09/09/2021 à 12:34, Yibo Cai a écrit : > > Hi, > > > > We have some rough ideas of applying Flight in HPC (High Performance > > Computation). Would like to hear comments. > > > > HPC infrastructure normally leverages RDMA for fast data transfer among > > storage nodes and compute nodes. Computation tasks are dispatched to > > compute nodes with best fit resources. > > > > Concretely, we are investigating porting UCX as Flight transport layer. > > UCX is a communication framework for modern networks. [1] > > Besides HPC usage, many projects (spark, dask, blazingsql, etc) also > > adopt UCX to accelerate network transmission. [2][3] > > > > I see a recent discussion about decoupling Flight from gRPC. Looks this > > is also what we should do first to adapt UCX to Flight. > > Is it actually necessary? Or is it possible to transmit HTTP/2 over UCX? > > (I'm not fond of gRPC, but decoupling Flight *and* building support for > another transport layer will be quite a bit of work) > > Regards > > Antoine. >
