UCX is interesting, relatively new and seems like it may be easier to
integrate. MPI is the most commonly used backend for HPC. Influencing
the development of UCX is more difficult than influencing the
development of MPI, but both have a slower pace of development than
Arrow. One may want to consider support for multiple accelerators, Arrow
has CUDA support but SYCL support seems like it will fit well with a C++
base for compute unit/node level parallelism and then UCX and/or MPI
support for multinode parallelism.
OLCF does support RAPIDS
https://docs.olcf.ornl.gov/software/analytics/nvidia-rapids.html so HPC
in the commercial cloud could also make of Arrow.
On 10/27/21 5:26 AM, Keith Kraus wrote:
Outside of just HPC, integrating UCX would potentially allow taking
advantage of its shared memory backend which would be interesting from a
performance perspective in the single-node, multi-process case in many
situations.
Not sure it's worth the UCX dependency in the long run, but would allow us
to experiment with a lot of different transport backends.
On Tue, Oct 26, 2021 at 10:10 PM Yibo Cai <yibo....@arm.com> wrote:
On 10/26/21 10:02 PM, David Li wrote:
Hi Yibo,
Just curious, has there been more thought on this from your/the HPC side?
Yes. I will investigate the possible approach. Maybe build a quick (and
dirty) POC test at first.
I also realized we never asked, what is motivating Flight in this space
in the first place? Presumably broader Arrow support in general?
No special reason. Will be great if comes up with something useful, or
an interesting experiment otherwise.
-David
On Fri, Sep 10, 2021, at 12:27, Micah Kornfield wrote:
I would support doing the work necessary to get UCX (or really any
other
transport) supported, even if it is a lot of work. (I'm hoping this
clears
the path to supporting a Flight-to-browser transport as well; a few
projects seem to have rolled their own approaches but I think Flight
itself
should really handle this, too.)
Another possible technical approach is investigating to see if coming up
with a custom gRPC "channel" implementation for new transports .
Searching around it seems like there were some defunct PRs trying to
enable UCX as one, I didn't look closely enough at why they might have
failed.
On Thu, Sep 9, 2021 at 11:07 AM David Li <lidav...@apache.org> wrote:
I would support doing the work necessary to get UCX (or really any
other
transport) supported, even if it is a lot of work. (I'm hoping this
clears
the path to supporting a Flight-to-browser transport as well; a few
projects seem to have rolled their own approaches but I think Flight
itself
should really handle this, too.)
From what I understand, you could tunnel gRPC over UCX as Keith
mentions,
or directly use UCX, which is what it sounds like you are thinking
about.
One idea we had previously was to stick to gRPC for 'control plane'
methods, and support alternate protocols only for 'data plane' methods
like
DoGet - this might be more manageable, depending on what you have in
mind.
In general - there's quite a bit of work here, so it would help to
separate the work into phases, and share some more detailed
design/implementation plans, to make review more manageable. (I
realize of
course this is just a general interest check right now.) Just splitting
gRPC/Flight is going to take a decent amount of work, and (from what
little
I understand) using UCX means choosing from various communication
methods
it offers and writing a decent amount of scaffolding code, so it would
be
good to establish what exactly a 'UCX' transport means. (For instance,
presumably there's no need to stick to the Protobuf-based wire format,
but
what format would we use?)
It would also be good to expand the benchmarks, to validate the
performance we get from UCX and have a way to compare it against gRPC.
Anecdotally I've found gRPC isn't quite able to saturate a connection
so it
would be interesting to see what other transports can do.
Jed - how would you see MPI and Flight interacting? As another
transport/alternative to UCX? I admit I'm not familiar with the HPC
space.
About transferring commands with data: Flight already has an
app_metadata
field in various places to allow things like this, it may be
interesting to
combine with the ComputeIR proposal on this mailing list, and
hopefully you
& your colleagues can take a look there as well.
-David
On Thu, Sep 9, 2021, at 11:24, Jed Brown wrote:
Yibo Cai <yibo....@arm.com> writes:
HPC infrastructure normally leverages RDMA for fast data transfer
among
storage nodes and compute nodes. Computation tasks are dispatched to
compute nodes with best fit resources.
Concretely, we are investigating porting UCX as Flight transport
layer.
UCX is a communication framework for modern networks. [1]
Besides HPC usage, many projects (spark, dask, blazingsql, etc) also
adopt UCX to accelerate network transmission. [2][3]
I'm interested in this topic and think it's important that even if the
focus is direct to UCX, that there be some thought into MPI
interoperability and support for scalable collectives. MPI considers
UCX to
be an implementation detail, but the two main implementations (MPICH
and
Open MPI) support it and vendor implementations are all derived from
these
two.
