Hi all, Regarding Plasma, you're right we should have started this conversation earlier! The way it's being developed in Ray currently isn't useful as a standalone project. We realized that tighter integration with Ray's object lifetime tracking could be important, and removing IPCs and making it a separate thread in the same process as our scheduler could make a big difference for performance. Some of these optimizations wouldn't be easy without a tight integration, so there are some trade-offs here.
Regarding the Python serialization format, I agree with Antoine that it should be deprecated. We began developing it before pickle 5, but now that pickle 5 has taken off, it makes less sense (it's useful in its own right, but at the end of the day, we were interested in it as a way to serialize arbitrary Python objects). -Robert On Sun, Jul 12, 2020 at 5:26 PM Wes McKinney <wesmck...@gmail.com> wrote: > I'll add deprecation warnings to the pyarrow.serialize functions in > question, it will be pretty simple. > > On Sun, Jul 12, 2020, 6:34 PM Neal Richardson <neal.p.richard...@gmail.com > > > wrote: > > > This seems like something to investigate after the 1.0 release. > > > > Neal > > > > On Sun, Jul 12, 2020 at 11:53 AM Antoine Pitrou <anto...@python.org> > > wrote: > > > > > > > > I'd certainly like to deprecate our custom Python serialization format, > > > and using pickle protocol 5 instead is a very good idea. > > > > > > We can probably keep it in 1.0 while raising a FutureWarning. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > Le 12/07/2020 à 19:22, Wes McKinney a écrit : > > > > It appears that the Ray developers have decided to fork Plasma and > > > > decouple from the Arrow codebase: > > > > > > > > https://github.com/ray-project/ray/pull/9154 > > > > > > > > This is a disappointing development to occur without any discussion > on > > > > this mailing list but given the lack of development activity on > Plasma > > > > I would like to see how others in the community would like to > proceed. > > > > > > > > It appears additionally that the Union-based serialization format > > > > implemented by arrow/python/serialize.h and the pyarrow/serialize.py > > > > has been dropped in favor of pickle5. If there is not value in > > > > maintaining this code then it would probably be preferable for us to > > > > remove this from the codebase. > > > > > > > > Thanks, > > > > Wes > > > > > > > > > >
