> > Regarding Plasma, you're right we should have started this conversation > earlier! The way it's being developed in Ray currently isn't useful as a > standalone project. We realized that tighter integration with Ray's object > lifetime tracking could be important, and removing IPCs and making it a > separate thread in the same process as our scheduler could make a big > difference for performance. Some of these optimizations wouldn't be easy > without a tight integration, so there are some trade-offs here.
So I guess the question is whether it is worth continuing to try to maintain a sepearate version of plasma within the Arrow repo? On Tue, Jul 21, 2020 at 9:28 AM Robert Nishihara <robertnishih...@gmail.com> wrote: > Hi all, > > Regarding Plasma, you're right we should have started this conversation > earlier! The way it's being developed in Ray currently isn't useful as a > standalone project. We realized that tighter integration with Ray's object > lifetime tracking could be important, and removing IPCs and making it a > separate thread in the same process as our scheduler could make a big > difference for performance. Some of these optimizations wouldn't be easy > without a tight integration, so there are some trade-offs here. > > Regarding the Python serialization format, I agree with Antoine that it > should be deprecated. We began developing it before pickle 5, but now that > pickle 5 has taken off, it makes less sense (it's useful in its own right, > but at the end of the day, we were interested in it as a way to serialize > arbitrary Python objects). > > -Robert > > On Sun, Jul 12, 2020 at 5:26 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > I'll add deprecation warnings to the pyarrow.serialize functions in > > question, it will be pretty simple. > > > > On Sun, Jul 12, 2020, 6:34 PM Neal Richardson < > neal.p.richard...@gmail.com > > > > > wrote: > > > > > This seems like something to investigate after the 1.0 release. > > > > > > Neal > > > > > > On Sun, Jul 12, 2020 at 11:53 AM Antoine Pitrou <anto...@python.org> > > > wrote: > > > > > > > > > > > I'd certainly like to deprecate our custom Python serialization > format, > > > > and using pickle protocol 5 instead is a very good idea. > > > > > > > > We can probably keep it in 1.0 while raising a FutureWarning. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > Le 12/07/2020 à 19:22, Wes McKinney a écrit : > > > > > It appears that the Ray developers have decided to fork Plasma and > > > > > decouple from the Arrow codebase: > > > > > > > > > > https://github.com/ray-project/ray/pull/9154 > > > > > > > > > > This is a disappointing development to occur without any discussion > > on > > > > > this mailing list but given the lack of development activity on > > Plasma > > > > > I would like to see how others in the community would like to > > proceed. > > > > > > > > > > It appears additionally that the Union-based serialization format > > > > > implemented by arrow/python/serialize.h and the > pyarrow/serialize.py > > > > > has been dropped in favor of pickle5. If there is not value in > > > > > maintaining this code then it would probably be preferable for us > to > > > > > remove this from the codebase. > > > > > > > > > > Thanks, > > > > > Wes > > > > > > > > > > > > > > >
