Thanks for the feedback on the benchmark. By switching from Unix domain socket to TCP and reducing the batch size to under 5MB I was able to get nearly 5Gbps throughput. I think Unix domain sockets are just slower on Macs. Updated that repo [1]
[1] https://github.com/wjones127/arrow-ipc-bench/tree/main On Fri, Mar 17, 2023 at 9:16 AM Antoine Pitrou <anto...@python.org> wrote: > > Le 17/03/2023 à 16:34, Alessandro Molina a écrit : > > How does PyArrow cope with multiprocessing.Manager? > > I'm not sure anyone tried it. Also, I don't think > multiprocessing.Manager was updated to use pickle v5 out-of-band buffers > (which would help reduce copying), so I wouldn't expect very high > performance. > > Generally, I don't think multiprocessing.Manager is very much used these > days. It also doesn't receive a lot of maintenance. > >
