I can help, we use emscripten-compiled arrow for perspective ( https://github.com/finos/perspective) and we now compile perspective's python side for pyodide so I have an interest in a fully functional pyarrow/pandas in pyodide on an ongoing basis.
Tim Paine tim.paine.nyc 908-721-1185 > On Jul 6, 2023, at 11:14, Antoine Pitrou <anto...@python.org> wrote: > > > Hi Joe, > > Thank you for working on that. > > The one question I have is: are you willing to help us maintain Arrow C++ on > the long term? The logic you're adding in > https://github.com/apache/arrow/pull/35672 is quite delicate; also I don't > think anyone among us is a Webassembly expert, which means that we might > break things unwillingly. So while it would be great to get Arrow C++ to work > with WASM, a dedicated expert is needed to help maintain and debug WASM > support in the future. > > Regards > > Antoine. > > >> Le 03/07/2023 à 17:29, Joe Marshall a écrit : >> Hi, >> I'm a pyodide developer amongst other things (webassembly cpython >> intepreter) and I've got some PRs in progress on arrow relating to >> webassembly support. I wondered if it might be worth discussing my broader >> ideas for this on the list or at the biweekly development meeting? >> So far I have 35176 in, which makes arrow run on a single thread. This is >> needed because in a lot of webassembly environments (browsers at least, >> pyodide), threading isn't available or is heavily constrained. >> With that I've aimed to make it relatively transparent to users, so that >> things like datasets and acero mostly just work (but slower obviously). It's >> kind of fiddly in the arrow code but working, and means users can port >> things easily. >> Once that is in, the plan is to submit a following pr that adds cmake >> presets for emscripten which can build the cpp libraries and pyarrow for >> pyodide. I've hacked this together in a build already, it's a bit fiddly and >> needs a load of tidying up, but I'm confident it can be done. >> Essentially, I'm wanting to get this stuff in because pandas is moving >> towards arrow as a pretty much required dependency, and webassembly is a >> pandas platform, as well as >> being an official python platform, so it would be great to get it working >> in pyodide without us needing to maintain a load of patches. I guess it >> could also come in handy with various container platforms that are moving to >> webassembly. >> Basically I thought it's probably worth a bit of a heads up relating to >> this, as I know the bigger picture of things is often hard to see from just >> pull requests. >> Thanks >> Joe >> This message and any attachment are intended solely for the addressee >> and may contain confidential information. If you have received this >> message in error, please contact the sender and delete the email and >> attachment. >> Any views or opinions expressed by the author of this email do not >> necessarily reflect the views of the University of Nottingham. Email >> communications with the University of Nottingham may be monitored >> where permitted by law.
