On Thu, Nov 23, 2023 at 11:52 AM Sebastien Binet <bi...@cern.ch> wrote:
> On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote: > > Also, I'd like to clarify: > > > > - [David]: is it a good idea to use cgo for Go-Python interop? > > - [Michael]: no. better with pipe or RPC > > > > I'm wrong about this. A conversation after the meeting clarified a > > misunderstanding I had about Go->Python calls specifically. Both > > Go->Python > > and Python->Go with cgo may very well be preferable to pipe/RPC in many > > circumstances. :) > > I think that, at least for the ML/AI use cases (where the API is > python-based), tackling this via the cgo dance will be cumbersome: > > to access the data or call the methods/functions, one will need to go > through the C.PyXYZ calls (even though most of the ML/AI modules are > implemented in C for performances) > > this means a world of C.PyIncref/Decref, C-callbacks, ... > (having done something along these lines with go-python/gopy (a set of > tools to automatically expose a Go package as a CPython module), I speak > from "some" experience) > > so, I'd tend to agree with your previous self :) > what did change your mind ? > Sure, I recognize that it's still cumbersome and it may be better to use some language-agnostic protocol to communicate in some cases. I don't have very much experience with this specific boundary, so maybe I'm now wrong again for a different reason. :) My misunderstanding was regarding a concern about concurrent Go->Python calls locking up the Python interpreter. I've come to understand that this generally isn't a problem in practice (or no worse than Python execution already mostly being serialized). > > -s > > > On Thursday, November 23, 2023 at 3:40:35 AM UTC-5 Sebastien Binet > > wrote: > > > > > On Thu Nov 23, 2023 at 01:24 CET, Eli Bendersky wrote: > > > > On Wed, Nov 22, 2023 at 11:31 AM Sebastien Binet > > > > <sebasti...@cern.ch> > > > > wrote: > > > > > > > > > Hi there, > > > > > > > > > > In this week "compiler minutes" [1], one can read: > > > > > > > > > > """ > > > > > - Go on future platforms (RAM efficiency. NUMA?) > > > > > - (maybe) Go-Python interop for AI-powered applications > > > > > - [David]: is it a good idea to use cgo for Go-Python interop? > > > > > - [Michael]: no. better with pipe or RPC > > > > > """ > > > > > > > > > > Would it be possible to have a bit more informations ? > > > > > What kind of interop is it ? Exchanging binary data ? On disk ? > > > > > Establishing a protobuf-based-like standard ? > > > > > Something else ? > > > > > > > > > > > > > All of it, maybe :-) > > > > We're just exploring the issue, throwing ideas around. There are many > > > > potential options, each with its own tradeoffs in terms of > performance > > > > vs. effort. > > > > > > depending on the timescale, one could also imagine having a Go-based > > > python VM (e.g. github.com/go-python/gpython) that can run a limited > set > > > of python modules (even the C-based ones, like pypy did at some point > ?). > > > alternatively, "just" rely on pickle-based, Apache Arrow-based or > numpy > > > array-based exchanged data. > > > > > > for Arrow and numpy, there are already packages that do offer a fair > > > amount of interop: > > > > > > - https://github.com/apache/arrow/tree/main/go > > > - https://github.com/sbinet/npyio/ (shameless plug.) > > > > > > (we could imagine also adding some buffer protocol implementation à la > > > PEP-3118) > > > > > > for pickle, gpython has support up to protocol=3, and gopickle seems > to > > > have support for up to 5: > > > - https://github.com/nlpodyssey/gopickle > > > (adding support for array.array to gopickle was relatively > straightforward) > > > > > > -s > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to golang-nuts+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/golang-nuts/cf461932-52bb-4ac1-9690-053bec7e432an%40googlegroups.com > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAFza%2Bu9-%3DeCaQjEQJ5qj5QijRP7TVhXEqEvrm%2BLuRdqDAx-Ftw%40mail.gmail.com.
