On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote: > Also, I'd like to clarify: > > - [David]: is it a good idea to use cgo for Go-Python interop? > - [Michael]: no. better with pipe or RPC > > I'm wrong about this. A conversation after the meeting clarified a > misunderstanding I had about Go->Python calls specifically. Both > Go->Python > and Python->Go with cgo may very well be preferable to pipe/RPC in many > circumstances. :)
I think that, at least for the ML/AI use cases (where the API is python-based), tackling this via the cgo dance will be cumbersome: to access the data or call the methods/functions, one will need to go through the C.PyXYZ calls (even though most of the ML/AI modules are implemented in C for performances) this means a world of C.PyIncref/Decref, C-callbacks, ... (having done something along these lines with go-python/gopy (a set of tools to automatically expose a Go package as a CPython module), I speak from "some" experience) so, I'd tend to agree with your previous self :) what did change your mind ? -s > On Thursday, November 23, 2023 at 3:40:35 AM UTC-5 Sebastien Binet > wrote: > > > On Thu Nov 23, 2023 at 01:24 CET, Eli Bendersky wrote: > > > On Wed, Nov 22, 2023 at 11:31 AM Sebastien Binet > > > <sebasti...@cern.ch> > > > wrote: > > > > > > > Hi there, > > > > > > > > In this week "compiler minutes" [1], one can read: > > > > > > > > """ > > > > - Go on future platforms (RAM efficiency. NUMA?) > > > > - (maybe) Go-Python interop for AI-powered applications > > > > - [David]: is it a good idea to use cgo for Go-Python interop? > > > > - [Michael]: no. better with pipe or RPC > > > > """ > > > > > > > > Would it be possible to have a bit more informations ? > > > > What kind of interop is it ? Exchanging binary data ? On disk ? > > > > Establishing a protobuf-based-like standard ? > > > > Something else ? > > > > > > > > > > All of it, maybe :-) > > > We're just exploring the issue, throwing ideas around. There are many > > > potential options, each with its own tradeoffs in terms of performance > > > vs. effort. > > > > depending on the timescale, one could also imagine having a Go-based > > python VM (e.g. github.com/go-python/gpython) that can run a limited set > > of python modules (even the C-based ones, like pypy did at some point ?). > > alternatively, "just" rely on pickle-based, Apache Arrow-based or numpy > > array-based exchanged data. > > > > for Arrow and numpy, there are already packages that do offer a fair > > amount of interop: > > > > - https://github.com/apache/arrow/tree/main/go > > - https://github.com/sbinet/npyio/ (shameless plug.) > > > > (we could imagine also adding some buffer protocol implementation à la > > PEP-3118) > > > > for pickle, gpython has support up to protocol=3, and gopickle seems to > > have support for up to 5: > > - https://github.com/nlpodyssey/gopickle > > (adding support for array.array to gopickle was relatively straightforward) > > > > -s > > > > -- > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/cf461932-52bb-4ac1-9690-053bec7e432an%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CX6CITBAJ3LS.2HFXLFQCV58PQ%40cern.ch.
