Thanks for all the replies. Unfortunately, as I found out today, including the `.a` (static libraries) in the Github just got to its limit (100Mb), so I don't think this is an option anymore :( I'm going to investigate the Gihtub LFS (Large File Storage), not sure how well it will integrate with `go get` though -- I've never used it before.
Interesting to know that an empty Go package file is no longer needed to fetch a directory with binary dependencies (handy also for go:embed in cases where the binary files are on a separate directory). Still for my Go library with a `.so` dependency I'm not sure yet what is the answer. I'm thinking about creating another tool that goes over all the dependencies of module, and automatically downloads release files to a configured directory ... so at least all one needs to do is execute one command and get all dependencies of all libraries installed. Any thoughts ? @Mike Schinkel: the go:embed is used in a different context (when embedding blobs of binary). We need prebuilt binary libraries that will be linked with the Go code during compilation. Preferably something that will transparently work with `go get` / `go build` / `go install`. cheers On Tuesday, October 17, 2023 at 8:46:01 PM UTC+2 Justin Israel wrote: > On Wednesday, October 18, 2023 at 3:12:29 AM UTC+13 Marc-Antoine Ruel > wrote: > > Sorry I wasn't clear. The static libraries are in a subdirectory because > the user should not care and these libraries are effectively third party > code. > > > Actually, you were totally clear. Sorry if my comment didn't make sense. I > get that the user doesn't need to worry about the third_party directory > with the header and pre-built static libraries. And I saw how you arrange > to pick up the right ones in the code. My point was that from previous > experience, "go get" would prune away directories that do not contain go > source code, after it performs the clone from the remote, into the module > cache. So I would have expected that when someone uses your library as a > dependency, it would have omitted the needed third_party directory, for not > having any Go source files in it. But when I tested it just now, it does > seem to retain the subdirectory and link correctly. Maybe this was improved > as of recent Go releases. > In earlier versions of the Go tool, I would have either included a dummy > Go source file in the subdirectory, or just kept the non-go files in the > root with the rest of the Go source. > Maybe someone else can clarify if this workflow has been improved? > > > This declares the generic code available everywhere: > https://github.com/periph/d2xx/blob/main/d2xx.go > > One set of files declare the import that is OS and CPU arch appropriate, > e.g. > https://github.com/periph/d2xx/blob/main/d2xx_linux_amd64.go > > Then another set of two files defines the OS specific code. In practice, > POSIX and windows; > https://github.com/periph/d2xx/blob/main/d2xx_posix.go > https://github.com/periph/d2xx/blob/main/d2xx_windows.go > > A CGO_ENABLED=0 build gets instead redirected to > https://github.com/periph/d2xx/blob/main/d2xx_posix_no_cgo.go > but not on windows because this package uses dynamic linking on windows. > Adapt as appropriate in your use case. > > I hope this helps. > > M-A > > > Le lun. 16 oct. 2023, 21 h 13, Justin Israel <justin...@gmail.com> a > écrit : > > > > On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote: > > I second Richard's suggestion. I used the same trick for > https://github.com/periph/d2xx. This git repository contains a minimalist > shim for the .a libraries and nothing else. It's designed to compile on any > platform. It falls back to a scaffolding if the corresponding .a library is > not present for the OS-arch combination or if CGO is disabled. It makes > testing easier. > > Then a proper package under https://github.com/periph/host/tree/main/ftdi > exposes a higher level abstraction for the user. > > > With only headers and static libs in the thirdparty directory, is this > package "go-gettable"? > Will go make the subdirectory available in that case? It usually ignores > if there is no Go source files. > > > M-A > > Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel <mi...@newclarity.net> a > écrit : > > Hi Jan, > > I'm going to go out on a limb here and suggest looking at using Go's > `embed` feature? > > https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/ > > I have not tried it with C libraries so there may be numerous reasons why > it may not work for your needs. For example, I do not know what is required > to *"install"* the libraries so that might be your sticking point that > embed would not address. But if building on the user's machine is the real > problem and you can distribute pre-build binaries then this might work for > you. > > Basically Go reads a `//go:embed` comment, converts your files into Go > source code, compiles that code, and then provides a package that allows > you to write those files to disk from an your Go app before you need to > load the libraries. > > Maybe this will work for you? If yes, would love to hear back how it > worked out. > > -Mike > On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote: > > I guess you switched things here. Shared libraries (.so) need to be > available at engine. Static libraries (.a) are bagged into the binary. > > -Bruno > > > On Sun, Oct 15, 2023, 3:55 PM Jan <pfe...@gmail.com> wrote: > > Thanks Tamas, this is useful information. > > One of my libraries is using a `.a` library -- as opposed to `.so`, which > is another level of complexity, since the library has to be available in > runtime, not only in compile time -- and I'm going to follow your > "incantation" suggestion. > > > On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote: > > Neither I see a convenient way. > BUT if you add a .go file into the directories where your precompiled > libraries live, > then "go get" will download them too (and only those dirs that have .go > files in it). > > So your next mission is to prepare the right #cgo CFLAGS LDFLAGS > incantations to use those libraries. > > Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2): > > Thanks Tamás, I may not be understanding correctly, but after taking a > look at github.com/godror/godror, and the odpi subdirectory, > I see it is including all the `.c` files on the fly > <https://github.com/godror/godror/blob/main/odpi/embed/dpi.c>. > > A couple of reasons immediately come to mind, that make this not a > generically valid option: > > * ODPI library is all C code (see src > <https://github.com/godror/godror/tree/main/odpi/src>) so it works > including in Go: my dependencies are C++/Rust code, for which I write a > small C wrapper (or for Rust just `extern "C"`). Also architecture > dependent compilation is different in C++/C/Rust code ... > * The C++ libraries I'm including have sub-dependencies of themselves (one > of which is llvm). It uses Bazel to organize it, and to manually move all > the required C++ files to a directory would be years of work :) Plus would > require me to slave to maintain things in sync. > * The C++ libraries take hours to compile ... I don't want to impose this > to users of my libraries. > > I think the only way to work this out is distributing the pre-compiled > C++/Rust libraries, so the Go simply refer to them (and we get the fast > compilation times from Go). But then, how to best distribute them in an > automatic fashion, so that users don't need to one by one figure out how to > install them (and clean up them later if they are no longer using...) ? > > Or maybe there is another convenient way I'm not seeing ? > > > On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote: > > Can't you build (make go build for you) those libraries? > For example, see github.com/godror/godror just includes the sources of > that third party library in an "odpi" subdir, and with > ``` > /* > #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed > > #include "dpi.c" > > */ > import "C" > ``` > it is compiled automatically. > > > Caveat: for "go get" to download a directory, it must include a sth.go > file ("require.go" in the odpi/* subdirs). > But it may work that your precompiled libs in a subdir, with a mock .go > file gets downloaded. > But how will it found by your lib/app ? > > Tamás > > > Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2): > > Thanks Richard. Indeed, as you pointed out the downside is the bloating of > the git repo, but it makes sense. > > But does the user have to manually clone the repository and move the `.a` > file to, let's say, /usr/local/lib, or does a simple `go get` magically > does everything ? > > > On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote: > > It isn't a great solution, but I currently include the built library files > and necessary headers in the git repo alongside the Go code. You can see an > example here > https://github.com/richardwilkes/unison/tree/main/internal/skia where I > include the skia library I built for use in my UI framework, unison. > > The main downside of this is bloating the git repo with the binary .a and > .dll files... but I've not found a better way to handle it. glfw, which > unison also depends upon, does something very similar. > > - Rich > > On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote: > > hi, > > I'm developing a couple of ML framework/libraries for Go that depend on > C/C++ code. Once C-libraries dependencies are installed, the CGO > integration work great. > > Now, for end-users that just want to use these Go libraries, having to > figure out how to manually build and install those C/C++/Rust dependencies > is a hassle -- sadly each one with a different type of build system. > > I offer pre-built `.tgz` files (for a limited set of architectures) with > the required `.h` and `.a/.so` files along the releases, which facilitates. > But it's still a hassle to install -- and no auto-uninstall if someone is > no longer using the Go module. > > I was wondering if others have figured out how to handle this in a nicer > way. Is there a recommended way to distribute prebuilt CGO dependencies ? > > I like how Python wheels (`.whl` files) solve the issue, by including the > pre-built libraries in a sub-directory of the python library installation. > I was hoping there would be a similar way (maybe with a separate tool) to > integrate with `go.mod`. > > Any pointers, thoughts or suggestions are very appreciated. > > Thanks! > Jan > > ps: While searching I saw similar questions, but none that exactly > answered this aspect of distribution. Just in case, apologies if it's a > duplicate question. > > -- > > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/a00b3a7b-200d-44ce-8060-913d9ac3c6cen%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/a00b3a7b-200d-44ce-8060-913d9ac3c6cen%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/ca2d8979-f741-4b81-9099-8672bc5e2312n%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/ca2d8979-f741-4b81-9099-8672bc5e2312n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/146f101d-8ab4-40ec-a03d-ea23aa84bf74n%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/146f101d-8ab4-40ec-a03d-ea23aa84bf74n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9e56ac27-386d-41d0-a520-263e3402c6d2n%40googlegroups.com.
