On Tue, Jan 19, 2021 at 7:02 AM Jay Conrod <jaycon...@google.com> wrote: > > > Interesting - is the difference the absolute paths vs relative? > > It looks like the bug has to do with whether the directory is below the main module root directory or not. If it is, the go command takes a path that assumes it's part of the main module, which it's not. > > > I hoped maybe `-modfile` would do the same trick, but alas not: > > -modfile lets you change the effective content of go.mod but not the module root directory. Unfortunately it doesn't look like that can be used to work around the issue. > > > It seems that is because the "main" (top-level dir) go.mod has > > `replace` directives with relative paths, which kubernetes really > > does. > > You may need to copy those over to the tmp go.mod and adjust the paths. Sorry this has gotten pretty involved.
I don't think I follow what you mean. If you look at https://github.com/thockin/go-list-modules, I generated this file in /tmp: ``` module tmp go 1.15 require ( example.com/m v0.0.0 example.com/m/submod v0.0.0 example.com/other1 v0.0.0 example.com/other2 v0.0.0 ) replace ( example.com/m v0.0.0 => /tmp/gomodhack/2/go-list-modules example.com/m/submod v0.0.0 => /tmp/gomodhack/2/go-list-modules/submod example.com/other1 v0.0.0 => /tmp/gomodhack/2/go-list-modules/staging/src/example.com/other1 example.com/other2 v0.0.0 => /tmp/gomodhack/2/go-list-modules/staging/src/example.com/other2 ) ``` Some queries pass: ``` $ (cd /tmp/gomodhack/2/hack; go list /tmp/gomodhack/2/go-list-modules/submod/used) example.com/m/submod/used ``` and some fail: ``` $ (cd /tmp/gomodhack/2/hack; go list /tmp/gomodhack/2/go-list-modules/staging/src/example.com/other1/used) go: finding module for package example.com/m/staging/src/example.com/other1/used cannot find module providing package example.com/m/staging/src/example.com/other1/used: unrecognized import path "example.com/m/staging/src/example.com/other1/used": reading https://example.com/m/staging/src/example.com/other1/used?go-get=1: 404 Not Found ``` I can't see what's different between them. If I remove the `example.com/m` lines from the tmp go.mod, it works. > > Yeah, I noticed. When GO111MODULE=off, everything I am doing is much > > faster. I'm wary of depending on that forever, though. > > Module-aware mode is quite a bit more complicated than GOPATH mode, so to some degree it's not surprising it's slower... it's surprising that it's a LOT slower though. I expect there's some optimization work for us to do in the next development cycle. > > We would eventually like to deprecate GOPATH mode though, so it's a good idea not to depend on it in new tooling today. 'go list' should be fine to get package dependency info in either module mode or GOPATH mode. go/packages is useful if you need additional information on top of that (parsed syntax trees, type info). That's what I expected, hence trying to get ahead of it. The issue in https://github.com/golang/go/issues/43733 SOUNDS like it would alleviate the main problem I have? > > I want to run a slow codegen process only if the packages it depends > > on have ACTUALLY changed (mtime is a good enough proxy) and I don't > > know a priori which packages need codegen. I want to scan the file > > tree, find the files that need codegen, check their deps, and only > > then run the codegen. > > How much dependency info do you need? If the codegen is only within packages with files that have changed, 'go list' might be overkill (it always loads dependencies, even if they aren't printed). If you need dependencies or reverse dependencies, 'go list' or go/packages are probably the right tools. We collect 1-level of deps, which is (in practice) sufficient, but is a compromise. > On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin <thoc...@google.com> wrote: >> >> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod <jaycon...@google.com> wrote: >> > >> > I was initially going to suggest adding the module subdirectories as requirements to the main go.mod, then replacing those with the subdirectories. >> > >> > module example.com/m >> > >> > go 1.15 >> > >> > require ( >> > example.com/other1 v0.0.0 >> > example.com/other2 v0.0.0 >> > example.com/m/submod v0.0.0 >> > ) >> > >> > replace ( >> > example.com/other1 => ./staging/src/example.com/other1 >> > example.com/other2 => ./staging/src/example.com/other2 >> > example.com/m/submod v0.0.0 => ./submod >> > ) >> > >> > >> > I think you might have tried this already. It gives the same "main module ... does not contain package" error. I believe that's a bug. I've opened #43733 to track it. >> >> Interesting. If that's a bug, then maybe I'll be able to do what I >> need once fixed. >> >> > In general, it should be possible to give 'go list' an absolute or relative path (starting with ./ or ../) to any directory containing a package which is part of any module in the build list. For example, some tools list directories in the module cache to find out what package a .go file belongs to. >> > >> > As a workaround, you could put a go.mod in an otherwise empty directory (in /tmp or something), then require the relevant modules from the repo and replace them with absolute paths. Then you can run 'go list' in that directory with absolute paths of package directories. >> >> Interesting - is the difference the absolute paths vs relative? >> >> I hoped maybe `-modfile` would do the same trick, but alas not: >> >> ``` >> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >> example.com/m/submod/used >> >> $ go list --modfile /tmp/gomodhack/go.mod /tmp/go-list-modules/submod/used/ >> main module (tmp) does not contain package tmp/submod/used >> ``` >> >> It also fails some cases: >> >> ``` >> (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >> example.com/m/submod/used >> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/; >> go list /tmp/go-list-modules/staging/src/example.com/other1/used/) >> go: finding module for package example.com/m/staging/src/example.com/other1/used >> cannot find module providing package >> example.com/m/staging/src/example.com/other1/used: unrecognized import >> path "example.com/m/staging/src/example.com/other1/used": reading >> https://example.com/m/staging/src/example.com/other1/used?go-get=1: >> 404 Not Found >> ``` >> >> It seems that is because the "main" (top-level dir) go.mod has >> `replace` directives with relative paths, which kubernetes really >> does. >> >> > Incidentally, golang.org/x/tools/go/packages will call 'go list' under the hood in module mode. go/build might do the same, depending on how it's invoked. 'go list' may be the best thing to use if it gives the information you need. >> >> Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >> faster. I'm wary of depending on that forever, though. >> >> Stepping back, I fear I am pushing the square peg into a round hole. >> Let me restate what I am trying to do. >> >> I want to run a slow codegen process only if the packages it depends >> on have ACTUALLY changed (mtime is a good enough proxy) and I don't >> know a priori which packages need codegen. I want to scan the file >> tree, find the files that need codegen, check their deps, and only >> then run the codegen. >> >> We do this today with `go list` and GO111MODULE=off, but I was advised >> at some point that x/tools/go/packages was the future-safe approach. >> >> If there's a better way, I am all ears. >> >> Tim >> GO111MODULE=off >> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts < golang-nuts@googlegroups.com> wrote: >> >> >> >> Hi. This isn't exactly burning urgent, but it is a long-term issue >> >> for Kubernetes. If there's anything I can do to catalyze the >> >> discussion - tests to run, info to dig up, etc - please let me know. >> >> >> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> wrote: >> >> > >> >> > Hi Paul! >> >> > >> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> wrote: >> >> > > >> >> > > > I just can't figure out how to do this. Maybe it can't be done in `go >> >> > > > list` ? Or maybe we're just missing some detail of go modules.. >> >> > > >> >> > > go list operates in the context of a single module (in the mode you >> >> > > are interested in), so you cannot do this with a single command across >> >> > > multiple modules. >> >> > >> >> > This might be a real problem for us. For this post I am reducing it >> >> > to `go list`, but in actuality we have a small program that we wrote >> >> > which does what we need in terms of `go/build`. It works great when >> >> > `GO111MODULE=off` but is more than 100x slower normally. I thought it >> >> > was finally time to rewrite it in terms of `go/packages` and get rid >> >> > of GO111MODULE=off. That didn't pan out, hence this post. >> >> > >> >> > More inline and below >> >> > >> >> > > > First I do a `find` for any file that has a specific comment tag, >> >> > > > indicating that the package needs codegen. The results span several >> >> > > > of the in-repo submodules. >> >> > > >> >> > > Just to check, I'm assuming the results of this find command are being >> >> > > translated to a list of packages? Because the transitive dependencies >> >> > > of a list of packages within a module can be done via a single go list >> >> > > command. >> >> > >> >> > The trick is "within a module". I'll update >> >> > https://github.com/thockin/go-list-modules to reflect the process >> >> > more. I've added a >> >> > get_codegen_deps.sh that models the behavior. Note that I really want >> >> > files, not packages, so I can express the dep-graph. >> >> > >> >> > What do you mean by "translated to a list of packages" - which specific syntax? >> >> > >> >> > What I end up with is something like `go list ./path/to/dir1 >> >> > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in >> >> > different modules. So `go list` tells me "main module ( example.com/m) >> >> > does not contain package example.com/m/path/to/dir1" and so on. >> >> > Setting `GO111MODULE=off` does work, but I fear the future of that. >> >> > >> >> > > > For each target package, I want to get the list of all deps and >> >> > > > extract the GoFiles. Then I can use that to determine if the codegen >> >> > > > needs to run. >> >> > > >> >> > > FWIW I wrote a tool to do just this: >> >> > > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate >> >> > > which might work in your situation. >> >> > >> >> > I will take a look - it seems I will need to restructure a bunch of >> >> > tooling to prove it works for us or doesn't :) >> >> > >> >> > > > Where it breaks down is that I can't seem to `go list` all at once: >> >> > > > >> >> > > > ``` >> >> > > > # This works within the "root" module >> >> > > > $ go list -f '{{.GoFiles}}' ./subdir >> >> > > > [file.go] >> >> > > >> >> > > This will work. >> >> > > >> >> > > > # This does not work across modules >> >> > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused >> >> > > > main module (example.com/m) does not contain package example.com/m/submod/used >> >> > > > main module (example.com/m) does not contain package example.com/m/submod/unused >> >> > > >> >> > > Per above, this will not work across module boundaries. >> >> > >> >> > It works with `GO111MODULE=off` which means that introducing modules >> >> > is a breaking change. Can I depend on GO111MODULE=off to work the >> >> > same way forever? >> >> > >> >> > > > # Nor does this work, even with module replacements >> >> > > > $ go list -f '{{.GoFiles}}' ./staging/src/ example.com/other1/used >> >> > > > ./staging/src/example.com/other1/unused >> >> > > > main module (example.com/m) does not contain package >> >> > > > example.com/m/staging/src/example.com/other1/used >> >> > > > main module (example.com/m) does not contain package >> >> > > > example.com/m/staging/src/example.com/other1/unused >> >> > > > ``` >> >> > > >> >> > > With replace directives in place this should work, but you won't be >> >> > > able to use the relative path to the modules (which is in fact >> >> > > interpreted as a directory): it will need to be the full >> >> > > module/package path. >> >> > >> >> > Given a "./path/to/pkg" - how do I convert that to a module/package >> >> > path? I can run `(cd $dir && go list -m)` but that is super slow. >> >> > Running JUST that for each directory that needs codegen in kubernetes >> >> > takes 20+ seconds. Is there a better way, short of writing my own >> >> > directory-climb and parsing go.mod? >> >> > >> >> > > > I can run `go list` multiple times, but that's INCREDIBLY slow - most >> >> > > > of these submodules have common deps that are large. This re-parses >> >> > > > everything over and over. It takes almost 60 seconds just to do `cd >> >> > > > $dir; go list` (on the real kubernetes repo). >> >> > > >> >> > > Do you have a repro of this taking 60 seconds? Because that really >> >> > > shouldn't be the case with a populated local module cache. >> >> > >> >> > github.com/kubernetes/kubernetes >> >> > >> >> > ``` >> >> > $ time \ >> >> > find . -type f -name \*.go \ >> >> > | xargs grep -l "^// *+k8s:" \ >> >> > | xargs -n 1 dirname \ >> >> > | sort \ >> >> > | uniq \ >> >> > | while read X; do \ >> >> > (cd $X; go list -f '{{.Deps}}'); \ >> >> > done \ >> >> > > /dev/null >> >> > >> >> > real 0m50.488s >> >> > user 0m46.686s >> >> > sys 0m18.416s >> >> > ``` >> >> > >> >> > Just running that inner `go list` with GO111MODULE=off cuts the run >> >> > time in half. >> >> > >> >> > Compare to: >> >> > >> >> > ``` >> >> > time \ >> >> > ( \ >> >> > export GO111MODULE=off; \ >> >> > find . -type f -name \*.go \ >> >> > | xargs grep -l "^// *+k8s:" \ >> >> > | xargs -n 1 dirname \ >> >> > | sort \ >> >> > | uniq \ >> >> > | xargs go list -e -f '{{.Deps}}' \ >> >> > ) \ >> >> > > /dev/null >> >> > >> >> > real 0m1.323s >> >> > user 0m1.174s >> >> > sys 0m0.567s >> >> > ``` >> >> > >> >> > The model repo doesn't show so significantly because it is small. >> >> > Kubernetes is not small. >> >> > >> >> > I'm happy to hear better approaches - I really don't like relying on >> >> > GO111MODULE=off forever - it seems like the sort of thing that will >> >> > eventually get removed. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups "golang-nuts" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. >> >> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa8sbKOUXbHThYp%2BidAv9XN2YA7LuGLZ1qFcTqxvFJtRg%40mail.gmail.com.
