Hi. This isn't exactly burning urgent, but it is a long-term issue for Kubernetes. If there's anything I can do to catalyze the discussion - tests to run, info to dig up, etc - please let me know.
On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> wrote: > > Hi Paul! > > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> wrote: > > > > > I just can't figure out how to do this. Maybe it can't be done in `go > > > list` ? Or maybe we're just missing some detail of go modules.. > > > > go list operates in the context of a single module (in the mode you > > are interested in), so you cannot do this with a single command across > > multiple modules. > > This might be a real problem for us. For this post I am reducing it > to `go list`, but in actuality we have a small program that we wrote > which does what we need in terms of `go/build`. It works great when > `GO111MODULE=off` but is more than 100x slower normally. I thought it > was finally time to rewrite it in terms of `go/packages` and get rid > of GO111MODULE=off. That didn't pan out, hence this post. > > More inline and below > > > > First I do a `find` for any file that has a specific comment tag, > > > indicating that the package needs codegen. The results span several > > > of the in-repo submodules. > > > > Just to check, I'm assuming the results of this find command are being > > translated to a list of packages? Because the transitive dependencies > > of a list of packages within a module can be done via a single go list > > command. > > The trick is "within a module". I'll update > https://github.com/thockin/go-list-modules to reflect the process > more. I've added a > get_codegen_deps.sh that models the behavior. Note that I really want > files, not packages, so I can express the dep-graph. > > What do you mean by "translated to a list of packages" - which specific > syntax? > > What I end up with is something like `go list ./path/to/dir1 > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in > different modules. So `go list` tells me "main module (example.com/m) > does not contain package example.com/m/path/to/dir1" and so on. > Setting `GO111MODULE=off` does work, but I fear the future of that. > > > > For each target package, I want to get the list of all deps and > > > extract the GoFiles. Then I can use that to determine if the codegen > > > needs to run. > > > > FWIW I wrote a tool to do just this: > > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate > > which might work in your situation. > > I will take a look - it seems I will need to restructure a bunch of > tooling to prove it works for us or doesn't :) > > > > Where it breaks down is that I can't seem to `go list` all at once: > > > > > > ``` > > > # This works within the "root" module > > > $ go list -f '{{.GoFiles}}' ./subdir > > > [file.go] > > > > This will work. > > > > > # This does not work across modules > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused > > > main module (example.com/m) does not contain package > > > example.com/m/submod/used > > > main module (example.com/m) does not contain package > > > example.com/m/submod/unused > > > > Per above, this will not work across module boundaries. > > It works with `GO111MODULE=off` which means that introducing modules > is a breaking change. Can I depend on GO111MODULE=off to work the > same way forever? > > > > # Nor does this work, even with module replacements > > > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used > > > ./staging/src/example.com/other1/unused > > > main module (example.com/m) does not contain package > > > example.com/m/staging/src/example.com/other1/used > > > main module (example.com/m) does not contain package > > > example.com/m/staging/src/example.com/other1/unused > > > ``` > > > > With replace directives in place this should work, but you won't be > > able to use the relative path to the modules (which is in fact > > interpreted as a directory): it will need to be the full > > module/package path. > > Given a "./path/to/pkg" - how do I convert that to a module/package > path? I can run `(cd $dir && go list -m)` but that is super slow. > Running JUST that for each directory that needs codegen in kubernetes > takes 20+ seconds. Is there a better way, short of writing my own > directory-climb and parsing go.mod? > > > > I can run `go list` multiple times, but that's INCREDIBLY slow - most > > > of these submodules have common deps that are large. This re-parses > > > everything over and over. It takes almost 60 seconds just to do `cd > > > $dir; go list` (on the real kubernetes repo). > > > > Do you have a repro of this taking 60 seconds? Because that really > > shouldn't be the case with a populated local module cache. > > github.com/kubernetes/kubernetes > > ``` > $ time \ > find . -type f -name \*.go \ > | xargs grep -l "^// *+k8s:" \ > | xargs -n 1 dirname \ > | sort \ > | uniq \ > | while read X; do \ > (cd $X; go list -f '{{.Deps}}'); \ > done \ > > /dev/null > > real 0m50.488s > user 0m46.686s > sys 0m18.416s > ``` > > Just running that inner `go list` with GO111MODULE=off cuts the run > time in half. > > Compare to: > > ``` > time \ > ( \ > export GO111MODULE=off; \ > find . -type f -name \*.go \ > | xargs grep -l "^// *+k8s:" \ > | xargs -n 1 dirname \ > | sort \ > | uniq \ > | xargs go list -e -f '{{.Deps}}' \ > ) \ > > /dev/null > > real 0m1.323s > user 0m1.174s > sys 0m0.567s > ``` > > The model repo doesn't show so significantly because it is small. > Kubernetes is not small. > > I'm happy to hear better approaches - I really don't like relying on > GO111MODULE=off forever - it seems like the sort of thing that will > eventually get removed. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com.
