Thanks for that. I spent some time over the weekend refactoring from raw ast <https://pkg.go.dev/go/ast> parser with a simpleImporter and universe to using types <https://pkg.go.dev/go/types>.
So far this has provided everything I need, and also massively simplified the code as no need to manually parse the ast.File apart from a basic ast.Visitor <https://pkg.go.dev/go/ast#Visitor> to create a lookup map for type comments and position information. I did find the types docs <https://pkg.go.dev/go/types> less consumable than the ast docs <https://pkg.go.dev/go/ast> as the relationship between the various types and the data in types.Info <https://pkg.go.dev/go/types#Info> wasn't particularly clear to me even after reading the tutorial <https://golang.org/s/types-tutorial>. This resulted in it taking quite a bit of trial and error, which I didn't need with raw ast. However once I combined an ast.Visitor <https://pkg.go.dev/go/ast#Visitor> and a pass over info.Defs to create a type name map I had most of all the elements. Getting the type name from types.Info <https://pkg.go.dev/go/types#Info> map had me scratching my head for some time as I couldn't believe that the Types field wouldn't have an easy way to get to the name of the type. Using types Config.Check <https://pkg.go.dev/go/types#Config.Check> didn't have any performance penalties that I saw with packages.Load <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#Load> so I'm hopeful that this is the correct route. I'd love to hear if there is a neat way to get from info.Types to the name of each type without having to process info.Defs first, but that's more to know if missed something obvious or not 😋 Thanks again to everyone who has provided pointers in this thread, as it demonstrates what great community this is! On Sun, 17 Oct 2021 at 22:16, K. Alex Mills <k.alex.mi...@gmail.com> wrote: > I'm guessing here, but you probably do need the full type-checker for > that, depending on what you mean by "a native type". > > In go we can alias types in two ways. > > type A uint8 > type B = uint8 > > There are subtle differences between the two which I don't claim to > understand... In any case, AFAIK both are essentially built-in types at > runtime. > > Suppose these types are declared in a third-party package on GitHub and > another package uses A and B (since they're exported) to declare a struct > like this. > > import "github.com/foo/bar" > > type MyStruct struct { > Byte1 bar.A > Byte2 bar.B > } > > AFAIK, the compiler cannot tell whether Byte1 and Byte2 are builtin types > or a struct without seeing their definition hosted on GitHub. Which means > downloading, parsing, and type-checking the code from GitHub. > > But it gets a little worse. Type-checking the remote code entails > downloading any of its dependencies recursively and doing the same thing... > I'm sure you can see how this can take some time -- especially for large > projects. > > Anyway, it sounds to me like what you're asking to do requires the use of > the type-checker and because of the dependency management involved I > believe the most convenient route to type-checking (at this time) lives in > the packages package. > > Happy to learn if anyone else knows otherwise. > > On Sun, Oct 17, 2021, 5:00 AM Steven Hartland <ste...@multiplay.co.uk> > wrote: > >> I need to be able to tell the types of fields, in particular are fields >> of a struct a native type or a struct themselves. >> >> The ast parse even with a simple importer don’t provide that info. >> >> On Sat, 16 Oct 2021 at 21:06, 'Richard Oudkerk' via golang-nuts < >> golang-nuts@googlegroups.com> wrote: >> >>> I am not sure what "import external packages" means. >>> >>> Apart dot imports (which I have never seen used for real) why would you >>> need to load the imported packages? >>> >>> On Saturday, 16 October 2021 at 20:34:17 UTC+1 Steven Hartland wrote: >>> >>>> Thanks Richard, that allowed me to replace a hand rolled >>>> universe scope 👍 >>>> >>>> My importer varies from yours in that for correct lookups for versioned >>>> packages or those with '-' in I had to copy ImportPathToAssumedName >>>> from x/tools/internal/imports/fix.go. >>>> >>>> func simpleImporter(imports map[string]*ast.Object, path string) >>>> (*ast.Object, error) { >>>> pkg := imports[path] >>>> if pkg == nil { >>>> pkg = ast.NewObj(ast.Pkg, ImportPathToAssumedName(path)) >>>> pkg.Data = ast.NewScope(nil) // required by >>>> ast.NewPackage for dot-import >>>> imports[path] = pkg >>>> } >>>> return pkg, nil >>>> } >>>> >>>> This now works for all cases which don't import external packages. So >>>> now I just need to do the on demand load of packages, which I suspect will >>>> lead me right back to packages.Load. >>>> >>>> On Sat, 16 Oct 2021 at 15:59, 'Richard Oudkerk' via golang-nuts < >>>> golan...@googlegroups.com> wrote: >>>> >>>>> You could try building the universe scope for ast.NewPackage from >>>>> types.Universe. For example >>>>> >>>>> https://play.golang.org/p/1E5Iu4vW3g9 >>>>> >>>>> func NewPackage(fset *token.FileSet, files map[string]*ast.File) >>>>> (*ast.Package, error) { >>>>> univ, err := universe() >>>>> if err != nil { >>>>> return nil, err >>>>> } >>>>> return ast.NewPackage(fset, files, dummyImporter, univ) >>>>> } >>>>> >>>>> func dummyImporter(imports map[string]*ast.Object, importPath string) >>>>> (*ast.Object, error) { >>>>> pkg := imports[importPath] >>>>> if pkg == nil { >>>>> pkg = ast.NewObj(ast.Pkg, path.Base(importPath)) >>>>> pkg.Data = ast.NewScope(nil) >>>>> imports[importPath] = pkg >>>>> } >>>>> return pkg, nil >>>>> } >>>>> >>>>> func universe() (*ast.Scope, error) { >>>>> u := ast.NewScope(nil) >>>>> for _, name := range types.Universe.Names() { >>>>> o := types.Universe.Lookup(name) >>>>> if o == nil { >>>>> return nil, fmt.Errorf("failed to lookup %s in universe scope", name) >>>>> } >>>>> var objKind ast.ObjKind >>>>> switch o.(type) { >>>>> case *types.Const, *types.Nil: >>>>> objKind = ast.Con >>>>> case *types.TypeName: >>>>> objKind = ast.Typ >>>>> case *types.Builtin: >>>>> objKind = ast.Fun >>>>> default: >>>>> return nil, fmt.Errorf("unexpected builtin %s of type %T", o.Name(), o) >>>>> } >>>>> obj := ast.NewObj(objKind, name) >>>>> if u.Insert(obj) != nil { >>>>> return nil, fmt.Errorf("types internal error: double declaration") >>>>> } >>>>> obj.Decl = u >>>>> } >>>>> return u, nil >>>>> } >>>>> >>>>> On Saturday, 16 October 2021 at 14:38:43 UTC+1 eli...@gmail.com wrote: >>>>> >>>>>> On Fri, Oct 15, 2021 at 2:13 PM Steven Hartland < >>>>>> ste...@multiplay.co.uk> wrote: >>>>>> >>>>>>> I converted my code to x/tools/go/packages >>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> and >>>>>>> while it did solve the problem it's VERY slow in comparison. >>>>>>> >>>>>>> I have a set of 21 tests operating on a single package which has at >>>>>>> most two very basic types, no imports and using go/parser >>>>>>> <https://pkg.go.dev/go/parser> they take 0.011s but with go/packages >>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> that >>>>>>> increases to 3.548s a 300x slow down. >>>>>>> >>>>>>> I'm setting a basic mode: packages.NeedName | packages.NeedSyntax >>>>>>> >>>>>>> The package.Load call takes ~220ms whereas ast.NewPackage only >>>>>>> takes 2.7µs. >>>>>>> >>>>>> >>>>>> Could you post a reproducer of your target package and analysis >>>>>> somewhere? 220ms for packages.Load sounds like a lot. It's true that >>>>>> packages does a lot more work than just the parser (*), but it's not >>>>>> supposed to be that slow. In my tests a simple Load with more >>>>>> functionality >>>>>> takes 60-70ms >>>>>> >>>>>> (*) The type checking takes a bit of time over just parsing to AST, >>>>>> but the biggest difference is loading multiple files from imports. For >>>>>> type >>>>>> checking you need to know, when you see: >>>>>> >>>>>> import foo >>>>>> >>>>>> x := foo.Foo() >>>>>> >>>>>> What the type of `x` is, so go/packages has to analyze the `foo` >>>>>> package as well. >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> As the resulting ast.File's are pretty much the same, I'm wondering >>>>>>> if for my use case packages.Load is doing way more than I need? >>>>>>> >>>>>>> Another downside is for tests run in a temporary directory outside >>>>>>> of the package space package.Load fails with: >>>>>>> directory /tmp/tests76985775 outside available modules >>>>>>> >>>>>>> I fixed it by calling ioutil.TempDir with "." but that's not ideal. >>>>>>> >>>>>>> Thoughts? >>>>>>> >>>>>>> On Tue, 12 Oct 2021 at 13:42, Steven Hartland < >>>>>>> ste...@multiplay.co.uk> wrote: >>>>>>> >>>>>>>> Thanks David, much appreciated, I will have a look at both. >>>>>>>> >>>>>>>> When migrating from go/ast to go/types did you hit anything of >>>>>>>> note I should look out for? >>>>>>>> >>>>>>>> On Mon, 11 Oct 2021 at 17:06, David Finkel <david....@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Oct 11, 2021 at 5:48 AM Steven Hartland < >>>>>>>>> ste...@multiplay.co.uk> wrote: >>>>>>>>> >>>>>>>>>> If the ast.Files passed to ast.NewPackage includes built in types >>>>>>>>>> such as int it returns an error e.g. >>>>>>>>>> file1.go:5:6: undeclared name: int >>>>>>>>>> >>>>>>>>>> Is there a way to prevent that? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Generally, I always add the `builtin` package to the list of >>>>>>>>> packages I'm parsing. >>>>>>>>> I wrote a little library for exactly this kind of package loading >>>>>>>>> a few years ago: >>>>>>>>> https://gitlab.com/dfinkel/goastpkg/-/blob/master/go_ast_parser.go >>>>>>>>> (https://pkg.go.dev/golang.spin-2.net/astpkg) >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Playground example: https://play.golang.org/p/Yg30TTzoLHP >>>>>>>>>> >>>>>>>>>> My goal is to take multiple files, resolve inter file >>>>>>>>>> dependencies e.g. a type referencing another type in a different >>>>>>>>>> file and >>>>>>>>>> process the resulting ast.Files. So if there is a better way to >>>>>>>>>> achieve >>>>>>>>>> this I'm all ears. >>>>>>>>>> >>>>>>>>> >>>>>>>>> In general, I've stopped using the `go/ast` internal references as >>>>>>>>> much and have started using resolved `go/types` references as they're >>>>>>>>> more >>>>>>>>> reliable and better-specified. >>>>>>>>> (golang.org/x/tools/go/packages >>>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> has a >>>>>>>>> LoadMode flag for generating `go/types.Info` (NeedTypesInfo >>>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#NeedTypesInfo> >>>>>>>>> )) >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "golang-nuts" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "golang-nuts" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>> >>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to golang-nuts...@googlegroups.com. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com >>> <https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqaQ9%3Dia2MXrVEXt--Qk3Nrx1UXK2JYU0D2BxBcZX%2B5mxw%40mail.gmail >> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqaQ9%3Dia2MXrVEXt--Qk3Nrx1UXK2JYU0D2BxBcZX%2B5mxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAHEMsqZ-MjFGA97x3kb3fxJUDRQ_ezcb-%3DjkU1f1yRGoj%3DnPKg%40mail.gmail.com.
