[v3] Binding generator using go/types

[abichinger]

Description

This PR continues the work done by @fbbdev in #3347. I used this commit 241c578 as a starting point and modified the binding and model generator code to work with the go/types package. Using go/types instead of go/ast fixes the problems with object resolution.
#3347 (comment)
golang/go#48141 (comment)

Breaking changes

• The identifier of a bound method is now packagePath.structName.methodName instead of packageName.structName.methodName to avoid collisions (a702897)
• GenerateBindingsOptions.UseIDs got replaced by GenerateBindingsOptions.UseNames making CallByID the new default (aa098b7)
• I changed the interface of the parser package

New features

• It is possible to bind structs from external packages. An example can be found at wails/v3/internal/parser/testdata/multiple_packages. You can run this example with go run .
• go types that are not structs or "enums" generate a Typescript alias type: https://github.com/abichinger/wails/blob/1660a12ba2d0f20e2ab5698b4156432c999b043f/v3/internal/parser/testdata/complex_json/frontend/bindings/main/models.ts#L5
• Support types implementing the encoding.TextMarshaler interface. The resulting Typescript type is string. https://github.com/abichinger/wails/blob/1660a12ba2d0f20e2ab5698b4156432c999b043f/v3/internal/parser/testdata/multiple_packages/frontend/bindings/github.com-google-uuid/models.ts#L9

Fixes

• Use the first constant value of an Enum as default
• The JSDoc enum definition was missing the <type> parameter
• The ID of bound methods from a nested package was previously incorrect

Limitations

• iota is not supported to define enums
• It is not possible to bind methods, which accept an interface as parameter
• Interface fields inside structs are dropped
• Generic types are ignored
• Can not bind packages
• Unable to predict fields of types, which implement json.Marshaler

Downsides

• The new binding generator is significantly slower. In my test project, generation takes 243ms (18 times slower than [v3] Binding generator improvements #3347)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

All tests have passed. Some expected results of tests inside v3/internal/parser/testdata have been corrected.
The binding example has been tested and updated.
The bindings of a test project have been generated.

• Windows
• macOS
• Linux

Test Configuration

# System
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Name         | Manjaro Linux                                                                                                  |
| Version      | Unknown                                                                                                        |
| ID           | manjaro                                                                                                        |
| Branding     |                                                                                                                |
| Platform     | linux                                                                                                          |
| Architecture | amd64                                                                                                          |
| CPU          | AMD Ryzen 5 3600 6-Core Processor                                                                              |
| GPU 1        | Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (Advanced Micro Devices, Inc. [AMD/ATI]) - Driver: amdgpu  |
| Memory       | 32GB                                                                                                           |
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

# Build Environment
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
| Wails CLI      | v3.0.0-alpha.4                                                                 |
| Go Version     | go1.22.0                                                                       |
| Revision       | 43418e96f36455467657822f8487d8a567dfda17                                       |
| Modified       | true                                                                           |
| -buildmode     | exe                                                                            |
| -compiler      | gc                                                                             |
| CGO_CFLAGS     |                                                                                |
| CGO_CPPFLAGS   |                                                                                |
| CGO_CXXFLAGS   |                                                                                |
| CGO_ENABLED    | 1                                                                              |
| CGO_LDFLAGS    |                                                                                |
| DefaultGODEBUG | httplaxcontentlength=1,httpmuxgo121=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1 |
| GOAMD64        | v1                                                                             |
| GOARCH         | amd64                                                                          |
| GOOS           | linux                                                                          |
| vcs            | git                                                                            |
| vcs.modified   | true                                                                           |
| vcs.revision   | 43418e96f36455467657822f8487d8a567dfda17                                       |
| vcs.time       | 2024-04-15T14:26:43Z                                                           |
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

# Dependencies
┌───────────────────────────┐
| pkg-config | 2.1.0-2      |
| gcc        | 13.2.1-5     |
| libgtk-3   | 1:3.24.41-1  |
| libwebkit  | 2.42.5-2     |
| npm        | 10.5.0       |
└─ * - Optional Dependency ─┘

# Diagnosis
 SUCCESS  Your system is ready for Wails development!

Checklist:

• I have updated website/src/pages/changelog.mdx with details of this PR
• My code follows the general coding style of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[abichinger]

I managed to improve the performance a bit. The previously mentioned test project now takes about 115ms (before: 243ms) to process. The main performance gain comes from calling LoadPackages() only once instead of multiple times.

During benchmarking, I noticed that ParseProject(=LoadPackages) is by far the most time consuming step. I glanced over the source code of packages [packages.go;l=849] and I think it already runs concurrently. Unfortunately, improving the packages.Load function is definitely beyond my skill set. So I guess I can not improve this PR much further.

Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkParser$ github.com/wailsapp/wails/v3/internal/parser

goos: linux
goarch: amd64
pkg: github.com/wailsapp/wails/v3/internal/parser
cpu: AMD Ryzen 5 3600 6-Core Processor              
BenchmarkParser/struct_literal_single/ParseProject-12         	       9	 122664394 ns/op	 6680450 B/op	   59841 allocs/op
BenchmarkParser/struct_literal_single/ParsePackages-12        	    7963	    244462 ns/op	   79549 B/op	    1532 allocs/op
BenchmarkParser/struct_literal_single/GenerateBindings-12     	     751	   1475458 ns/op	  191068 B/op	    5965 allocs/op
BenchmarkParser/struct_literal_single/GenerateModels-12       	    5888	    220428 ns/op	   32871 B/op	     792 allocs/op
BenchmarkParser/complex_json/ParseProject-12                  	       9	 121939557 ns/op	 6273351 B/op	   56070 allocs/op
BenchmarkParser/complex_json/ParsePackages-12                 	   10000	    122095 ns/op	   34350 B/op	     723 allocs/op
BenchmarkParser/complex_json/GenerateBindings-12              	   18594	     65682 ns/op	    8942 B/op	     241 allocs/op
BenchmarkParser/complex_json/GenerateModels-12                	    1755	    614594 ns/op	   85877 B/op	    2502 allocs/op
BenchmarkParser/multiple_packages/ParseProject-12             	       9	 117322676 ns/op	 8458869 B/op	   69577 allocs/op
BenchmarkParser/multiple_packages/ParsePackages-12            	      98	  12656978 ns/op	  831900 B/op	   18418 allocs/op
BenchmarkParser/multiple_packages/GenerateBindings-12         	    3052	    516789 ns/op	   86948 B/op	    2411 allocs/op
BenchmarkParser/multiple_packages/GenerateModels-12           	    1986	    523006 ns/op	   80129 B/op	    1954 allocs/op
PASS
ok  	github.com/wailsapp/wails/v3/internal/parser	20.733s

BenchmarkParser/multiple_packages/ParsePackages takes 13ms, because ParsePackages loads the documentation for each struct coming from an imported package.

[overlordtm]

Hey, just want to give some feedback, I have quite complicated models and this is first version of parser that managed to parse it and generate TS models correctly. Only error types are still problem, it seems go/types does not handle this.

I have also fixed parse so call to application.New can be anywhere, not only in main. Seems it works :D

[leaanthony]

@overlordtm - It would be great if you could contribute to this PR 🙏

[abichinger]

Thank you for testing!

Only error types are still problem, it seems go/types does not handle this.

@overlordtm could you provide an example?

[leaanthony]

@abichinger - do you have a feature list for this enhancement? I'm thinking this will help with the integration of @fbbdev 's work.

[abichinger]

Apart from the bugfixes and features I already mentioned above, this PR is functionally identical to #3347.

I think it is best to wait for @fbbdev to publish his new architecture and then we can compare.

[leaanthony]

I spoke to @fbbdev earlier and he suggested closing the other PR and concentrating efforts on this, so that's what I'll do.

[abichinger]

I would appreciate some feedback on the generated package directories.

The current behaviour is as follows.

1. Services and models inside the project directory will be placed inside main.
2. Each subpackage in the project directory ends up in the same subfolder as in the project.
3. External packages get placed inside a folder identical to their package path (/ is replaced with -)

Let's asume our project directory has the package path github.com/alice/foo.
Services and models defined inside ...

• github.com/alice/foo will end up in main
• github.com/alice/foo/bar will end up in bar
• github.com/alice/foo/bar/baz will end up in bar/baz
• log (standard package) will end up in log
• github.com/google/uuid will end up in github.com-google-uuid

The current solution could still cause collisions. For example, a package log inside the project could collide with the standard log package.

Do you consider the current solution to be okay?

Update 2024-04-21

With 3be3606 the generated package directories are somewhat configurable. The wails3 generate bindings command now has two additional options.

-base string
        The base package path (default ".")
-n    Place the base package bindings inside a directory with the base package name

Here are some examples:

Results for executing wails3 generate bindings -d="." inside github.com/alice/foo:

• github.com/alice/foo -> .
• github.com/alice/foo/bar -> bar
• github.com/alice/foo/bar/baz -> bar/baz
• log (standard package) -> log
• github.com/google/uuid -> github.com/google/uuid

Results for executing wails3 generate bindings -d . -n inside github.com/alice/foo with package name main:

• github.com/alice/foo -> main
• everything else stays the same

Results for executing wails3 generate bindings -d . -base github.com/alice inside github.com/alice/foo:

• github.com/alice/foo -> foo
• github.com/alice/foo/bar -> foo/bar
• github.com/alice/foo/bar/baz -> foo/bar/baz
• log (standard package) -> log
• github.com/google/uuid -> github.com/google/uuid

[leaanthony]

Awesome work 😎

I've created a post on discord to discuss the ains/strategy of the parser which I think would be a better forum than GH.

[leaanthony]

@abichinger - Can we close this in favour of #3468 now?
@overlordtm - It would be amazing if you could let us know if the next iteration of the parser (linked above ☝️ ) works for your scenarios 🙏

[abichinger]

Closing in favour of #3468