Skip to content

Manifest parsing with syntax#9901

Open
DougGregor wants to merge 4 commits intoswiftlang:mainfrom
DougGregor:manifest-parsing-with-syntax
Open

Manifest parsing with syntax#9901
DougGregor wants to merge 4 commits intoswiftlang:mainfrom
DougGregor:manifest-parsing-with-syntax

Conversation

@DougGregor
Copy link
Copy Markdown
Member

This is #9759 but reverting #9883

package manifest directly, and then walks the resulting syntax tree
to find the package declaration, its targets, dependencies, and so on.
For anything in the syntax tree that the parsing manifest loader does
not recognize, it records a "limitation" associated with that syntax
tree node. The presence of any limitations means that the result of
the parsing manifest loader should not be trusted, in which case we
can fall back to the existing manifest loader. One can request that
the limitations, if found, are printed to the terminal using the
experimental flag --experimental-show-manifest-parser-limitations.
This can be used to determine why a particular package manifest isn't
handled by the parsing manifest loader. We could use this information
to expand the capabilities of the parsing manifest loader in the
future, so that more manifests would benefit from it as a fast path.

The parsing manifest loader takes advantage of the SwiftIfConfig
library to handle #if in the package manifest. It queries the compiler
frontend to determine the configuration of the compiler that would be
used to build the manifest so we get the same results from #if.
Additionally, the parsing manifest loader maintains a static version
of the Context that is available to the package manifest, so it can
substitute (e.g.) the package directory, Git information, or
environment information into the package manifest.

The large amount of churn in the package loading tests are because
most of the tests have been switched from just using the existing
manifest loader to iterating over both loaders (using
forEachManifestLoader). This ensures that both loaders pass the same
set of tests. Additionally, we check that the resulting manifests from
both loaders matches exactly (by encoding them into JSON) to ensure
that the results are the same. This way, we can keep the manifest
loaders in sync for the subset of package manifests that are meant to work.

This also adds an experimental command-line option,
--experimental-manifest-processing-mode, that enables the parsing
manifest loader. It must be provided with one of the following options:

- only-parsed: only use the parsing loader. This is not correct, because
manifests can require execution, but can be used for testing.
- only-executed: only using the executing loader, which is the normal
one. This is the default for now.
- parsed-with-fallback: try using the parsing loader. If it encounters
limitations, fall back to the executing loader without failing. This
option is what can provide build-time performance improvements when
the parsing manifest loader takes over.
- crosscheck: try both the parsing and the executing loaders and compare
the results to ensure they match. If the parsing loader encounters
limitations, no further checking will be done. On success, it will
print the relative times. If the results differ, it will print the
JSON form of the manifest from each parser.
The loop at the end was silly. Keep track of where the condition is so
we can parse it later.
@DougGregor
Copy link
Copy Markdown
Member Author

@swift-ci please test

@DougGregor
Copy link
Copy Markdown
Member Author

@swift-ci build toolchain

@DougGregor
Copy link
Copy Markdown
Member Author

@swift-ci please test macOS

The host library path contains the shared libraries against which the
various tools in the toolchain are built, which means the host compiler---not
the runtime library path for the platform.
@DougGregor
Copy link
Copy Markdown
Member Author

@swift-ci please test

@DougGregor
Copy link
Copy Markdown
Member Author

@swift-ci please smoke test Windows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant