From 404236379df1389c0988adca6c1f8cc0e1ef1011 Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Tue, 21 Jan 2025 10:42:19 +0100 Subject: [PATCH] One pager writeups (#11045) These are the one pagers for 3 features. Evaluation performance with @rainersigwald. PerfStar improvements with @AR-May. VS and SDK decoupling with @baronfel --- .../specs/proposed/decoupling-vs-sdk.md | 43 +++++++++++++++++++ .../specs/proposed/evaluation-perf.md | 20 +++++++++ documentation/specs/proposed/perfStar.md | 40 +++++++++++++++++ 3 files changed, 103 insertions(+) create mode 100644 documentation/specs/proposed/decoupling-vs-sdk.md create mode 100644 documentation/specs/proposed/evaluation-perf.md create mode 100644 documentation/specs/proposed/perfStar.md diff --git a/documentation/specs/proposed/decoupling-vs-sdk.md b/documentation/specs/proposed/decoupling-vs-sdk.md new file mode 100644 index 00000000000..58b5fe7c321 --- /dev/null +++ b/documentation/specs/proposed/decoupling-vs-sdk.md @@ -0,0 +1,43 @@ +# Decoupling VS builds of .NET SDK projects +The experience of building a .NET SDK project can differ significantly depending if the project was built using Visual Studio / MSBuild or `dotnet build`. The build can produce different diagnostics, use different language rules, etc. This is because building a .NET SDK project from Visual Studio mixes components from Visual Studio and the .NET SDK. This means core tooling, like the compiler, can be substantially different between the two types of build. This leads to customer confusion and hard-to-diagnose problems. To solve this want to ensure that when building a .NET SDK project we use more components from the .NET SDK to do so. + +## Goals and Motivation + +We are aiming for: + - More consistent end-user experience for build between .NET CLI and Visual Studio. + - Decoupling the .NET SDK experience from Visual Studio + +There are a few reasons that makes us persue this effort. +The first, we want a better experience when using or writting Roslyn analyzers and MSBuild Tasks. Currently tooling authors need to target NetStandard2.0 for their projects to be recognized by VS, and doing so blocks out newer features available. + +Second, tasks will not need to be multitargeted to cover both VS and .NET SDK. Right now, a lot of tasks need different versions to cover both of these scenarios, but with the changes in this features, authors will be able to use the same version for both situations. + + +## Impact +There are a few area of impact: + - .NET SDK style project builds will be more stable between VS and CLI builds, as the tooling will not be devided between different versions. + - Reduced cost of development for external and internal teams that author Roslyn Analyzers, source generators, or MSBuild Tasks. + - End-user will not experience mismatch between analyzer versions, and gain higher confidence that their .NET SDK project builds will behave the same way in VS and in the command line. + +## Stakeholders +Other teams will need to work to fully complete the VS and .NET SDK decoupling feature after our base work is done. There are two handovers in this project: + +1. After providing enough information to do so through MSBuild and the SDK, Roslyn will need to use it to invoke their .NET compiler in VS. +2. After MSBuild enables tasks to target .NET even for VS use, task-owning teams like the .NET SDK will need to migrate their targets to use .NET Core instead of keeping them targeting .NET Framework. + +The handovers should allow other teams to proceed with their work smoothly and no unexpected change in build behavior should be present within MSBuild. + +## Risks +A few risks associated with this feature: + - If .NET Core tasks is discovered to have too large of a performance impact (due to IPC overhead to a .NET process), core partner teams may choose to keep multitargeting their tasks for improved user perf. + - There is a hard deadline for this feature, VS17.14. As a consequence of how we support versions we would need to get all the work of this feature completed before that release. If we do not reach the deadline for this feature we would need to change policies on SDK level to be able to continue support. + + +## Plan + 1. Ensure that MSBuild.exe provides the same execution state as the dotnet command line invocation. + - This is should take around 1 dev week to complete, and will be handed over to Roslyn team. + 2. Implement .NET Core task host, tasks can be executed on the .NET Core vresion instead of .NET framework. + - This should take 1 to 2 dev months to complete, including extensive testing. This would be handed over to internal teams that have custom tasks so they can be updated and tested. + 3. Load common targets from the .NET SDK and not from .NET NetFramework on VS. This work depends on other team's finilizing their part of the feature and might not be in scope for .NET 10. + - This should take a dev week for code changes. For everything else, analysis, testing, etc... the time is very dependent on what happens after the code change, which we can't fully predict at this moment. + diff --git a/documentation/specs/proposed/evaluation-perf.md b/documentation/specs/proposed/evaluation-perf.md new file mode 100644 index 00000000000..22ea253c93d --- /dev/null +++ b/documentation/specs/proposed/evaluation-perf.md @@ -0,0 +1,20 @@ +# Evaluation performance investigations +In the current effort to improve performance of MSBuild, we identified evaluation as one of the focus areas of this effort. Evaluation is the first step when loading or building, and it determines references, how projects are connected and what needs to be build. Because of this it runs in every MSBuild scenario, from solution load and design-time builds in Visual Studio, to up-to-date builds or full builds in VS or on the command line. + +## Description +Current performance state of evaluation is mostly unkown, we have a few measures but no easy way of accessing and assessing them. As such, we are unsure which specific areas can be improve. + +## Goals and Motivation +We are trying to make evaluation phase of the build more performant, since it is almost always executed any performance gain becomes noticeable. A performant evaluation phase would decrease build times in general, in CI cases it frees up resources, and in individual cases it can increase dev-loop performance by making up-to-date and incremental builds go faster. + +In this moment we are still in investigation phase, the objective is to make the markers we have in code more accessible to the team, so we can idetentify low hanging fixes, and improvement areas when testing builds within PerfStar. + +Constraint - needs to work as it does today, but faster. We may be able to break some edge cases. + +## Risks +One of the big risks is accidentally changing the current behaviour of evaluation. One of the constraints of improvement is that evaluation has the same behavior, with the exception of edge cases where we can sometimes change it. + +## Plan +The plan for evaluation at the moment is to add more code markers during execution so we can use PerfStar to have a detailed view of how long each part of evaluation phase takes. + +Larger changes to the evaluation are possible and under consideration for future iterations, like trying to cache the evaluation result in MSBuild. However we are focusing on investigation and performance gains with less work at the moment. \ No newline at end of file diff --git a/documentation/specs/proposed/perfStar.md b/documentation/specs/proposed/perfStar.md new file mode 100644 index 00000000000..dffb0343aee --- /dev/null +++ b/documentation/specs/proposed/perfStar.md @@ -0,0 +1,40 @@ +# PerfStar +PerfStar is a performance tracking and investigation tool for MSBuild. PerfStar infrastructure captures performance measurements of the `main` MSBuild branch on a schedule and allows us to request experimental runs and collect performance data for proposed changes. The first version of this project is being finalized, with some fixes necessary to run it automatically and according to prerequisites. + +## Goals and Motivation +MSBuild currently does not have a lot of performance data outside of Visual Studio performance tracking, which has a lot of variables that are beyond the team's control. PerfStar enables us to measure our performance with less interference of elements that the team does not own. As such, we can measure the performance of in-development features and how it will impact build times, as well as have concrete numbers when working on performance improvement tasks. + +## Impact +Perfstar's impact is focused on the team. We will be able to track performance with concrete numbers. Because of that the team will be able to take more informed decisions about performance improvement work, as well as implementation of new features. In turn, those decisions will accrue value to users via higher build performance. + +## Risks +The risks associated with our dependencies is about Crank, which is owned by the ASP.NET team and we use it to help us with machine setup to run the performance tests. + +PerfStar also runs as a service. One that the mostly the team uses, but it is a service and carry the same risks as any other service product. Including possible downtime, maintanance, and some security areas. + +## Plan +Investiment for .NET 10: + 1. Making PerfStar execute automatically the way the design doc indicates + - Around 1 dev week. +2. The PowerBI reporting is working and updating the new information + - Around 2 dev weeks. +3. New performance tests for new features, and writing docs on how to write those tests. Next feature planned for tests: BuildCheck. + - Around 3 dev days per feature. +4. Analyze stability of performance tests, and fix any noise found. This will be done through multiple iterations of the same test in the same machine, as well as updating the PowerBI report to handle the new data. + - Around 2 dev weeks. +5. Add more tests using `msbuild.exe` for build in addition to `dotnet build`. + - Around 1 dev week. +6. Timeboxed collection of feedback from our team, as well as performance investigations that can derive from those. + - 1 - 2 dev month depending on feedback and requests for improvement from the team. +7. Add more test cases. For example, build time with different verbosity levels. + - Around 1 dev week. + +There are more improvements form PerfStar, but these are not planned for .NET 10 as they depend on the team's feedback to PerfStar. +1. Add more measurements, like dotnet counter tool. + - Around 3 dev weeks. +2. Trace collection when specific features are turned on for the test. + - Around 2 - 3 dev weeks. +3. Report improvements: + - Compare performance numbers between two different iterations that are not from `main` branch. Around 2 dev weeks. + - Automatic detection of performance issues, so we don't need to check the reports to see regressions. Around 1 dev month. + - Run MSBuild API tests, so we can check performance of calls relating to Visual Studio builds. Around 1 dev month. \ No newline at end of file