Analyze NuGet.org packages π¦ using Azure Functions β‘.
This project enables you to write a bit of code that will be executed for each package on NuGet.org in parallel. The results of the code will be collected into CSV files stored Azure Blob Storage. These CSV files can be imported into any query system you want, for easy analysis. This project is about building those CSV blobs in a fast, scalable, and reproducible way as well as keeping those files up to date.
The data sets are great for:
- π Ad-hoc investigations of the .NET ecosystem
- π Estimate the blast radius of a bug affecting NuGet packages
- π Check the trends over time on NuGet.org
- π Look at adoption of various NuGet or .NET features
The data sets currently produced by NuGet Insights are listed in
docs/tables/README.md
.
We follow a 3 step process to go from nothing to a completely deployed Azure solution.
- Build the code
- Deploy to Azure
- Start analysis from the admin panel
- Ensure you have the .NET 8 SDK installed. Install it if
needed.
dotnet --info
- Clone the repository.
git clone https://github.com/NuGet/Insights.git
- Run
dotnet publish
on the website and worker projects. This produces compiled directories that can be deployed to Azure later.cd Insights dotnet publish src/Worker -c Release dotnet publish src/Website -c Release
Read about how to run the automated tests in TESTING.md
.
PowerShell is used for the following steps. I have tested Windows PowerShell 5.1, Windows PowerShell 7.3.7, and Linux PowerShell 7.3.7.
- Ensure you have the Az PowerShell modules. Install them if
needed.
Connect-AzAccount
- Ensure you have Bicep installed. Install it if
needed.
bicep --version
- Ensure you have the desired Azure subscription selected.
Set-AzContext -Subscription $mySubscriptionId
- From the root of the repo, deploy with the desired config
and stamp name.
If you run into trouble, try adding the
./deploy/deploy.ps1 -ConfigName dev -StampName Joel -AllowDeployUser
-Debug
option to get more diagnostic information.
This will create a new resource group with name NuGet.Insights-{StampName}
deploy several resources into it including:
- an App Service, containing a website for starting scans
- a Function App with Consumption plan, for running the scans
- a Storage account, for maintaining intermediate state and results (CSV files)
- an Application Insights instance, for investigating metrics and error logs
- a Key Vault for auto-rotating the storage access key
When the deployment completes successfully, a "website URL" will be reporting in the console as part of a warm-up. You can use this to access the admin panel. The end of the output looks like this:
... Warming up the website and workers... https://nugetinsights-joel.azurewebsites.net/ - 200 OK https://nugetinsights-joel-worker-0.azurewebsites.net/ - 200 OK Deployment is complete. Go to here for the admin panel: https://nugetinsights-joel.azurewebsites.net/Admin
You can go the first URL which is the website URL in your web browser and click on the Admin link in the nav bar. Then, you can start a short run using the "All catalog scans" section, "Use custom cursor" checkbox, and "Start all" button.
For more information about running catalog scans, see Starting a catalog scan.
Use one of the following approaches to run Insights locally. Using Project Tye is the easiest if you have Docker installed, otherwise use a standalone Azure Storage emulator.
From Project Tye's GitHub page:
Tye is a developer tool that makes developing, testing, and deploying microservices and distributed applications easier. Project Tye includes a local orchestrator to make developing microservices easier and the ability to deploy microservices to Kubernetes with minimal configuration.
It's a great way to run the Insights website, worker, and the Azurite storage emulator all at once with a single command.
- Clone the Insights repository.
- Install Project Tye if you haven't already.
- Make sure you have Docker installed since it is used for running Azurite.
- Execute
tye run
in the root of the repository. - Open the Tye dashboard using the URL printed to stdout, e.g.
Dashboard running on http://127.0.0.1:8000
- From the Tye dashboard, you can navigate to the website URL (shown in the Bindings).
Proceed to the Starting a catalog scan section.
- Clone the repository.
- Install and start an Azure Storage emulator for Blob, Queue, and Table
storage.
- Azurite: can run from VS Code, npm, and more; make sure to use version 3.19.0 or newer.
- Azure Storage Emulator: this emulator only works on Windows and is deprecated.
- Execute
dotnet run --project src/Worker
from the root of the repository. - From another terminal window, run
dotnet run --project src/Website
from the root of the repository.- The website and the worker don't necessarily need to run in parallel, but it's easier to watch the progress if you leave both running.
- Open the website URL printed to stdout, e.g.
Now listening on: http://localhost:60491
Proceed to the Starting a catalog scan section.
A catalog scan is a unit of work for Insights which runs analysis against all of the packages published during some time range. The time range for a catalog scan is bounded by the a previous catalog stamp used (as an exclusive minimum) and an arbitrary timestamp to process up to (as an inclusive maximum). For more information, see the architecture section.
Once you have opened the localhost website URL mentioned in the section above, follow these steps to start your first catalog scan from the Insights admin panel.
- In your web browser, viewing the website URL, click on the "Admin" link in the navigation bar.
- Start one or more catalog scans.
- For your first try, run a single driver against a single NuGet V3
catalog
commit.
- Expand the Load package archive section.
- Check Use custom cursor.
- Use the default value of
2015-02-01T06:22:45.8488496Z
, which is the very first commit timestamp in the NuGet V3 catalog. - Click Start.
- You can start all of the catalog scans with the same timestamp using the "All catalog scans" section but this will take many hours while running on your local machine. There are a lot of drivers and a lot of packages on NuGet.org π.
- For your first try, run a single driver against a single NuGet V3
catalog
commit.
- Make sure the background worker is running (either via Tye or starting the Worker project from the terminal).
- Wait until the catalog scan is done. You can check the current progress by refreshing the admin panel and looking at the number of messages in the queues (first section in the admin panel) or by looking at the catalog scan record created in the previous step.
If you ran a driver like Load package archive, data will be populated into
your Azure Table Storage emulator in the packagearchives
table. If you ran a
driver like Package asset to CSV, CSV files will be populated into your
Azure Blob Storage emulator in the packageassets
container. For more information on what each driver does, see the drivers list.
You can use the Azure Storage Explorer to interact with your Azure Storage endpoints (either the storage emulator running locally or in Azure).
When running locally, you can check the application logs shown in the Tye dashboard or terminal stdout. When running in Azure, you can use Application Insights (note the default logging is Warning or higher to reduce cost). You can also look at the Azure Queue Storage queues to understand what sort of work the Worker has left.
- Tables - documentation for all of the data tables produced by this project
- Adding a new driver - a guide to help you enhance Insights to suit your needs
- Drivers - the list of existing drivers
- Reusable classes - interesting or useful classes or concepts supporting this project
- Blog posts - blog posts about lessons learned from this project
- Cost - approximately how much it costs to run several of the implemented catalog scans
Read about the project's architecture in ARCHITECTURE.md
.
These are what the resources look like in Azure after deployment.
This is what the Azure Function looks like running locally, for the Package Manifest to CSV driver.
This is what the results look like in Azure Table Storage. Each row is a package .nuspec stored as compressed MessagePack bytes.
This is what the admin panel looks like to start catalog scans.
This is the driver that reads the file list and package signature from all NuGet packages on NuGet.org and loads them into Azure Table Storage. It took about 35 minutes to do this and costed about $3.37.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.