Improve UX and Performance of Install step

[notatallshaw]

What's the problem this feature will solve?

At the moment when the final install step starts pip gives no output what it is doing, in some real world cases (e.g. large pytorch installations or airflow installs) this steps can take over 30 seconds on fast machines, so minutes on slow machines. The user is left wondering if anything is happening.

Describe the solution you'd like

I would like to see the following improvements:

1. Log a message that pip is starting to install packages
2. Present a progress bar that tracks the number of packages installed out of the total packages to be installed
3. Improve any obvious performance bottlenecks (see follow up post with profile)
4. Run installs in parallel (made seperate issue Install packages in parallel #12742)

Alternative Solutions

I think at a bare minimum there should be a log message that lets the user know what's happening.

Additional context

uv runs installs in parallel, and following their issue tracker it does not appear to be problematic, to do this a cli option to control the maximum number would need to be added, the same as how there is a PR for parallel downloads to do this.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

This scenario is artifically constructed to best profile the installer code by removing the need to download, build sdists, or resolve:

1. python3.12 -m venv .venv
2. source .venv/bin/activate
3. <install latest/dev pip>
4. wget https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-3.12.txt
5. python -m pip download -d downloads -r constraints-3.12.txt
6. cd downloads
7. for file in $(ls *.tar.gz); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
8. for file in $(ls *.zip); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
9. cd -
10. python -m pip install --only-binary=:all: --no-index --ignore-installed --no-deps --find-links file://${PWD}/downloads -r constraints-3.12.txt

I ran with and without --dry-run to see the timing difference:
Dry Run: 32s
Regular Install: 144s

I profiled with and without --dry-run to see the profile difference:

Dry Run Profile

Regular Install Profile

There are some clear hotspots here, I will take a look when I have time if there are some easy ways to reduce those hotspots if no one else does.

[ichard26]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

[pfmoore]

I see no issues with the UI proposal, but I'd want parallel installs to be a separate feature. I can imagine pathological cases where things could break when installing in parallel, and while the experience of uv is encouraging (as is the fact that normal cases are clearly safe) my instinct is that every pathological case is being exercised by some user of pip, somewhere. So we should isolate the risk here by making it a separate feature.

[notatallshaw]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

Great, I'll reprofile with this PR. I personally wasn't imagining any of these ideas would land for 24.1.

I'd want parallel installs to be a separate feature

Agree, I'll make a seperate issue for that.

Honestly, the others I feel like I could make PRs that safely improve pip, I'm unsure about parralel installs, I think it would at a minimum carefully need to look at what current multiple installs tests there are and potentially expanding them to have a good matrix of different possibilities.

[notatallshaw]

Log a message that pip is starting to install packages

Btw, I was looking at this recently because I noticed pip does tell you it's installing packages. The specific scenario I was seeing was the following:

1. You install a large number of packages
2. You then install a large number of semi-overlapping packages

On step two this produces the following behavior:

1. Packages are resolved and pip tells you what packages it is going to install
2. Pip then quickly uninstalls old packages, filling up the screen
3. There is a long wait with no update on the screen while pip is installing
4. Pip then lists all packages it installed

The real world situaiton this happens is installing large machine learning packages, particularly because you install a bunch of packages from the pytorch index, and then install a bunch of packages from pypi.

I think there are a couple of possible solutions:

1. Re-order or add additional messages, e.g. move or add and "install" message after the uninstalls have completed
2. Add progress bars to both uninstalling packages and installing packages, so it's clear pip is doing things

I will take a look at PRs when I have a chance.

[ichard26]

Caching the result of utils.compatibility_tags.get_supported() in the resolver factory should be another easy win¹ (~3% or 4s in the example above)

pip/src/pip/_internal/resolution/resolvelib/factory.py

Lines 608 to 612 in 86b8b23

	return self._wheel_cache.get_cache_entry(
	link=link,
	package_name=name,
	supported_tags=get_supported(),
	)

I'll submit a PR when I get the chance.

Footnotes

1. I strongly suspect that get_supported() is only "slow" (as in, 1-5ms) on Linux due to the large amount of supported tags per system. ↩