Bump to recent versions, and make bootstrap.py update to those when run

[consideRatio]

PR summary

I saw quite a bit of very specific pinnings, all the way to the patch version. I figure that is problematic so I made it instead pin the latest major version, or in case no major version has been released to the first minor version.

I updated:

• the tljh package dependencies installed with the root python environment
• the hub environment requirements, installed by tljh
• the user environment requirements, installed by tljh
• the ensure_pip/conda_package functions to accept a flag to make packages be upgraded to resolve Running pip install oauthenticator==1.* should include the --upgrade flag - right? #732 introduced by not pinning strictly any more
  • --upgraded added to pip install
  • --update-deps and --update-spec added to conda install
    My intention is that this should update the package and its dependencies, just like I expect pip install --upgrade works.

Related

• Install error Ubuntu 20.04 header pyconfig.h not found #708 - A failure related to chardet==3.0.4
• tljh-config ContextualVersionConflict - chardet 4 vs jupyterhub-traefik-proxy #642 - A failure related to chardet version 4
• Figure out if and why we should install chardet #737 - A dedicated issue regarding chardet pinning.
• Closes Running pip install oauthenticator==1.* should include the --upgrade flag - right? #732
• Closes Ensure the jupyterhub version in hub/user environment matches #736

[minrk]

Looks great to me. I'll defer to @GeorgianaElena on the chardet question.

For fully frozen environments, it would be best to use pip-tools or conda lockfiles to separate the 'actual' loosely defined dependencies from the fully frozen ones. I think it's good to move away from manually managed exact pins.

[consideRatio]

@yuvipanda @minrk I've rebased this PR on #721 now - could you re-review it?

Since @minrk this PR approved, I've added the commit 1dab737 that makes us do pip install --upgrade when installing some things that are no longer pinned to a specific version, but instead pinned to something like oauthenticator==1.*.

[yuvipanda]

I think this is good to go once the tests pass

[consideRatio]

@yuvipanda I did conda install --update-deps --update-specs but the flags were exclusive. I wanted to mirror pip install --upgrade but I wasn't sure how to accomplish it. Do you have ideas on what the equivalent variant would be?

To clarify, my assumption is that if I do pip install --upgrade package==1.* that has some dependencies, but the package and the dependencies will get updated. My assumption about conda install is that only the package will be updated unless some flags are passed, but I'm very uncertain about conda in general.

Conda flags details

[--freeze-installed | --update-deps | -S | --update-all | --update-specs]

  --freeze-installed, --no-update-deps
                        Do not update or change already-installed dependencies.
  --update-deps         Update dependencies.
  -S, --satisfied-skip-solve
                        Exit early and do not run the solver if the requested specs are satisfied. Also skips aggressive updates as configured by 'aggressive_update_packages'. Similar to the
                        default behavior of 'pip install'.
  --update-all, --all   Update all installed packages in the environment.
  --update-specs        Update based on provided specifications.

[consideRatio]

I'm thinking that either we want to have --update-specs or no flag at all for conda. Hmmm...

I think --update-deps is the right call, I'm not 100%, but I think so.

EDIT: aaargh well it seems like dependencies are updated by default. I'm not changing conda to update anything explicitly with a flag.

[consideRatio]

Final summary

@yuvipanda, I made pip install --upgrade be used now that we don't declare exact versions, but instead declare versions like oauthenticator==1.*. I reverted changes to the mamba install command regarding something equivalent to the pip --upgrade flag, as conda/mamba seem to update the deps by default.

[minrk]

Not for this PR, but we we probably don't want to call pip with --no-cache-dir. This is common as a space optimization for keeping images small, but isn't the right thing for any pip install on a persistent VM.

[consideRatio]

Oh note I retained --no-cache-dir, but I can remove it part of this PR, makes sense to me to remove!

[minrk]

Actually, now that I see you've added --no-cache-dir here, let's not do that. And you can remove it above, as well.

[consideRatio]

Oh, you are right, I added it here! Woops!

[minrk]

Minor comment that we shouldn't use --no-cache-dir with pip, but LGTM!

[consideRatio]

With recent approvals (and fix of removing --no-cache-dir), ill go for a merge on this!

Thank you @minrk and @yuvipanda for the review efforts!!