Upgrading Ubuntu from 18.04 to Ubuntu 20.04 made JupyterHub unable to start

[shakthimaan]

Bug description

I had installed tljh using the documentation at https://tljh.jupyter.org/en/latest/install/custom-server.html on Ubuntu 18.04. After upgrading the system to Ubuntu 20.04, tljh fails to start.

Expected behaviour

$ sudo systemctl status jupyterhub

should say the service is running.

Actual behaviour

$ sudo journalctl -u jupyterhub
...
"/opt/tljh/hub/bin/python3: Error while finding module specification for 'jupyterhub.app' (ModuleNotFoundError: No module named 'jupyterhub'). 

Also:

$ sudo tljh-config show
ModuleNotFoundError: no module named 'pkg_resources. 

I see the folder /opt/tljh/hub/lib/python3.6/site-packages/tljh/, but, Ubuntu 20.04 has upgraded python to 3.8.

I tried to re-install, but, the bootstrap.py script fails again:

$ curl -L https://tljh.jupyter.org/bootstrap.py | sudo -E python3 - --admin admin
ModuleNotFoundError: No module named pip. subprocess.CalledProcessError: Command '['/opt/tljh/hub/bin/pip', 'install', '--upgrade', 'pip==20.0.*']' returned non-zero exit status 1

The traefix service is running fine though.

How to reproduce

Setup tljh on Ubuntu 18.04, and then upgrade the distribution to Ubuntu 20.04. We see that the jupyterhub service fails to start.

Your personal set up

• OS: Ubuntu
• Version: 20.04
• Configuration: Linux 4.4.0-47-generic Linux kernel
• Architecture: x86_64

Please let me know as to how to proceed further. Thanks!

[jawiv]

Has anyone come up with an official fix for those of us who foolishly updated to 20.04?

[evilmav]

Has anyone come up with an official fix for those of us who foolishly updated to 20.04?

For anyone looking for a temporary fix:
Probably the best way would have been to use venv --upgrade on hub environment, but it has been broken on debian forever and I could not work around it. What worked for me is (from memory, so might have typos):

1. install python3.7 from deadsnakes:

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7

2. Manually fix python3 symlink in /opt/tljh/hub/bin to point to 3.7:

$ cd /opt/tljh/hub/bin
$ rm python3
$ ln -s /usr/bin/python3.7 python3

3. Rerun TLJH installer, it automatically runs in upgrade mode and makes you end up with current version.

[szarma]

Ah, ended up in the same hole. I followed the suggestion by @evilmav but got stuck on the last part: the installer recognizes the previous deployment, runs in upgrade mode, but throws the following error

Checking if TLJH is already installed...
TLJH already installed, upgrading...
Upgrading TLJH installer...
Upgraded pip
Setup tljh package
Starting TLJH installer...
Setting up admin users
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 534, in <module>
    main()
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 504, in main
    ensure_admins(args.admin)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/installer.py", line 307, in ensure_admins
    config = yaml.load(f)
  File "/opt/tljh/hub/lib/python3.6/site-packages/ruamel/yaml/main.py", line 231, in load
    return constructor.get_single_data()
  File "/opt/tljh/hub/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 90, in get_single_data
    node = self.composer.get_single_node()
  File "/opt/tljh/hub/lib/python3.6/site-packages/ruamel/yaml/composer.py", line 75, in get_single_node
    document = self.compose_document()
  File "/opt/tljh/hub/lib/python3.6/site-packages/ruamel/yaml/composer.py", line 96, in compose_document
    node = self.compose_node(None, None)
  File "/opt/tljh/hub/lib/python3.6/site-packages/ruamel/yaml/composer.py", line 115, in compose_node
    anchor = event.anchor
AttributeError: 'NoneType' object has no attribute 'anchor'

I tried ... sudo -E python3.7 ..., ...python3.6..., and just python3 but the output is exactly the same.

Any help is appreciated.

PS: Oh, and thanks for tljh, It's great.

[consideRatio]

Thank you for documenting this experience, I think it is absolutely in scope to have TLJH be able to recover after an upgrade from one Ubuntu LTS version to the next.

This issue relates to #81 about upgrading the version of TLJH itself to some degree.

[asmusod]

Any news on this? :D

[vsisl]

Hello everyone, any news regarding this topic? It seems that I ran into the same pothole while trying to update from ubuntu 18.04 to 22.04. #957

[janikmu]

I had the same problem upgrading from Ubuntu 20.04 to 22.04.
What solved it for me was to fix the virtual hub environment from the upgraded system Python installation:
sudo python3 -m venv --upgrade --upgrade-deps /opt/tljh/hub

After that I was also able to carry out the TLJH upgrade from 0.2.0 to 1.0.0 using the bootstrap script, which I thought could be a way to fix things but was previously getting stuck on ModuleNotFoundError: No module named 'pip'.