404 Trying to connect to single-user notebook page

[ivhacks]

(Note: I've removed my username, domain name, and indentifiable IP addresses just to be safe.)

Bug description

After the SLURM process is started, the user is directed to a 404 page.

Expected behaviour

• On /hub/spawn, I click the spawn button.
• I see a loading bar while batchspawner waits for the process to start.
• I'm brought to /user/[email protected]/tree and I see a list of my notebooks.

Actual behaviour

• After the loading bar, I'm brought to a 404 page.

How to reproduce

This happens every time I try to use Jupyterhub. Probably system-specific, because it's unavoidable.

Setup

• OS:
  CentOS 7

• Version(s):
  Jupyterhub 1.4.1, Python 3.9.4, Conda 4.10.1, Batchspawner 1.1.0

Other relevant info:

• I have access to a working equivalent of this setup (using an older version of Jupyterhub). Most of the GET requests that 404 on the new version to /user/[email protected]/[path] on the working version, but the new version lacks the /user/[email protected] part.

• The working version sets cookies in the way described here: https://jupyterhub.readthedocs.io/en/0.7.2/howitworks.html. The new version sets them differently.

• A lot of 302s occur in response to the get requests, promptng my browser to try the specified location, which 404s.

• I have attachted a HAR file showing the interaction between my browser (Firefox 89.0) and the site up to when the problem occurs.

Logs

Full environment

XDG_SESSION_ID=182
HOSTNAME=jupyterhub-test
SELINUX_ROLE_REQUESTED=
__LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/modulefiles:1
TERM=xterm-256color
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=aaa.aaa.aaa.aaa 63602 22
CONDA_SHLVL=1
CONDA_PROMPT_MODIFIER=(jupyterenv)
SELINUX_USE_CURRENT_RANGE=
LMOD_PKG=/opt/ohpc/admin/lmod/lmod
OLDPWD=/home/iv
LMOD_VERSION=7.8.1
SSH_TTY=/dev/pts/0
USER=iv
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
CONDA_EXE=/usr/local/anaconda3/bin/conda
_CE_CONDA=
LMOD_PREPEND_BLOCK=normal
_ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcyIsfQ==
MAIL=/var/spool/mail/iv
PATH=/home/iv/.conda/envs/jupyterenv/bin:/usr/local/anaconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/iv/.local/bin:/home/iv/bin
CONDA_PREFIX=/home/iv/.conda/envs/jupyterenv
LMOD_SETTARG_CMD=:
PWD=/home/iv/jupyterhub
LANG=en_US.UTF-8
MODULEPATH=/opt/ohpc/pub/modulefiles
_ModuleTable_Sz_=1
SELINUX_LEVEL_REQUESTED=
LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod
_CE_M=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/iv
BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash
CONDA_PYTHON_EXE=/usr/local/anaconda3/bin/python
LOGNAME=iv
SSH_CONNECTION=aaa.aaa.aaa.aaa 63602 aaa.aaa.aaa.aaa 22
MODULESHOME=/opt/ohpc/admin/lmod/lmod
CONDA_DEFAULT_ENV=jupyterenv
LMOD_SETTARG_FULL_SUPPORT=no
LESSOPEN=||/usr/bin/lesspipe.sh %s
LMOD_FULL_SETTARG_SUPPORT=no
XDG_RUNTIME_DIR=/run/user/1002
LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec
LMOD_COLORIZE=no
BASH_FUNC_module()=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
BASH_FUNC_ml()=() {  eval $($LMOD_DIR/ml_cmd "$@")
}
_=/usr/bin/env

conda list output

(jupyterenv) [iv@jupyterhub-test jupyterhub]$ conda list
# packages in environment at /home/iv/.conda/envs/jupyterenv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
alembic                   1.6.4              pyhd8ed1ab_0    conda-forge
anyio                     2.2.0            py39h06a4308_1
argon2-cffi               20.1.0           py39h27cfd23_1
async_generator           1.10               pyhd3eb1b0_0
attrs                     21.2.0             pyhd3eb1b0_0
babel                     2.9.1              pyhd3eb1b0_0
backcall                  0.2.0              pyhd3eb1b0_0
batchspawner              1.1.0                    pypi_0    pypi
bleach                    3.3.0              pyhd3eb1b0_0
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py39h27cfd23_1003
ca-certificates           2020.12.5            ha878542_0    conda-forge
certifi                   2020.12.5        py39hf3d152e_1    conda-forge
certipy                   0.1.3                      py_0    conda-forge
cffi                      1.14.5           py39h261ae71_0
chardet                   4.0.0           py39h06a4308_1003
configurable-http-proxy   4.3.2           node12_hac6e442_0    conda-forge
cryptography              3.4.7            py39hd23ed53_0
decorator                 5.0.9              pyhd3eb1b0_0
defusedxml                0.7.1              pyhd3eb1b0_0
elementpath               2.2.2                    pypi_0    pypi
entrypoints               0.3              py39h06a4308_0
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10               pyhd3eb1b0_0
importlib-metadata        3.10.0           py39h06a4308_0
importlib-resources       5.1.2                    pypi_0    pypi
importlib_metadata        3.10.0               hd3eb1b0_0
iniconfig                 1.1.1                    pypi_0    pypi
ipykernel                 5.3.4            py39hb070fc8_0
ipython                   7.22.0           py39hb070fc8_0
ipython_genutils          0.2.0              pyhd3eb1b0_1
jedi                      0.17.2           py39h06a4308_1
jinja2                    3.0.0              pyhd3eb1b0_0
json5                     0.9.5                      py_0
jsonschema                3.2.0                      py_2
jupyter-packaging         0.7.12             pyhd3eb1b0_0
jupyter-saml2authenticator 0.1.0.dev0               pypi_0    pypi
jupyter_client            6.1.12             pyhd3eb1b0_0
jupyter_core              4.7.1            py39h06a4308_0
jupyter_server            1.4.1            py39h06a4308_0
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyterhub                1.4.1            py39hf3d152e_0    conda-forge
jupyterhub-base           1.4.1            py39hf3d152e_0    conda-forge
jupyterlab                3.0.14             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_server         2.4.0              pyhd3eb1b0_0
krb5                      1.18.2               h173b8e3_0
ld_impl_linux-64          2.33.1               h53a641e_7
libcurl                   7.71.1               h20c2e04_1
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.1.0                hdf63c60_0
libsodium                 1.0.18               h7b6447c_0
libssh2                   1.9.0                hab1572f_5    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0
libuv                     1.40.0               h7b6447c_0
mako                      1.1.4              pyh44b312d_0    conda-forge
markupsafe                2.0.1            py39h27cfd23_0
mistune                   0.8.4           py39h27cfd23_1000
nbclassic                 0.2.6              pyhd3eb1b0_0
nbclient                  0.5.3              pyhd3eb1b0_0
nbconvert                 6.0.7            py39h06a4308_0
nbformat                  5.1.3              pyhd3eb1b0_0
ncurses                   6.2                  he6710b0_1
nest-asyncio              1.5.1              pyhd3eb1b0_0
nodejs                    12.19.0              hfa01f41_2    conda-forge
notebook                  6.4.0            py39h06a4308_0
oauthlib                  3.0.1                      py_0    conda-forge
openssl                   1.1.1k               h27cfd23_0
packaging                 20.9               pyhd3eb1b0_0
pamela                    1.0.0                      py_0    conda-forge
pandoc                    2.12                 h06a4308_0
pandocfilters             1.4.3            py39h06a4308_1
parso                     0.7.0                      py_0
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pip                       21.1.1           py39h06a4308_0
pluggy                    0.13.1                   pypi_0    pypi
prometheus_client         0.10.1             pyhd3eb1b0_0
prompt-toolkit            3.0.17             pyh06a4308_0
ptyprocess                0.7.0              pyhd3eb1b0_2
py                        1.10.0                   pypi_0    pypi
pycparser                 2.20                       py_2
pycurl                    7.43.0.6         py39h8cd828d_0    conda-forge
pygments                  2.9.0              pyhd3eb1b0_0
pyjwt                     2.1.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7              pyhd3eb1b0_0
pyrsistent                0.17.3           py39h27cfd23_0
pysaml2                   6.5.1                    pypi_0    pypi
pysocks                   1.7.1            py39h06a4308_0
pytest                    6.2.4                    pypi_0    pypi
python                    3.9.4                hdb3f193_0
python-dateutil           2.8.1              pyhd3eb1b0_0
python-editor             1.0.4                      py_0    conda-forge
python-json-logger        2.0.1              pyh9f0ad1d_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd3eb1b0_0
pyzmq                     20.0.0           py39h2531618_1
readline                  8.1                  h27cfd23_0
requests                  2.25.1             pyhd3eb1b0_0
ruamel.yaml               0.16.12          py39h27cfd23_1
ruamel.yaml.clib          0.2.2            py39hbd71b63_1    conda-forge
send2trash                1.5.0              pyhd3eb1b0_1
setuptools                52.0.0           py39h06a4308_0
six                       1.15.0           py39h06a4308_0
sniffio                   1.2.0            py39h06a4308_1
sqlalchemy                1.3.23           py39h27cfd23_0
sqlite                    3.35.4               hdfb4753_0
terminado                 0.9.4            py39h06a4308_0
testpath                  0.4.4              pyhd3eb1b0_0
tk                        8.6.10               hbc83047_0
toml                      0.10.2                   pypi_0    pypi
tornado                   6.1              py39h27cfd23_0
traitlets                 5.0.5              pyhd3eb1b0_0
tzdata                    2020f                h52ac0ba_0
urllib3                   1.26.4             pyhd3eb1b0_0
wcwidth                   0.2.5                      py_0
webencodings              0.5.1            py39h06a4308_1
wheel                     0.36.2             pyhd3eb1b0_0
wrapspawner               1.0.0                    pypi_0    pypi
xmlschema                 1.6.2                    pypi_0    pypi
xz                        5.2.5                h7b6447c_0
zeromq                    4.3.4                h2531618_0
zipp                      3.4.1              pyhd3eb1b0_0
zlib                      1.2.11               h7b6447c_3

Configuration

c.JupyterHub.bind_url = 'https://[jupyterhub-test.contoso.com]:8000/'
c.JupyterHub.cookie_secret_file = 'jupyterhub_cookie_secret'
c.JupyterHub.hub_ip = 'jupyterhub-test.contoso.com'
c.JupyterHub.ip = ''
c.JupyterHub.port = 8000
import batchspawner.api
c = get_config()
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 300
c.Spawner.start_timeout = 300
c.BatchSpawnerBase.req_host = 'jupyterhub-test.contoso.com'
c.BatchSpawnerBase.req_nprocs = '2'
c.BatchSpawnerBase.req_memory = '2gb'
c.BatchSpawnerBase.req_runtime = '6:00:00'
c.TorqueSpawner.state_exechost_exp = r'in-\1.jupyterhub-test.contoso.com'
c.ProfilesSpawner.profiles = [
   ('2 cores/2 GB/6 hours', '2c2gb', 'batchspawner.SlurmSpawner',
      dict(req_nprocs='2', req_queue='normal', req_runtime='6:00:00', req_memory='2gb')),
   ('2 cores/8 GB/2 hours', '2c8gb', 'batchspawner.SlurmSpawner',
      dict(req_nprocs='2', req_queue='normal', req_runtime='2:00:00', req_memory='8gb'))
   ]
c.JupyterHub.ssl_cert = '/opt/jupyterhub/jupyterhub-test.contoso.com.cer'
c.JupyterHub.ssl_key = '/opt/jupyterhub/jupyterhub-test.contoso.com.key'
from jupyter_saml2authenticator import Saml2Authenticator
c.JupyterHub.authenticator_class = Saml2Authenticator
c.Saml2Authenticator.saml2_metadata_url = *Prefer not to share*
c.Saml2Authenticator.saml2_entity_id = 'https://jupyterhub-test.contoso.com/saml2_auth/ent'
c.Saml2Authenticator.saml2_attribute_username = 'emailAddress'
c.Saml2Authenticator.login_service = "Contoso Single Sign-on"
c.Saml2Authenticator.saml2_strip_username = False
c.Saml2Authenticator.saml2_login_url = r'/saml2_auth/login'
c.Saml2Authenticator.saml2_acs_url = r'/saml2_auth/acs'

Logs

This is the log from the compute node that the single-user notebook app runs on (errors at the end are from pressing ctrl+c):

/usr/local/anaconda3/bin/jupyterhub-singleuser
[I 2021-06-12 11:24:16.326 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[I 2021-06-12 11:24:17.826 SingleUserNotebookApp handlers:78] Loading IPython parallel extension
[I 2021-06-12 11:24:20.378 LabApp] JupyterLab extension loaded from /usr/local/anaconda3/lib/python3.9/site-packages/jupyterlab
[I 2021-06-12 11:24:20.379 LabApp] JupyterLab application directory is /usr/local/anaconda3/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-06-12 11:24:20.392 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[I 2021-06-12 11:24:20.399 SingleUserNotebookApp log:189] 302 GET /user/[email protected]/ -> /user/[email protected] (@aaa.aaa.aaa.aaa) 3.08ms
[I 2021-06-12 11:24:20.409 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/[email protected]
[I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2302] http://jupyterhub-c5:34582/
[I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-06-12 11:24:20.442 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[I 2021-06-12 11:24:20.767 SingleUserNotebookApp log:189] 302 GET /user/[email protected]/ -> /user/[email protected] (@aaa.aaa.aaa.aaa) 1.52ms
[W 2021-06-12 11:24:20.992 SingleUserNotebookApp log:189] 404 GET /user/[email protected] (@aaa.aaa.aaa.aaa) 102.75ms
[I 2021-06-12 12:14:13.269 SingleUserNotebookApp log:189] 302 GET /user/[email protected]/ -> /user/[email protected] (@aaa.aaa.aaa.aaa) 2.12ms
[W 2021-06-12 12:14:13.458 SingleUserNotebookApp log:189] 404 GET /user/[email protected] (@aaa.aaa.aaa.aaa) 8.65ms
[W 2021-06-12 12:14:38.003 SingleUserNotebookApp log:189] 404 GET /user/[email protected] (@aaa.aaa.aaa.aaa) 4.26ms
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** JOB 187 ON jupyterhub-c5 CANCELLED AT 2021-06-12T14:37:50 ***
slurmstepd: error: *** STEP 187.0 ON jupyterhub-c5 CANCELLED AT 2021-06-12T14:37:50 ***
[C 2021-06-12 14:37:50.741 SingleUserNotebookApp notebookapp:1978] received signal 15, stopping
[I 2021-06-12 14:37:51.022 SingleUserNotebookApp notebookapp:2145] Shutting down 0 kernels
[I 2021-06-12 14:37:51.022 SingleUserNotebookApp notebookapp:2160] Shutting down 0 terminals
SIGTERM received
jupyterhub-singleuser ended gracefully

This is the output from running the jupyterhub command on the login node:

[I 2021-06-12 17:17:18.610 JupyterHub app:2463] Running JupyterHub version 1.4.1
[I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Authenticator: jupyter_saml2authenticator.auth.Saml2Authenticator
[I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.1
[I 2021-06-12 17:17:18.967 JupyterHub app:1534] Loading cookie_secret from /home/iv/jupyterhub/jupyterhub_cookie_secret
[I 2021-06-12 17:17:19.326 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-06-12 17:17:19.328 JupyterHub app:1808] No admin users, admin interface will be unavailable.
[W 2021-06-12 17:17:19.328 JupyterHub app:1809] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-06-12 17:17:19.328 JupyterHub app:1838] Not using allowed_users. Any authenticated user will be allowed.
[I 2021-06-12 17:17:19.451 JupyterHub app:2530] Initialized 0 spawners in 0.006 seconds
[I 2021-06-12 17:17:19.461 JupyterHub proxy:703] Starting proxy @ https://jupyterhub-test.contoso.com:8000/
17:17:25.092 [ConfigProxy] info: Proxying https://jupyterhub-test.contoso.com:8000 to (no default)
17:17:25.187 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
17:17:25.679 [ConfigProxy] info: 200 GET /api/routes
[I 2021-06-12 17:17:25.712 JupyterHub app:2778] Hub API listening on http://jupyterhub-test.contoso.com:8081/hub/
17:17:25.714 [ConfigProxy] info: 200 GET /api/routes
[I 2021-06-12 17:17:25.714 JupyterHub proxy:347] Checking routes
[I 2021-06-12 17:17:25.714 JupyterHub proxy:432] Adding route for Hub: / => http://jupyterhub-test.contoso.com:8081
17:17:25.717 [ConfigProxy] info: Adding route / -> http://jupyterhub-test.contoso.com:8081
17:17:25.718 [ConfigProxy] info: Route added / -> http://jupyterhub-test.contoso.com:8081
17:17:25.719 [ConfigProxy] info: 201 POST /api/routes/
[I 2021-06-12 17:17:25.720 JupyterHub app:2853] JupyterHub is now running at https://jupyterhub-test.contoso.com:8000/
[I 2021-06-12 17:17:56.716 JupyterHub log:189] 302 GET / -> /hub/ (@aaa.aaa.aaa.aaa) 43.69ms
[I 2021-06-12 17:17:56.951 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn ([email protected]@aaa.aaa.aaa.aaa) 122.05ms
[I 2021-06-12 17:17:57.202 JupyterHub log:189] 200 GET /hub/spawn ([email protected]@aaa.aaa.aaa.aaa) 140.49ms
[I 2021-06-12 17:17:58.929 JupyterHub provider:574] Creating oauth client jupyterhub-user-aerobinsoniv%40contoso.com
[I 2021-06-12 17:17:59.148 JupyterHub batchspawner:262] Spawner submitting job using sudo -E -u [email protected] sbatch --parsable
[I 2021-06-12 17:17:59.148 JupyterHub batchspawner:263] Spawner submitted script:
    #!/bin/bash
    #SBATCH --output=/home/[email protected]/jupyterhub_slurmspawner_%j.log
    #SBATCH --job-name=spawner-jupyterhub
    #SBATCH --chdir=/home/[email protected]
    #SBATCH --export=PATH,CONDA_DEFAULT_ENV,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,USER,HOME,SHELL
    #SBATCH --get-user-env=L
    #SBATCH --time=6:00:00
    #SBATCH --mem=2gb
    #SBATCH --cpus-per-task=2


    set -euo pipefail

    trap 'echo SIGTERM received' TERM

    which jupyterhub-singleuser
    srun batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0
    echo "jupyterhub-singleuser ended gracefully"

[I 2021-06-12 17:17:59.419 JupyterHub batchspawner:266] Job submitted. cmd: sudo -E -u [email protected] sbatch --parsable output: 188
[I 2021-06-12 17:17:59.874 JupyterHub log:189] 302 POST /hub/spawn -> /hub/spawn-pending/[email protected] ([email protected]@aaa.aaa.aaa.aaa) 1009.44ms
[I 2021-06-12 17:18:00.049 JupyterHub pages:402] [email protected] is pending spawn
[I 2021-06-12 17:18:00.061 JupyterHub log:189] 200 GET /hub/spawn-pending/[email protected] ([email protected]@aaa.aaa.aaa.aaa) 32.47ms
[I 2021-06-12 17:18:03.978 JupyterHub api:23] key is port, value 41166
[I 2021-06-12 17:18:03.982 JupyterHub api:24] hasattr(spawner, key) is True
[I 2021-06-12 17:18:03.986 JupyterHub log:189] 201 POST /hub/api/batchspawner ([email protected]@aaa.aaa.aaa.aaa) 40.37ms
[I 2021-06-12 17:18:04.078 JupyterHub batchspawner:420] Notebook server job 188 started at jupyterhub-c5:41166
[I 2021-06-12 17:18:08.636 JupyterHub base:909] User [email protected] took 9.766 seconds to start
[I 2021-06-12 17:18:08.639 JupyterHub proxy:285] Adding user [email protected] to proxy /user/[email protected]/ => http://jupyterhub-c5:41166
[I 2021-06-12 17:18:08.643 JupyterHub log:189] 200 GET /hub/api (@aaa.aaa.aaa.aaa) 1.39ms
17:18:08.668 [ConfigProxy] info: Adding route /user/[email protected] -> http://jupyterhub-c5:41166
17:18:08.669 [ConfigProxy] info: Route added /user/[email protected] -> http://jupyterhub-c5:41166
17:18:08.670 [ConfigProxy] info: 201 POST /api/routes/user/[email protected]
[I 2021-06-12 17:18:08.718 JupyterHub log:189] 200 POST /hub/api/users/[email protected]/activity ([email protected]@aaa.aaa.aaa.aaa) 47.25ms
[I 2021-06-12 17:18:08.719 JupyterHub users:677] Server [email protected] is ready
[I 2021-06-12 17:18:08.720 JupyterHub log:189] 200 GET /hub/api/users/[email protected]/server/progress ([email protected]@aaa.aaa.aaa.aaa) 8398.65ms
[I 2021-06-12 17:18:08.880 JupyterHub log:189] 302 GET /hub/spawn-pending/[email protected] -> /user/[email protected]/ ([email protected]@aaa.aaa.aaa.aaa) 13.14ms
[I 2021-06-12 17:18:09.415 JupyterHub log:189] 302 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb -> /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@aaa.aaa.aaa.aaa) 1.53ms
[I 2021-06-12 17:18:09.418 JupyterHub log:189] 302 GET /static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd -> /hub/static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd (@aaa.aaa.aaa.aaa) 0.93ms
[I 2021-06-12 17:18:09.422 JupyterHub log:189] 302 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a -> /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@aaa.aaa.aaa.aaa) 0.95ms
[I 2021-06-12 17:18:09.431 JupyterHub log:189] 302 GET /custom/custom.css -> /hub/custom/custom.css (@aaa.aaa.aaa.aaa) 1.63ms
[I 2021-06-12 17:18:09.432 JupyterHub log:189] 302 GET /static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 -> /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@aaa.aaa.aaa.aaa) 1.63ms
[I 2021-06-12 17:18:09.435 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 0.93ms
[I 2021-06-12 17:18:09.531 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 0.99ms
[I 2021-06-12 17:18:09.545 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 2.38ms
[I 2021-06-12 17:18:09.546 JupyterHub log:189] 302 GET /static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 -> /hub/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (@aaa.aaa.aaa.aaa) 2.45ms
[W 2021-06-12 17:18:09.548 JupyterHub log:189] 404 GET /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@aaa.aaa.aaa.aaa) 3.09ms
[W 2021-06-12 17:18:09.637 JupyterHub log:189] 404 GET /hub/static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd (@aaa.aaa.aaa.aaa) 1.63ms
[W 2021-06-12 17:18:09.640 JupyterHub log:189] 404 GET /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@aaa.aaa.aaa.aaa) 1.39ms
[W 2021-06-12 17:18:09.729 JupyterHub log:189] 404 GET /hub/custom/custom.css ([email protected]@aaa.aaa.aaa.aaa) 57.21ms
[W 2021-06-12 17:18:09.733 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 2.69ms
[W 2021-06-12 17:18:09.734 JupyterHub log:189] 404 GET /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@aaa.aaa.aaa.aaa) 3.16ms
[W 2021-06-12 17:18:09.735 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 3.33ms
[W 2021-06-12 17:18:09.779 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 1.23ms
[I 2021-06-12 17:18:09.854 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 1.19ms
[W 2021-06-12 17:18:09.961 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 1.05ms
[I 2021-06-12 17:18:10.067 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 1.01ms
[W 2021-06-12 17:18:10.171 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 1.02ms
[I 2021-06-12 17:18:10.278 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 0.99ms
[W 2021-06-12 17:18:10.383 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 0.97ms
[I 2021-06-12 17:18:10.521 JupyterHub log:189] 302 GET /static/components/jquery/jquery.min.js?v=20210612171805 -> /hub/static/components/jquery/jquery.min.js?v=20210612171805 (@aaa.aaa.aaa.aaa) 0.95ms
[W 2021-06-12 17:18:10.625 JupyterHub log:189] 404 GET /hub/static/components/jquery/jquery.min.js?v=20210612171805 (@aaa.aaa.aaa.aaa) 1.41ms
^C[C 2021-06-12 17:18:15.594 JupyterHub app:2941] Received signal SIGINT, initiating shutdown...
[I 2021-06-12 17:18:15.595 JupyterHub app:2582] Cleaning up single-user servers...
[I 2021-06-12 17:18:15.595 JupyterHub proxy:772] Cleaning up proxy[4430]...
17:18:15.607 [ConfigProxy] warn: Terminated
[I 2021-06-12 17:18:15.809 JupyterHub batchspawner:431] Stopping server job 188
[I 2021-06-12 17:18:15.828 JupyterHub batchspawner:316] Cancelling job 188: sudo -E -u [email protected] scancel 188
[I 2021-06-12 17:18:18.195 JupyterHub app:2614] ...done

[Jupyterhub.har.zip](https://github.com/jupyterhub/batchspawner/files/6673895/Jupyterhub.har.zip)

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[jbaksta]

I find that usually the 404 on the spawn has something to do with a missing environment variable that describes one of the communication paths/routes. We've got some add-on scripts that attempt to verify that the variables are set and if not, set them. The following are ones that I've made sure are accessible in our environment to work right (and this can be a little tricky as every once in a while, we had some environment sanitization for various reasons.

JUPYTERHUB_API_URL
JUPYTERHUB_BASE_URL
JUPYTERHUB_CLIENT_ID
JUPYTERHUB_OAUTH_CALLBACK_URL
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX
JUPYTERHUB_USER

[ivhacks]

Hi @jbaksta , thanks for your response, this looks very promising!
How did you figure out what values to use in your setup?

[jbaksta]

Well, I had a test cluster set up w/ Docker basically that was extremely vanilla Slurm installation when I did this. Then I just started comparing the outputs of the environment already knowing that we didn't allow many environment variables to be passed to the jobs because of OS differences (SLES submission host, CentOS batch nodes). So I hacked on our batch_script to eventually make it look like my vanilla job instance. Most of the variables rely on some notion of your site setup and the jupyterhub API routes I guess. Some of them are just proper combinations of the others; some are something you can probably regenerate from your jupyterhub_config.py

[ivhacks]

I just logged into the working production server and looked at the environment variables. Hardcoding JUPYTERHUB_SERVICE_PREFIX to "/user/[my email]" on line 805 of spawner.py (for the benefit of future readers) caused it to start working. Obviously this isn't a final fix, but now that the cause of the issue has been isolated I can solve it. Thank you so much, I have been trying to figure this out for months!!

[jbaksta]

Under normal circumstances, I'd think it should work without modifications. But often times, this ends up being something missing in the keepvars or something I think you can add to the keepvars_extra. We override our batch_script elements quite heavily and based on specific clusters that are being submitted to and that's when you find things missing depending on how the batch system is configured. Regardless, happy to see that gets you farther.

[ivhacks]

Alas, for software like this meant for HPC setups with significant user configuration expected, "normal circumstances" seem to be mostly theoretical.... Anyway, thanks for the additional info.

Since it's unclear if this is a real bug or just user error, I'll leave it up to you and the other Batchspawner devs whether to close this issue.

[basvandervlies]

I just installed Jupyterhub 1.4.1, batchspawner 1.1.0 and wrapspawner and encounterd the same http error. I had to set:

if [[ -z ${JUPYTERHUB_SERVICE_PREFIX} ]]
then
    export JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}
fi

I our previous environment with HUB 1.1.0, and batchspawner/wrapspawner installed from git I did not encounter this problems

[ivhacks]

That's the exact same thing I did as my final fix, just in Bash instead of Python. So maybe it isn't just me. @basvandervlies are you willing to provide more information about your setup?

[basvandervlies]

@aerobinsonIV of course. We have HPC cluster we run multiple jupyterhub setups for course/training and production. For each jupyterhub setup we can specify which version/software must be used and we generate the config files based on templates and json data. For this we have written a services framework:

• https://github.com/basvandervlies/cf_surfsara_lib
• https://github.com/basvandervlies/cf_surfsara_lib/blob/master/doc/services/jupyterhub.md

This framework will generate the apache, systemd unit files , jupyterhub start script and the jupyterhub configuration file.

I am now testing the jupytherhub 14.1 version with batchspawner and wrapspawner and encountered that it would not start . That is fixed by the patch from @jbaksta. After that we got 404 errors. So I added the above bash statement to the batch script that is defined in the jupyterhub config.

c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH -o "jupyterhub-2021-%j.out"
#SBATCH -e "jupyterhub-2021-%j.error"
#SBATCH --job-name=jupyterhub-2021-singleuser
#SBATCH -D {{homedir}}
#SBATCH --get-user-env=L
{% if partition  %}#SBATCH --partition={{partition}}
{% endif %}{% if runtime    %}#SBATCH --time={{runtime}}
{% endif %}{% if memory     %}#SBATCH --mem={{memory}}
{% endif %}{% if nprocs     %}#SBATCH --cpus-per-task={{nprocs}}
{% endif %}{% if reservation%}#SBATCH --reservation={{reservation}}
{% endif %}{% if options    %}#SBATCH {{options}}{% endif %}

trap 'echo SIGTERM received' TERM

## Set the User writeable jupyter notebook config directory
#
export JUPYTER_CONFIG_DIR=${HOME}/.jupyter-2021

## Always clean the module environment
#
module purge
if [[ -f ${JUPYTER_CONFIG_DIR}/env ]]
then
    source ${JUPYTER_CONFIG_DIR}/env
else
    {{prologue}}
fi

## Some batch settings before starting the notebook
#
export LMOD_CMD=/hpc/eb/modules-4.0.0/libexec/modulecmd.tcl

if [[ -x ${JUPYTER_CONFIG_DIR}/script ]]
then
    ${JUPYTER_CONFIG_DIR}/script
fi

## jupyter notebook extensions enable/disable/remove
jupyter nbextension enable --py jupyterlmod --user

## jupyter server extensions enable/disable
#
jupyter serverextension enable --py jupyterlmod --user
jupyter serverextension enable jupyter_server_proxy --user

if [[ -z ${JUPYTERHUB_SERVICE_PREFIX} ]]
then
    export JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}
fi
echo "JUPYTERHUB_API_URL = ${JUPYTERHUB_API_URL}"
echo "JUPYTERHUB_BASE_URL = ${JUPYTERHUB_BASE_URL}"
echo "JUPYTERHUB_CLIENT_ID = ${JUPYTERHUB_CLIENT_ID}"
echo "JUPYTERHUB_OAUTH_CALLBACK_URL = ${JUPYTERHUB_OAUTH_CALLBACK_URL}"
echo "JUPYTERHUB_SERVER_NAME = ${JUPYTERHUB_SERVER_NAME}"
echo "JUPYTERHUB_SERVICE_PREFIX = ${JUPYTERHUB_SERVICE_PREFIX}"
echo "JUPYTERHUB_USER = ${JUPYTERHUB_USER}"

which batchspawner-singleuser
which jupyterhub-singleuser
#{% if srun %}{{srun}} {% endif %} {{cmd}}
# {{cmd}}   --NotebookApp.terminado_settings="{'shell_command': ['bash']}" --notebook-dir=${notebook_dir}
{{cmd}}   --NotebookApp.terminado_settings="{'shell_command': ['bash']}"

echo "jupyterhub-singleuser ended gracefully"
{{epilogue}}