Ondemand loses connection with ongoing Jupyter session

[ndjones]

After some period of time, ondemand lost its handle on a running jupyter session, reporting:

Jupyter Lab - staging (jupyterlab-t0x4kmix) Undetermined

Your session has entered a bad state. Feel free to contact support for further information.

[chrisdjscott]

When this happens does the app still show up as running under interactive sessions: https://163-7-144-39.sslip.io/pun/sys/dashboard/batch_connect/sessions ?

[ndjones]

the error above is copied from the session box for the related Jupyter session under interactive sessions. I didn't copy all the text, just the two indications of an issue.

[lbrick]

I believe I have recreated the issue.

It looks related to the OIDC token expiration, if I log in and out then click the "restart web server" once or twice the token gets reissued and the card returns to green.

I dont believe this is related to Jupyter as i have a feeling you can also recreate this with any app running

[chrisdjscott]

did we have this issue on the training environment too and had to adjust some timeout values in keycloak?

[chrisdjscott]

maybe this bit https://github.com/nesi/training-environment/blob/main/roles/keycloak/tasks/main.yml#L151-L154

[lbrick]

Unsure if we did have the issue on the training environment

The Keycloak we are now using is controlled/maintained by CNI so will need to discuss with them if we can increase the SSO timeouts for tokens

[lbrick]

Yuriy and I have managed to resolve this.

We are investigating a few other options, however this is fixed in the current test environment and is related to OOD => kubernetes configuration more then the OOD application itself

I will leave this issue open for now till Yuriy and I have tried a few other ways.