Spawner event stream malfunctions :/

[consideRatio]

astronomy-commons/genesis-jupyterhub-automator#5

[bitnik]

@consideRatio I just tested jupyterhub-0.9.0-alpha.1.060.6698eb9 locally and I get the same error, I got many "Server requested" message. And in the hub pod, this error was occurring many times:

[E 2019-11-28 14:57:37.018 JupyterHub web:1788] Uncaught exception GET /hub/api/users/dummy/server/progress (172.17.0.1)
    HTTPServerRequest(protocol='http', host='192.168.99.100:31212', method='GET', uri='/hub/api/users/dummy/server/progress', version='HTTP/1.1', remote_ip='172.17.0.1')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 597, in get
        await self.send_event(event)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 509, in send_event
        self.write('data: {}\n\n'.format(json.dumps(event)))
      File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/usr/lib/python3.6/json/encoder.py", line 180, in default
        o.__class__.__name__)
    TypeError: Object of type 'V1Event' is not JSON serializable

I think the problem is with raw_event object in event ( https://github.com/jupyterhub/kubespawner/blob/0.11.0/kubespawner/spawner.py#L1631). When I comment that line, events work as expected.

[consideRatio]

Excellent debugging work!!! Hmm, I dont come up with a clear idea on what we should do to resolve the situation...

[mjuric]

I see the same messages as @bitnik -- nice detective work!

[consideRatio]

Hmmm perhaps it relates to using a post-1.0.0 jupyterhub commit, and that this was tested to work with kubespawner only on 1.0.0, so makes the breaking change part of jh perhaps?

[consideRatio]

So raw_event is new for kubespawner 0.11.0: https://github.com/jupyterhub/kubespawner/blob/master/docs/source/changelog.md#new

And I also recently made it test against jupyterhub master, so its not something magically new in jupyterhub master.

It simply wasn't covered in the test by kubespawner.

import json
from kubernetes.client import V1Event
event = V1Event(involved_object="arne", metadata="kalle")
with open("/tmp/test", "w") as f:
    json.dump(e, f)

# results in...
# TypeError: Object of type 'V1Event' is not JSON serializable
# If we on the other hand would do json.dump(e.to_dict(), f)
# that would work!

Or not...

This is required as part of the solution then I figure: https://code-maven.com/serialize-datetime-object-as-json-in-python - we need to make sure nested datetime objects in the V1Event is serializable, already in KubeSpawners spawner.py

[mjuric]

FWIW, I'm still seeing this issue with jupyterhub-0.9.0-alpha.1.085.6ccbe82 chart (which has the jupyterhub/k8s-hub:0.9.0-alpha.1.071.b0d70c2 hub image) -- not sure if I'm doing something wrong?

Relevant part of the log:

[E 2019-12-06 19:15:21.451 JupyterHub web:1788] Uncaught exception GET /hub/api/users/mjuric/server/progress (10.138.38.196)
    HTTPServerRequest(protocol='https', host='hub.alerts.wtf', method='GET', uri='/hub/api/users/mjuric/server/progress', version='HTTP/1.1', remote_ip='10.138.38.196')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 597, in get
        await self.send_event(event)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 509, in send_event
        self.write('data: {}\n\n'.format(json.dumps(event)))
      File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/usr/lib/python3.6/json/encoder.py", line 180, in default
        o.__class__.__name__)
    TypeError: Object of type 'V1Event' is not JSON serializable

[consideRatio]

Thanks, I've verified everything you say to be correct. I visited https://jupyterhub.github.io/helm-chart/ and downloaded the latest helm chart, alpha.1.085, and concluded that its values.yaml indeed used the alpha.1.071 image as could make sense.

docker run -it --rm jupyterhub/k8s-hub:0.9.0-alpha.1.071.b0d70c2 bash

jovyan@3b3b38d8368f:/srv/jupyterhub$ python3 -c "import kubespawner.spawner; print(kubespawner.spawner.__version__)"
0.11.0

We need to use 0.11.1 because the bug is fixed there. The issue is apparently that I've falsely assumed that a requirements with jupyterhub-kubespawner==0.11 would imply ==0.11.* but it didn't, it implied 0.11.0.

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/images/hub/requirements.txt#L52

[manics]

The confusion probably comes from pip and conda behaving differently (conda will install the latest .*).

You should get the same behaviour with pip using kubespawner>=0.11,<0.12.

[consideRatio]

Ah, thanks @manics, I'll submit a PR soon! Got stuck fixing ltiauthenticator that had conflicting dependencies.

[mjuric]

Happy to confirm event stream now works for me on v1.16 with the latest Helm chart. Thanks for all the hard work!