Feedback on implementing the dss commands with timeout

[NohaIhab]

Bug Description

Implementing the dss initialize and dss create-notebook commands with timeout affects the dss user experience, where the timeout failure logs commands are sometimes not helpful. For example as mentioned here for dss initialize and here for dss create-notebook.

It's necessary to consider, with the help of the UX team, whether the commands should have timeout and how to make the status of dss visible to users.

To Reproduce

Install dss from source and run the commands

KUBECONFIG=~/.kube/config
pip install .
dss initialize --kubeconfig $KUBECONFIG

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5512.

This message was autogenerated

[ca-scribner]

Two options come to mind:

Remove the timeout entirely - hold the terminal until commands complete or the user cancels

Why do we need the timeout? All it does really is set a fixed time after which the command dies. Users can already break the command themselves with ctrl+c.

What if we remove the timeout but also handle the SIGINT from ctrl+c, and maybe print some debug info if we handle SIGINT? That avoids the issue of setting a timeout differently for different people

Run commands async by default but add --wait for synchronous mode

We can make dss initialize (and other commands) being async by default (they do not wait for things to deploy), but then add an argument like --wait that'll make them wait for everything to complete (like the default is now, maybe with no timeout?). It is definitely convenient sometimes to have the wait behaviour, so I wouldn't want to remove it entirely

[ca-scribner]

Regardless of how we do this, if the commands are waiting on completion we should be providing the user with good status details so they can see progress

[NohaIhab]

Synced with @misohu :
This also applies to dss remove, but in that case in PR #51 I have not implemented timeout, because the behavior of kubectl delete is that it waits on the resource to be deleted by default. However, without having timeout there is no visibility for the user as to what is going on while the command hangs.
We should decide whether to include timeout, and be consistent in all the commands, or otherwise have a strong argument of why some commands have timeout and others don't.

[misohu]

Synced with UX team.

Following are the blocking commands which should not have timeout

• initialize
• create
• start
• purge

Following are the non blocking commands which can happen behind the scenes

• remove
• stop

With these change we introduce states of the notebook lifecycle which will be tracked with dss list namely:

• Downloading (notebook is downloading image)
• Starting (notebook is starting so image is downloaded but the pod is not up)
• Removing (remove was executed but the notebook is still present)
• Stopping (notebook was scaled down but there is still replica)