Investigate buffering issues on GitHub runners

[0x2b3bfa0]

#1299 is a duct tape workaround to what seems to be a buffering issue on the GitHub self-hosted runner output, which prevents our logic from detecting when the runner transitions to a busy state in a timely manner; see #1255 (comment) for context.

[dacbd]

hmm I suspect this to be the source of my failed termination issues, on cml-playground, I say that I have observed the opposite though... the runner detects the first job and enters its busy state but fails to detect the job finishing and never enters an idle state.

[tasdomas]

Is there no better way of detecting the runner state, other than parsing logs?

…

On Fri, Dec 30, 2022, 18:40 Daniel Barnes ***@***.***> wrote: hmm I suspect this to be the source of my failed termination issues, on cml-playground, I say that I have observed the opposite though... the runner detects the first job and enters its busy state but fails to detect the job finishing and never enters an idle state. — Reply to this email directly, view it on GitHub <#1301 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5DNAE4YCE6KXLEVS752TWP4GAPANCNFSM6AAAAAATM776OA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dacbd]

We also call the GitHub API, but that also has its own issues, and it's only used as final validation.

[0x2b3bfa0]

the runner detects the first job and enters its busy state but fails to detect the job finishing and never enters an idle state

Sounds, indeed, like a buffering issue. 🤔 Node.js treatment of file descriptors is, indeed, cough very particular, and anything is possible.

[0x2b3bfa0]

Is there no better way of detecting the runner state, other than parsing logs?

At least, none known to me. 😰