The browser tab stops responding if a particular stage has lot of logs

[itsluminous]

Issue Type

• Bug Report

Summary

When we open logs for any stage which has a lot of logs, then the tab just hangs.
It takes more than a couple mins in my case (on chrome with M1 Max chip & 64 GB ram)
It would be great if we can enable Raw output button always (while rendering continues). This way I can check logs without waiting or crashing my tabs.

Basic environment details

• Go Version: 22.3.0 (15301-9d23ed19a9ea46eaf7f18bd16671ae0569871f53)
• JAVA Version: 17.0.5
• OS: Linux 3.10.0-1160.88.1.el7.x86_64
• Browser vendor and version (if relevant): Chrome 113.0.5672.126 (arm64)

Steps to Reproduce

1. Write a lot of logs in some stage
2. Open that stage from gocd in browser

Expected Results

The tab should not hang

Actual Results

The tab hangs and no click works

Possible Fix

Enable Raw output button always (while rendering continues).

[chadlwilson]

Can you define what 'a lot of logs' is? Size of file, # of lines, length of lines? GoCD is not a logging / log aggregation system - if the logs are crazy big, I would question the value of those build logs, but depends how big we are talking about.

Also, a simular size/shape of log have you seen any regression over versions?

[itsluminous]

I have logs over 500k lines. Sometimes reaches 900k.
The amount of logging here may look huge but it is needed for the pipeline we are running. We want to be able to trace out issues when they appear.

I think the issue is not version dependent. I have seen this problem ever since I started using GoCD (2 years)

[chadlwilson]

I don't think that is reasonable to expect a tool such as GoCD to handle such an amount of build output gracefully within a live logging web UI. Certainly not with the current design.

Rather than using console logging directly from your tasks, you might consider redirecting their output from tools to files, and then publish those files as GoCD artifacts from your job. That way only "raw" output is available. Internally the console log is treated as a "special" artifact anyway, and uses the same API to return the raw output.

Nevertheless, I'd need more data (e.g total log file raw size) and some metrics on how big the log lines are since the length of lines also matters to how slow it is to process. The easier you can make to replicate the issue, the more likely it is to be understood and addressed.

You may also want to experiment with enabling console log streaming over websockets via feature toggle to see if the behaviour is different from a UI perspective. if you are interested in trying that, let me know and I'll dig up how to configure it (sadly does not seem to be documented anywhere I can find)

[itsluminous]

The average count of characters in one line can be said to be about 100.
One of the log with 875036 lines is of size 67.9 MB
In our case, it is a nightly job, so most of the time we don't monitor it live, hence I was saying that it would be great if I have option to see raw output directly instead of goCD trying to parse or render the logs.

[chadlwilson]

Yeah, as I suggested earlier, as a workaround I would redirect the bulk of the output to file, and artifact it separately rather than using the console like this.

There is an option to see the console output directly as you mentioned above, as well as via the "artifacts" tab, however if the whole UI hangs that is more challenging since the Console tab is the default when navigating to view job results and there is no direct navigation to the artifacts tab.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.