Capture stdout and stderr

[rpkak]

Closes #886

Captures the stdout and the stderr of a job and makes it accessible using job.stdout ans job.stderr.

[codecov]

Codecov Report

Attention: Patch coverage is 87.35632% with 11 lines in your changes missing coverage. Please review.

Project coverage is 95.49%. Comparing base (76ba690) to head (ae3ed94).
Report is 312 commits behind head on master.

Files with missing lines	Patch %	Lines
rq/worker.py	57.14%	6 Missing ⚠️
rq/job.py	93.61%	3 Missing ⚠️
tests/fixtures.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1515      +/-   ##
==========================================
- Coverage   95.59%   95.49%   -0.11%     
==========================================
  Files          46       46              
  Lines        7058     7145      +87     
==========================================
+ Hits         6747     6823      +76     
- Misses        311      322      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[selwin]

I like the idea of workers writing to stdout and stderr, think should be achievable by tweaking the default logging configuration.

I'm not sure about saving stdout and stderr it to Redis though, it may significantly increase storage usage.

[rpkak]

I made this optional now.

You have to activate this using queue.enqueue(func, capture_stdout=True, capture_stderr=True).

[kuraga]

Any news?

[Hongyuy]

looking forward to using this feature.

[caiobsilva]

any reason as to why this PR hasn't been merged?

[dberardo-com]

i could confirm that this PR works as expected, but the streams are not closed when the function has run till the end. is this by design ?

i need to use this workaround:

job=q.enqueue(test_loop, capture_stdout=True, capture_stderr=True, retry=Retry(max=1, interval=[.2, 1, 2]), iterations=6)
try:
    for line in iter(job._stdout.strio.readline, b''):
        if line:
            print(">>> " + line.rstrip())
        if job.is_finished and not job._stdout.strio.closed: job._stdout.strio.close()
except ValueError: pass
assert job._stdout.strio.closed and job.is_finished

[melizeche]

@rpkak This feature would solve many issues for me, there's any way that I can help to get this approved and merged?

[selwin]

@melizeche you can use Python’s logging mechanism to log to stdout and stderr

[kuraga]

you can use Python’s logging mechanism to log to stdout and stderr

We would want to log per-job, I think?