Support stdout and stderr on the worker

[logikaljay]

I want to stream the stdout and stderr of the tasks to a log file/database/websocket etc.

I think it is achievable, providing a change is made to the options provided to the Worker constructor here:

piscina/src/index.ts

Lines 570 to 577 in 2f95678

	const worker = new Worker(resolve(__dirname, 'worker.js'), {
	env: this.options.env,
	argv: this.options.argv,
	execArgv: this.options.execArgv,
	resourceLimits: this.options.resourceLimits,
	workerData: this.options.workerData,
	trackUnmanagedFds: this.options.trackUnmanagedFds
	});

I implemented this locally and it seemed to work.. mostly.. (happy to open a PR).
The issue I found was that the task never ended.. I couldn't figure this out, it may have been my stream.. I used fs.createWriteStream to see if it would create a log file of the tasks stdout, which it did.
However as I described, when the task finished, the application did not exit as it does when stdout is set to false on the Worker.

There is probably something I am missing, especially when it comes to the stdout and stderr checks being made here:

piscina/src/worker.ts

Lines 158 to 167 in 2f95678

	// If the task used e.g. console.log(), wait for the stream to drain
	// before potentially entering the `Atomics.wait()` loop, and before
	// returning the result so that messages will always be printed even
	// if the process would otherwise be ready to exit.
	if (process.stdout.writableLength > 0) {
	await new Promise((resolve) => process.stdout.write('', resolve));
	}
	if (process.stderr.writableLength > 0) {
	await new Promise((resolve) => process.stderr.write('', resolve));
	}

There were also some warnings in the console about a "Possible EventEmitter memory leak" being detected:

(node:3721611) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 17 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit

I believe this is because I am adding the same listener to each worker. setting setMaxListeners higher than the number of CPU cores resolved this warning - this is probably not the correct way to go about this though.

Any thoughts/comments would be appreciated. Hopefully I am not wasting anyones time.

[jasnell]

@addaleax would likely be able to explain what's happening here a lot better than I can since she wrote that part of the workers code... generally tho... process.stdout and process.stderr in workers essentially forward the data on to main thread for processing. What would appear to be happening here is that the write callbacks might not be getting called.

[taxilian]

my guess would be that if it's not closing the thread it has something to do with needing to close at least stdin, maybe both, when it's time to shut down the thread.

[metcoder95]

Have you tried setting useAtomics: false?

[github-actions]

This issue has been marked as stale because it has been opened 30 days without activity. Remove stale label or comment or this will be closed in 5 days.

[metcoder95]

@logikaljay is this still valid?

[logikaljay]

@metcoder95 no its all good, closing.