Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vm: add experimental NodeRealm implementation #47855

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -2186,3 +2186,28 @@ The externally maintained libraries used by Node.js are:
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""

- synchronous-worker, located at lib/internal/vm/localworker.js, is licensed as follows:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be updated to point to the correct location via the license builder.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you point me at the docs for this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if there are docs for it, but I think you can change this line to lib/internal/vm/localworker.js where it currently says lib/worker_threads.js, rerun the license builder, and this should be updated.

"""
The MIT License (MIT)

Copyright (c) 2020 Anna Henningsen

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
17 changes: 17 additions & 0 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,22 @@ changes:
Specify the `module` of a custom experimental [ECMAScript module loader][].
`module` may be any string accepted as an [`import` specifier][].

### `--experimental-localworker`

<!-- YAML
added: REPLACEME
-->

Enable experimental support for `vm.LocalWorker`.

### `--no-experimental-localworker`

<!-- YAML
added: REPLACEME
-->

Disable experimental support for `vm.LocalWorker`.

### `--experimental-network-imports`

<!-- YAML
Expand Down Expand Up @@ -2113,6 +2129,7 @@ Node.js options that are allowed are:
* `--experimental-import-meta-resolve`
* `--experimental-json-modules`
* `--experimental-loader`
* `--experimental-localworker`
* `--experimental-modules`
* `--experimental-network-imports`
* `--experimental-permission`
Expand Down
99 changes: 99 additions & 0 deletions doc/api/vm.md
Original file line number Diff line number Diff line change
Expand Up @@ -1573,6 +1573,105 @@ inside a `vm.Context`, functions passed to them will be added to global queues,
which are shared by all contexts. Therefore, callbacks passed to those functions
are not controllable through the timeout either.

## Local Worker
mcollina marked this conversation as resolved.
Show resolved Hide resolved

> Stability: 1 - Experimental
mcollina marked this conversation as resolved.
Show resolved Hide resolved

### Class: `LocalWorker`

<!-- YAML
added: REPLACEME
-->

* Extends: {EventEmitter}

A `LocalWorker` is effectively a Node.js environment that runs within the
same thread.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need a bit more detail than this. A Node.js environment . . . with its own global scope? That can have separate NODE_OPTIONS? Is it CommonJS or ESM, or either?

To others’ points, how does this differ from Realm?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Realm (in specs terms) does not support ESM, require and all Node.js core modules. It's like a Realm, but with all the Node.js stuff.


```mjs
import { LocalWorker } from 'vm';
import { fileURLToPath } from 'url';
mcollina marked this conversation as resolved.
Show resolved Hide resolved
const w = new LocalWorker();
const myAsyncFunction = w.createRequire(fileURLToPath(import.meta.url))('my-module');
console.log(await myAsyncFunction());
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the docs should clarify the difference between this and a ShadowRealm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also like to understand the differences (and similarities) between this and a worker. Because they look very similar. For example, does a realm have an event loop? Does it share globals? (I'm assuming yes and no?)


#### `new LocalWorker()`

<!-- YAML
added: REPLACEME
-->

#### `localworker.runInWorkerScope(fn)`

<!-- YAML
added: REPLACEME
-->

* `fn` {Function}

Wrap `fn` and run it as if it were run on the event loop of the inner Node.js
instance. In particular, this ensures that Promises created by the function
itself are resolved correctly. You should generally use this to run any code
inside the inner Node.js instance that performs asynchronous activity and that
is not already running in an asynchronous context (you can compare this to
the code that runs synchronously from the main file of a Node.js application).
mcollina marked this conversation as resolved.
Show resolved Hide resolved

#### `localworker.stop()`

<!-- YAML
added: REPLACEME
-->

mcollina marked this conversation as resolved.
Show resolved Hide resolved
This will render the Node.js instance unusable
mcollina marked this conversation as resolved.
Show resolved Hide resolved
and is generally comparable to running `process.exit()`.

This method returns a `Promise` that will be resolved when all resources
associated with this Node.js instance are released. This `Promise` resolves on
the event loop of the _outer_ Node.js instance.
mcollina marked this conversation as resolved.
Show resolved Hide resolved

#### `localworker.createRequire(filename)`

<!-- YAML
added: REPLACEME
-->

* `filename` {string}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really any module specifier that can be passed to require, right? So not just filenames but also bare specifiers like lodash? Or node:fs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just require, yes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is the same as module.createRequire(), we should copy some of the docs from that API. For example, filename can also be a URL.


Create a `require()` function that can be used for loading CommonJS modules
inside the inner Node.js instance.

#### `localworker.createImport(filename)`

<!-- YAML
added: REPLACEME
-->

* `filename` {string}
mcollina marked this conversation as resolved.
Show resolved Hide resolved
mcollina marked this conversation as resolved.
Show resolved Hide resolved

Create a dynamic `import()` function that can be used for loading EcmaScript
mcollina marked this conversation as resolved.
Show resolved Hide resolved
modules inside the inner Node.js instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the signature of the returned function? Does it return a promise? Does it match the signature of import(), where the second argument is an options bag (like { assert: { type: 'json' } })?


#### `localworker.globalThis`

<!-- YAML
added: REPLACEME
-->

* Type: {Object}

Returns a reference to the global object of the inner Node.js instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be clarified whether this value is mutable. e.g. is it possible to localworker.globalThis.foo = 1 and have that value reflected within the local worker.


#### `localworker.process`

<!-- YAML
added: REPLACEME
-->

* Type: {Object}

Returns a reference to the `process` object of the inner Node.js instance.

[Cyclic Module Record]: https://tc39.es/ecma262/#sec-cyclic-module-records
[ECMAScript Module Loader]: esm.md#modules-ecmascript-modules
[Evaluate() concrete method]: https://tc39.es/ecma262/#sec-moduleevaluation
Expand Down
11 changes: 11 additions & 0 deletions lib/internal/process/pre_execution.js
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ function prepareExecution(options) {
setupInspectorHooks();
setupWarningHandler();
setupFetch();
setupLocalWorker();
setupWebCrypto();
setupCustomEvent();
setupCodeCoverage();
Expand Down Expand Up @@ -267,6 +268,16 @@ function setupFetch() {
});
}

function setupLocalWorker() {
// Patch the vm module when --experimental-localworker is on.
// Please update the comments in vm.js when this block changes.
if (getOptionValue('--experimental-localworker')) {
const LocalWorker = require('internal/vm/localworker');
const vm = require('vm');
vm.LocalWorker = LocalWorker;
}
}

// TODO(aduh95): move this to internal/bootstrap/web/* when the CLI flag is
// removed.
function setupWebCrypto() {
Expand Down
141 changes: 141 additions & 0 deletions lib/internal/vm/localworker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
'use strict';

// LocalWorker was originally a separate module developed by
// Anna Henningsen and published separately on npm as the
// synchronous-worker module under the MIT license. It has been
// incorporated into Node.js with Anna's permission.
// See the LICENSE file for LICENSE and copyright attribution.

const {
Promise,
} = primordials;

const {
LocalWorker: LocalWorkerImpl,
} = internalBinding('contextify');

const EventEmitter = require('events');
const { setTimeout } = require('timers');
const { dirname, join } = require('path');

let debug = require('internal/util/debuglog').debuglog('localworker', (fn) => {
debug = fn;
});

class LocalWorker extends EventEmitter {
#handle = undefined;
#process = undefined;
#global = undefined;
#module = undefined;
#stoppedPromise = undefined;
/**
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: empty comment?

constructor() {
super();
this.#handle = new LocalWorkerImpl();
this.#handle.onexit = (code) => {
this.stop();
this.emit('exit', code);
};
try {
this.#handle.start();
this.#handle.load((process, nativeRequire, globalThis) => {
this.#process = process;
this.#module = nativeRequire('module');
this.#global = globalThis;
process.on('uncaughtException', (err) => {
if (process.listenerCount('uncaughtException') === 1) {
this.emit('error', err);
process.exit(1);
}
});
});
} catch (err) {
this.#handle.stop();
throw err;
}
}

/**
* @returns {Promise<void>}
*/
async stop() {
// TODO(@mcollina): add support for AbortController, we want to abort this,
// or add a timeout.
return this.#stoppedPromise ??= new Promise((resolve) => {
const onExit = () => {
debug('stopping localworker');
this.#handle.stop();
resolve();
};

const tryClosing = () => {
const closed = this.#handle.tryCloseAllHandles();
debug('closed %d handles', closed);
if (closed > 0) {
// This is an active wait for the handles to close.
// We might want to change this in the future to use a callback,
// but at this point it seems like a premature optimization.
// TODO(@mcollina): refactor to use a close callback
setTimeout(tryClosing, 100);
} else {
this.#handle.signalStop();

setTimeout(onExit, 100);
}
};

// We use setTimeout instead of setImmediate because it runs in a different
// phase of the event loop. This is important because the immediate queue
// would crash if the environment it refers to has been already closed.
setTimeout(tryClosing, 100);
});
}

get process() {
return this.#process;
}

get globalThis() {
return this.#global;
}

createRequire(...args) {
return this.#module.createRequire(...args);
}

/**
* @param {() => any} method
*/
runInWorkerScope(method) {
return this.#handle.runInCallbackScope(method);
}

/**
* @param {string} filename
*/
async createImport(filename) {
mcollina marked this conversation as resolved.
Show resolved Hide resolved
// This is a hack to get around creating a dynamic import function
// from code. We create a temporary file that exports the import
// function, and then delete it.
// TODO(@mcollina): figure out how to do this using internal APIs.

const req = this.createRequire(filename);
const fs = req('fs/promises');

const sourceText = `
module.exports = (file) => import(file);
`;

const dest = join(dirname(filename), `_import_jump_${process.pid}.js`);
await fs.writeFile(dest, sourceText);

const ownImport = req(dest);

await fs.unlink(dest);

return ownImport;
}
}

module.exports = LocalWorker;
2 changes: 2 additions & 0 deletions lib/vm.js
Original file line number Diff line number Diff line change
Expand Up @@ -343,3 +343,5 @@ module.exports = {
// The vm module is patched to include vm.Module, vm.SourceTextModule
// and vm.SyntheticModule in the pre-execution phase when
// --experimental-vm-modules is on.
// The vm module is also patched to include vm.LocalWorker in the
// pre-execution phase when --experimental-localworker is on.
mcollina marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions src/env_properties.h
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,7 @@
V(socketaddress_constructor_template, v8::FunctionTemplate) \
V(streambaseentry_ctor_template, v8::FunctionTemplate) \
V(streambaseoutputstream_constructor_template, v8::ObjectTemplate) \
V(localworker_constructor_template, v8::FunctionTemplate) \
V(streamentry_ctor_template, v8::FunctionTemplate) \
V(streamentry_opaque_ctor_template, v8::FunctionTemplate) \
V(qlogoutputstream_constructor_template, v8::ObjectTemplate) \
Expand Down
Loading