feat(compartment-mapper)!: new hooks & type overhaul #2988

boneskull · 2025-10-09T23:48:39Z

This PR supersedes #2929 which should now be considered abandoned.

This PR targets #2915, if you didn't notice.

Description

This PR is "complete" and ready for review. It makes a significant overhaul to types and adds new options to several public APIs.

Description of changes to Types

The types around CompartmentMapDescriptor have been vastly expanded to reflect the different "flavors" of CompartmentMapDescriptor, CompartmentDescriptor, ModuleDescriptor (now ModuleDescriptorConfiguration to differentiate it from ses' ModuleDescriptor), and the formatting of the compartment names (keys).
CompartmentDescriptor.label is now a canonical name.
CompartmentDescriptor.path is removed
CompartmentDescriptor.compartments is removed
Added many type guards for validation of these differing types
- Validation of different types of compartment map
- Stricter validation
- Validation of all types of compartment descriptors, module descriptor configurations, etc.

New Default Behavior in `mapNodeModules()`

Currently, mapNodeModules() avoids adding a ModuleDescriptor (ModuleDescriptorConfiguration) to a nascent CompartmentDescriptor's modules prop if policy disallows it. This PR changes the behavior to consider policy & excise the dependency from the Node itself before the (dependency) Graph is translated into a CompartmentDescriptor. This has implications around what sort of errors are thrown when Compartment A attempts to load a Compartment B to which it has no access (see note about policy tests below).

Other Notes

Since CompartmentDescriptor.compartments has been removed, that means implicitly allowing dynamic requires of parent/ancestor Compartments from a given Compartment via absolute path is no longer supported. That's fine, since:
- This should be explicit in policy anyway
- @lavamoat/node was the only user of this feature and no longer needs it
The entry compartment now has a canonical name of $root$ . It is valid within PackagePolicy; i.e. some other package can be allowed to access the $root$ compartment.
Many tests in test/policy.test.ts needed to change because of how mapNodeModules' packageDependencies hook works. Instead of rejecting module descriptors (ModuleDescriptorConfigurations) based on policy, we reject entire dependencies before they can be "digested" into module descriptors. This results in ScopeDescriptors not being populated, in addition to ModuleDescriptorConfigurations, and causes downstream effects. Different exceptions are thrown at different times, but the intent of these tests doesn't deviate from "ensure this naughty behavior is not allowed."

Security Considerations

None that I'm aware of.

Scaling Considerations

Any consumer providing a resource-intensive blocking hook to @endo/compartment-mapper gets the performance they deserve.

Documentation Considerations

This is a breaking change to the contents of the archives as well as a breaking change to the types. It is not a breaking change to the archive format, but it will contain different values (instead of packageName-v<version> the compartment descriptors will be keyed on the canonical name).

Testing Considerations

There are some unit tests but we need more integration tests.

Compatibility Considerations

The breaking changes mentioned above, but it also solves like ten different problems @lavamoat/node was having trying to get things to work. It significantly improves ecosystem compatibility and will improve the performance of @lavamoat/node.

Upgrade Considerations

Include *BREAKING*: in the commit message with migration instructions for any breaking change.
Update NEWS.md for user-facing changes.

This commit introduces a comprehensive universal hook system for @endo/compartment-mapper,
enabling extensible customization of module mapping, bundling, and policy enforcement.

BREAKING CHANGES: CompartmentDescriptor objects no longer have a path property and the label property is now a canonical name. Types have changed dramatically. Enhanced validation of CompartmentMapDescriptor objects.

New Options

`mapNodeModules()`

packageDataHook: Called once before translateGraph with data about all packages found while crawling node_modules. Receives a read-only Map<CanonicalName, PackageData> where each entry contains:
- name: Package name
- packageDescriptor: The package.json contents
- location: File URL to the package
- canonicalName: The canonical name used in policy
packageDependenciesHook: Called for each package's dependencies during graph translation, allowing dynamic modification of the dependency graph. Can add or remove dependencies from packages based on policy or other criteria.
unknownCanonicalNameHook: Called when policy references canonical names that don't exist in the dependency graph, with optional suggestions for typos/similar names.

`makeImportHookMaker()`

moduleSourceHook(): Called when module source objects are created (either from local files containing bytes, exit modules, or "error-type" sources)

`captureFromMap()`

packageConnectionsHook: Called during digest; surfaces "connections" between compartment descriptors

Type System Overhaul (`src/types/`)

Major Breaking Type Changes in `compartment-map-schema.ts`:

CompartmentDescriptor Interface Restructure:

BREAKING: Removed path property - compartment descriptors no longer track dependency paths
BREAKING: label property type changed from string to CanonicalName<U> - labels are now type-safe canonical names
BREAKING: Made CompartmentDescriptor generic with <T extends ModuleDescriptorConfiguration, U extends string> for better type safety
Added location: string as a required property for all compartment descriptors
Added optional sourceDirname as found in the sources

CompartmentMapDescriptor Genericization (Is That A Word? No):

BREAKING: CompartmentMapDescriptor is now generic: CompartmentMapDescriptor<T, Name, EntryName>
BREAKING: compartments property changed from Record<string, CompartmentDescriptor> to CompartmentDescriptors<T, Name>
BREAKING: entry property type changed from EntryDescriptor to EntryDescriptor<EntryName>
New specialized types: PackageCompartmentMapDescriptor, FileCompartmentMapDescriptor, and DigestedCompartmentMapDescriptor

Enhanced Module Descriptor System:

ModuleDescriptor is now ModuleDescriptorConfiguration to differentiate between it & the ModuleDescriptor from ses
Added ModuleDescriptorConfigurationCreator enum tracking module creation source ('link' | 'transform' | 'import-hook' | 'digest' | 'node-modules')
Enhanced BaseModuleDescriptorConfiguration with __createdBy property for debugging
Added ErrorModuleDescriptorConfiguration for deferred error handling
Improved type discrimination with ModuleDescriptorConfigurationKindToType utility type

New Type Infrastructure:

`canonical-name.ts` - Canonical Name Type System:

NEW: Type-level canonical name validation using template literal types
CanonicalName<S> - validates npm package name chains separated by '>' (e.g., "foo>bar", "@scope/pkg>dep")
ScopedPackageName and UnscopedPackageName for npm package name validation
SplitOnGt<S> and AllValidPackageNames<Parts> for compile-time canonical name parsing
IsCanonicalName<S> predicate type for conditional type logic

Enhanced Type Safety Features:

Compartment Descriptor Validation:

DigestedCompartmentDescriptor - restricted descriptor for archived compartment maps
Properties marked as never for digested descriptors: path, retained, scopes, parsers, types, __createdBy, sourceDirname
CompartmentDescriptorWithPolicy<T> - enforces policy presence where required

Module Configuration Type Safety:

ModuleDescriptorConfigurationKind union for module type discrimination
Type-safe module configuration creators with __createdBy tracking

Policy Integration:

Enhanced policy types in PackageCompartmentDescriptor with canonical name constraints
LiteralUnion usage for special canonical names (ATTENUATORS_COMPARTMENT, ENTRY_COMPARTMENT)
Policy-aware compartment descriptor types with enhanced validation

Type System Utilities:

`typescript.ts` Enhancements:

Enhanced LiteralUnion<L, B> for better literal type preservation
Type utilities supporting the new generic compartment map architecture
Moved Simplify from tests here (because it's useful dammit)

Enhanced Policy Validation

Policy Validation:

Unknown canonical name detection with suggestion system
Cross-reference policy resources against actual compartment contents
Detailed error reporting with path information for policy issues
Hook integration for custom policy validation logic

boneskull · 2025-10-10T03:10:10Z

Fix:

Error: ../compartment-mapper/src/policy-format.d.ts(9,76): error TS2304: Cannot find name 'WildcardPolicy'.
Error: ../compartment-mapper/src/policy-format.d.ts(16,81): error TS2304: Cannot find name 'SomePolicy'.

boneskull · 2025-10-16T22:21:23Z

policy.test.ts needs further fixes since it breaks prior behaviors.

packages/compartment-mapper/src/policy.js

naugtur · 2025-10-29T17:30:38Z

packages/compartment-mapper/src/node-modules.js

+  message,
+  log,
+}) => {
+  log(`WARN: Invalid resource ${q(canonicalName)} in policy: ${message}`);


This kind of things, at least in my experience with webpack, worked better if aggregated and displayed as a single warning. In a debug mode of sorts they could be used to generate proposed policy fixes.

This sounds like more work 😄

I could do this, but I think even reporting this at all is going out on a limb (in terms of the overall user-friendliness of @endo/compartment-mapper). I would not be necessarily opposed to expanding this and implementing other friendly improvements, but I don't think they need to be in this PR.

packages/compartment-mapper/src/link.js

naugtur · 2025-10-31T12:28:49Z

packages/compartment-mapper/src/node-modules.js

  }
+
+  // #region packageDescriptor hook
+  if (packageDescriptorHook) {


I'm not sure this is useful, but it could be useful if it allowed altering the descriptor (eg. in case it contain something that was not supported or needed a fix.

naugtur · 2025-10-31T12:31:46Z

packages/compartment-mapper/src/capture-lite.js

    Compartment: CompartmentOption = DefaultCompartment,
    log = noop,
    preload = [],
+    packageConnectionsHook,


packageDependenciesHook and packageConnectionsHook are similar

I looked at both of these hooks and I think packageConnectionsHook could be replaced with packageDependenciesHook and the main difference is that one of them allows returning a mutated set.
If we could offer the same in digest, we'd have a single hook for both.

The data from packageDependenciesHook and packageConnectionsHook come from two different types of compartment map, and the latter happens post-digest. I would not want to do this.

I'm not enjoying there being two flows that are somewhat similar but ultimately distinct. But the solution to that is ultimately not forcing the unification fo the hook, but finding way to split regular flow into steps so that capture was no longer necessary a all.
Can leave this be if you're certain unifying the 2 hooks wouldn't be feasible.
Note that I'm not fond of explaining the details and differences between the two different types of compartment maps to the end user 🙈

Yeah, I mean, ideally the end user wouldn't need to touch a compartment map.

packages/compartment-mapper/src/policy.js

packages/compartment-mapper/src/import-hook.js

…duleConfiguration This narrows `CompartmentDescriptor` into three discrete flavors; `PackageCompartmentDescriptor` (as returned by `mapNodeModules()`, `DigestedCompartmentDescriptor` (as returned by `digestCompartmentMap()`, and `FileCompartmentDescriptor` (as used in `parseArchive()`. `ModuleDescriptor` has been renamed to `ModuleConfiguration` to avoid conflicts with `ses`' `ModuleDescriptor`. `ModuleConfiguration` has been narrowed into several types: - `CompartmentModuleConfiguration` - references another `ModuleConfiguration` - `FileModuleConfiguration` - refers to a module on disk - `ErrorModuleConfiguration` - a module configuration having a `deferredError` prop - `ExitModuleConfiguration` - refers to an exit module

…to avoid cycle

…a.ts

exclude .d.ts files (which may exist after running `prepack`)

…jects This adds some helpers to snapshot `CompartmentMapDescriptor`s, and `CaptureResult` objects

…ties `ProjectFixture` objects can now have `entrypoint` properties. Extend project fixture read powers to actually read phony files, not just package descriptors. Abuses node's custom `inspect()` facilities to make a real sweet dump of a `CompartmentMapDescriptor`.

…env var This changes `test/scaffold.js` to recognize a `SCAFFOLD_LOGGING` env var. If it is set, then calls to `log()` throughout various exported functions in the package will actually log (via AVA) instead of doing nothing.

BREAKING CHANGES: `CompartmentMapDescriptor` no longer has a `path` property. `CompartmentMapDescriptor`'s `label` property is now a _canonical name_ (a string of one or more npm package names separated by `>`). The `CompartmentMapDescriptor` returned by `captureFromMap()` now uses canonical names as the keys in its `compartments` property. - Breaking types: `CompartmentMapDescriptor`, `CompartmentDescriptor`, `ModuleConfiguration` (renamed from `ModuleDescriptor`) and `ModuleSource` have all been narrowed into discrete subtypes. - `captureFromMap()`, `loadLocation()` and `importLocation()` now accept a `moduleSourceHook` option. This hook is called when processing each module source, receiving the module source data (location, language, bytes, or error information) and the canonical name of the containing package. - `mapNodeModules()`, `loadLocation()`, `importLocation()`, `makeScript()`, `makeFunctor()`, and `writeScript()` now accept the following hook options: - `unknownCanonicalNameHook`: Called for each canonical name mentioned in policy but not found in the compartment map. Useful for detecting policy misconfigurations. - `packageDependenciesHook`: Called for each package with its set of dependencies. Can return partial updates to modify the dependencies, enabling dependency filtering or injection based on policy. - `packageDataHook`: Called once with data about all packages found while crawling `node_modules`, just prior to creation of a compartment map. * * * - When packages are disallowed by policy, the disallowing bit happens earlier than it used to. This has downstream effects on which errors are thrown when access is requested to a disallowed compartment. Regardless, the ultimate behavior doesn't change—disallowed compartments are still disallowed. The new behavior is reflected in the changes to `src/policy.test.js`. - Improved compartment map validation to work with discrete `CompartmentMapDescriptor` subtypes - Type guards used - `mapNodeModules()` now has a concept of a `FinalNode` having a `label` prop - `makePackagePolicy()` replaces `getPolicyForPackage()` in `src/policy.js` - Many tests added for new and old features and fixed (such as nonsensical policies in `test/dynamic-require.test.js`)

naugtur · 2025-12-02T10:35:53Z

@boneskull Why remove the hooks.md with a graph documenting how hooks are reachable from public API? I found it very helpful in understanding how it all goes together.

naugtur

LGTM.
Would prefer to keep hooks.md

boneskull · 2025-12-02T22:02:31Z

@naugtur For some reason I thought that doc was outdated and reflected the old system. If it's not, I'll restore it

UPDATE: I restored it.

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 6c51178 to d2e827c Compare October 10, 2025 00:21

boneskull mentioned this pull request Oct 10, 2025

feat(compartment-mapper)!: create Compartment Map Transforms #2929

Closed

boneskull self-assigned this Oct 10, 2025

boneskull added enhancement New feature or request ecosystem-compatibility Tracks a compatibility issue in a third-party package or packages. labels Oct 10, 2025

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from d2e827c to ee56c40 Compare October 10, 2025 00:35

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from ee56c40 to 5e78f40 Compare October 16, 2025 22:18

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch 4 times, most recently from c37edbe to 1d864b5 Compare October 17, 2025 21:39

boneskull force-pushed the boneskull/force-load branch 2 times, most recently from 8b5ac3e to 38e98bf Compare October 21, 2025 21:05

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from b21326c to 983ad2e Compare October 21, 2025 21:40

boneskull marked this pull request as ready for review October 22, 2025 18:11

naugtur added the lavamoat label Oct 23, 2025

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch 2 times, most recently from 7916a39 to d73f3d9 Compare October 24, 2025 20:41

boneskull added a commit to LavaMoat/LavaMoat that referenced this pull request Oct 27, 2025

fix(node): update code to use latest version of endojs/endo#2988

66ccc47

naugtur reviewed Oct 29, 2025

View reviewed changes

packages/compartment-mapper/src/policy.js Outdated Show resolved Hide resolved

naugtur reviewed Oct 29, 2025

View reviewed changes

packages/compartment-mapper/src/policy.js Outdated Show resolved Hide resolved

naugtur reviewed Oct 29, 2025

View reviewed changes

naugtur reviewed Oct 30, 2025

View reviewed changes

packages/compartment-mapper/src/link.js Show resolved Hide resolved

naugtur reviewed Oct 31, 2025

View reviewed changes

boneskull changed the title ~~feat(compartment-mapper)!: universal hooks implementation~~ feat(compartment-mapper)!: new hooks Oct 31, 2025

boneskull changed the title ~~feat(compartment-mapper)!: new hooks~~ feat(compartment-mapper)!: new hooks & type overhaul Oct 31, 2025

boneskull mentioned this pull request Oct 31, 2025

chore(node): add support for webpack LavaMoat/LavaMoat#1593

Open

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 284b755 to d764dcc Compare November 24, 2025 23:22

naugtur reviewed Nov 25, 2025

View reviewed changes

packages/compartment-mapper/src/policy.js Show resolved Hide resolved

naugtur requested changes Nov 25, 2025

View reviewed changes

packages/compartment-mapper/src/import-hook.js Outdated Show resolved Hide resolved

naugtur reviewed Nov 26, 2025

View reviewed changes

packages/compartment-mapper/src/import-hook.js Outdated Show resolved Hide resolved

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 2255797 to 4afedea Compare December 1, 2025 20:37

boneskull requested a review from naugtur December 1, 2025 20:38

boneskull added 11 commits December 1, 2025 13:00

chore(compartment-mapper): create guards module

9a1aff5

chore(compartment-mapper): add Simplify to util types

01f4f3b

chore(compartment-mapper): create CanonicalName type and module

7f092ac

chore(compartment-mapper): move PackageDescriptor to node-modules.ts …

db6fec4

…to avoid cycle

chore(compartment-mapper): add comments & nice things to policy-schem…

025454c

…a.ts

chore(compartment-mapper): update tsconfig.json

f29da29

exclude .d.ts files (which may exist after running `prepack`)

fix(env-options): actually expose the generated types

4015a6b

chore(compartment-mapper): add test harness for snapshotting large ob…

0326b8a

…jects This adds some helpers to snapshot `CompartmentMapDescriptor`s, and `CaptureResult` objects

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 868a287 to 70b3440 Compare December 1, 2025 22:35

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 70b3440 to e3a1f27 Compare December 1, 2025 22:40

naugtur approved these changes Dec 2, 2025

View reviewed changes

boneskull and others added 2 commits December 2, 2025 14:10

docs(compartment-mapper): update NEWS.md

974176a

docs(compartment-mapper): add internal documentation of hooks wiring

ccf0ae1

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from e3a1f27 to ccf0ae1 Compare December 2, 2025 22:10

boneskull enabled auto-merge December 2, 2025 22:13

boneskull merged commit 3319469 into master Dec 2, 2025
19 checks passed

boneskull deleted the boneskull/compartment-mapper-hook-system branch December 2, 2025 22:16

boneskull added a commit to LavaMoat/LavaMoat that referenced this pull request Dec 2, 2025

fix(node): updates for endojs/endo#2988 and endojs/endo#2915

0436afc

feat(compartment-mapper)!: new hooks & type overhaul #2988

feat(compartment-mapper)!: new hooks & type overhaul #2988

Conversation

boneskull commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Description of changes to Types

New Default Behavior in mapNodeModules()

Other Notes

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

New Options

mapNodeModules()

makeImportHookMaker()

captureFromMap()

Type System Overhaul (src/types/)

Major Breaking Type Changes in compartment-map-schema.ts:

CompartmentDescriptor Interface Restructure:

CompartmentMapDescriptor Genericization (Is That A Word? No):

Enhanced Module Descriptor System:

New Type Infrastructure:

canonical-name.ts - Canonical Name Type System:

Enhanced Type Safety Features:

Compartment Descriptor Validation:

Module Configuration Type Safety:

Policy Integration:

Type System Utilities:

typescript.ts Enhancements:

Enhanced Policy Validation

Policy Validation:

Uh oh!

boneskull commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boneskull commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

naugtur Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

boneskull Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

naugtur Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boneskull Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

boneskull Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

naugtur commented Dec 2, 2025

Uh oh!

naugtur left a comment

Choose a reason for hiding this comment

Uh oh!

boneskull commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

boneskull commented Oct 9, 2025 •

edited

Loading

New Default Behavior in `mapNodeModules()`

`mapNodeModules()`

`makeImportHookMaker()`

`captureFromMap()`

Type System Overhaul (`src/types/`)

Major Breaking Type Changes in `compartment-map-schema.ts`:

`canonical-name.ts` - Canonical Name Type System:

`typescript.ts` Enhancements:

boneskull commented Oct 10, 2025 •

edited

Loading

naugtur Oct 31, 2025 •

edited

Loading

boneskull commented Dec 2, 2025 •

edited

Loading