[BUG] running npm install has nondeterministic behaviour and race conditions for packages with link references

[domdomegg]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

This happens when a package ('packageA') has a link to a dependency bundled within itself ('packageB'). This results in a condition when reifying the tree where:

• if 'packageA' is installed before packageB, the files for packageB exist and the link is created successfully
• otherwise, the link silently fails to create

Because both packageA and packageB are installed in a Promise.all(), there is a race where either one of these can happen first, resulting in nondeterministic behaviour.

I discovered this as I was trying to fix #8199. This blocks us from erroring when a link target does not exist, because at the point of reifying we can't easily tell if the link target doesn't exist because it depends on another package, or because it just doesn't exist.

Expected Behavior

The packages should be installed probably in dependency order, e.g. if packageB is a local file and part of packageA, it should wait for packageA to complete installing.

OR

External packages relying on local file links should just not be supported as a feature, if it is too unstable/causes performance issues.

Steps To Reproduce

npm init
npm install @isaacs/testing-link-dep

(notice that probably your node_modules will now not have a linked-dep link, and this will have silently failed to create)

npm install

(notice that probably your node_modules will now have a linked-dep link)

You can also then remove node_modules and run npm install multiple times to see the race condition.

Environment

• npm: 11.2.0 (present on latest branch)
• Node.js: v23.10.0
• OS Name: macOS
• System Model Name: macOS
• npm config:

; "builtin" config from /opt/homebrew/lib/node_modules/npm/npmrc

prefix = "/opt/homebrew"

; "user" config from /Users/adam/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /opt/homebrew/Cellar/node/23.10.0_1/bin/node
; node version = v23.10.0
; npm local prefix = /Users/adam
; npm version = 11.2.0
; cwd = /Users/adam
; HOME = /Users/adam
; Run `npm config ls -l` to show all defaults.

[milaninfy]

This does appear to work

~/workarea/rep/test $ npm init -y
Wrote to /Users/milaninfy/workarea/rep/test/package.json:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "commonjs"
}



~/workarea/rep/test $ npm install @isaacs/testing-link-dep


added 2 packages, and audited 4 packages in 635ms

found 0 vulnerabilities
~/workarea/rep/test $ tree node_modules 
node_modules
├── @isaacs
│   └── testing-link-dep
│       ├── package.json
│       └── target
│           └── package.json
└── linked-dep -> @isaacs/testing-link-dep/target

5 directories, 2 files


~/workarea/rep/test $ npm ls --all
test@1.0.0 /Users/milaninfy/workarea/rep/test
└─┬ @isaacs/testing-link-dep@2.0.0
  └── linked-dep@npm:target@ -> ./node_modules/@isaacs/testing-link-dep/target

[domdomegg]

Apologies - please ignore the below. I need to do more investigating...

This is what I get - the first time round, linked-dep is not set up correctly:

Console logs (incorrect)

Last login: Wed Apr  2 15:40:54 on ttys001
adam@Adams-MacBook-Pro ~ % cd Documents 
adam@Adams-MacBook-Pro Documents % mkdir npm-8201-repro
adam@Adams-MacBook-Pro Documents % cd npm-8201-repro 
adam@Adams-MacBook-Pro npm-8201-repro % npm init -y
Wrote to /Users/adam/Documents/npm-8201-repro/package.json:

{
  "name": "npm-8201-repro",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "commonjs"
}



adam@Adams-MacBook-Pro npm-8201-repro % npm install @isaacs/testing-link-dep

added 2 packages, and audited 4 packages in 380ms

found 0 vulnerabilities
adam@Adams-MacBook-Pro npm-8201-repro % find node_modules
node_modules
node_modules/@isaacs
node_modules/@isaacs/testing-link-dep
node_modules/@isaacs/testing-link-dep/target
node_modules/@isaacs/testing-link-dep/target/package.json
node_modules/@isaacs/testing-link-dep/package.json
node_modules/.package-lock.json
adam@Adams-MacBook-Pro npm-8201-repro % npm install

added 1 package, and audited 4 packages in 681ms

found 0 vulnerabilities
adam@Adams-MacBook-Pro npm-8201-repro % find node_modules
node_modules
node_modules/@isaacs
node_modules/@isaacs/testing-link-dep
node_modules/@isaacs/testing-link-dep/target
node_modules/@isaacs/testing-link-dep/target/package.json
node_modules/@isaacs/testing-link-dep/package.json
node_modules/linked-dep                                         # <-- this line is new!
node_modules/.package-lock.json
adam@Adams-MacBook-Pro npm-8201-repro % 

[k-walsh-gmg]

Seeing the same behavior as previously with the other days outtage