[BUG] `npm install` creates unsync `package-lock.json` #6787

regseb · 2023-09-10T14:29:43Z

Is there an existing issue for this?

I have searched the existing issues

This issue exists in the latest npm version

I am using the latest npm

Current Behavior

When I run npm install, the generated package-lock.json file isn't synchronized with the package.json file. The npm ci command fails. If I run npm install a second time: the package-lock.json file is modified (and synchronized).

Expected Behavior

npm install creates a package-lock.json file synchronized.

Steps To Reproduce

Create package.json with this content:

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

npm install

added 261 packages, and audited 262 packages in 27s

62 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

npm ci

npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Invalid: lock file's [email protected] does not satisfy [email protected]
npm ERR! 
npm ERR! Clean install a project
npm ERR! 
npm ERR! Usage:
npm ERR! npm ci
npm ERR! 
npm ERR! Options:
npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm ERR! 
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR! 
npm ERR! Run "npm help ci" for more info

npm ERR! A complete log of this run can be found in: /home/regseb/.npm/_logs/2023-09-10T14_22_27_045Z-debug-0.log

cp package-lock.json package-save.json

npm install

changed 1 package, and audited 262 packages in 1s

62 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

diff package-lock.json package-save.json

1837,1839c1837,1839
<       "version": "2.6.11",
<       "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.11.tgz",
<       "integrity": "sha512-4I6pdBY1EthSqDmJkiNk3JIT8cswwR9nfeW/cPdUagJYEQG7R95WRH74wpz7ma8Gh/9dI9FP+OU+0E4FvtA55w==",
---
>       "version": "2.7.0",
>       "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
>       "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",

The directory at the end with the files package.json, package-lock.json, package-save.json, and the directory node_modules/: testcase.zip

Environment

npm: 10.0.0
Node.js: v20.5.1
OS Name: Ubuntu 22.04.3 LTS
System Model Name: Dell Inspiron
npm config:

; "user" config from /home/regseb/.npmrc

//registry.npmjs.org/:_authToken = (protected) 

; node bin location = /usr/bin/node
; node version = v20.5.1
; npm local prefix = /home/regseb/dev/testcase
; npm version = 10.0.0
; cwd = /home/regseb/dev/testcase
; HOME = /home/regseb
; Run `npm config ls -l` to show all defaults.

Related issues / pull request

The text was updated successfully, but these errors were encountered:

lukekarrys · 2023-09-11T16:20:21Z

Triaged and this is happening on both npm 9 and npm 10.

regseb · 2023-09-11T17:47:19Z

I reproduce the bug with this package too:

{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}

addons-scanner-utils has node-fetch 2.6.11 in peerDependencies (marked as optional);
htmlhint has node-fetch ^2.6.2 in dependencies.

siemhesda · 2023-09-28T18:14:07Z

There is a weird thing that happens within the package.json that you may have missed out @regseb

When you do the first npm install, the package.json changes a bit from

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

to

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  },
  "dependencies": {
    "npm": "file:.."
  }
}

I am not sure if the second part is expected and should be added by npm. Otherwise, it still isn't the culprit on why the package-lock.json is not synced with package.json on the first npm install. Does this only happen with node-fetch @regseb?

@lukekarrys can you confirm that this issue happens on your end too? (Modification of the package.json)

regseb · 2023-09-28T18:40:28Z

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

siemhesda · 2023-09-28T19:09:33Z

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

I noticed it is behaving so on my end since I was working on workspaces. In a normal set up it doesn't modify the package.json

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

regseb · 2023-09-28T19:34:44Z

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

I only have one case because addons-scanner-utils is a dependency of addons-linter:

testcase
+-- addons-linter 6.13.0
|   +-- addons-scanner-utils 9.3.0
|       +-- node-fetch 2.6.11 (peer)
+-- htmlhint
    +-- node-fetch ^2.6.2

My second example is just a version with one less intermediate step.

allanpaiste · 2023-10-06T16:06:00Z

TL;DR: this goes back to 7.0.9 which was the first release that caused the locked versions to be ignored on npm ci. The error Invalid: lock file's ... does not satisfy ... was introduced in 8.4.1 and is indeed correctly thrown as a guard against sneaky upgrades, but sneaky upgrades are actually caused by these two changes from the past (both still cause npm ci to misbehave):

7.0.9: 0e58e6f (npm ci installs newer version than what is in the lock).
8.6.0: bd96ae4 (npm ci thows the 8.4.1 assertion but would otherwise install newer version than what is in the lock).

Thus, the last glitch-free npm version is: 7.0.8

(As for the currently implemented installation error - it would serve much better as a test-case in integration test suite rather than user-facing functionality where it just causes confusion due to config-vs-lock being completely valid when it is thrown).

Our steps that get us to the error:

install package X with semver lock on a project Y: ^1.2.3 (new package-lock.json gets generated)
release new version of package X: 1.3.0
run npm ci for project Y.

EXPECTED: install still goes through and installs version 1.2.3 (as package-lock.json indicates)
ACTUAL: process errors out and informs that lock file's 1.2.3 does not satisfy 1.3.0

Some history:

This all goes way back to v7.X which got a bug introduced to it of which there have been attempts on fixing it ever since with varying level of success (the change in 8.4.1 being one of them).

8.4.0 - had an issue where when ^1.0.0 had produced locked version of 1.1.0 but latest available version in package index was 1.2.0 then 1.2.0 got installed (which violated the frozen version in lock).
8.4.1 - did not fix this but introduced an error that was thrown when this was attempted (in a sense it was not really a fix to the situation but rather a workaround that forced users to perform an upgrade to 1.2.0; problem: the same would repeat as soon as new version would be available).

The actual issue had been around longer and that specific release (and the error we have all been seeing) just represents an attempt to rein in the whole situation.

The hunt for the cause:

Though I initially focused on tracing down where the error originates from, it became since clear that instead of hunting for a release that does not fail the npm ci with the error - I should instead be searching for behaviour switch that the 8.4.1 code was catching. Little did I know that I'd be going back to the v7.X.

Below then are the findings by just going through the versions one-by-one to identify where the break happened (given in 3 steps where versions represent versions found in: package.json > package-lock.json > node_modules

7.0.8 - ^1.0.0 > 1.1.0 (lock) > npm ci > 1.1.0 (installed)
7.0.9 - ^1.0.0 > 1.1.0 (lock) > npm ci >1.2.0 (installed)

Thus the issue was somewhere in v7.0.8...v7.0.9.

From there, when doing another round of blind pin-pointing leads to this commit: 0e58e6f.

Starting from that specific commit, you'll start getting newer version installed than what is specified in the lock (which in turn in 8.4.1 was turned into a throw).

Most notably it's something about the change in build-ideal-tree.js

-      // didn't find a parent for it, but we're filling in external
-      // link targets, so go ahead and process it.
-      if (this[_follow] && !link.target.parent && !link.target.fsParent) {
+      // didn't find a parent for it or it has not been seen yet
+      // so go ahead and process it.
+      const unseenLink = (link.target.parent || link.target.fsParent)
+        && !this[_depsSeen].has(link.target)
+      if (this[_follow]
+        && !link.target.parent
+        && !link.target.fsParent
+        || unseenLink) {
         this.addTracker('idealTree', link.target.name, link.target.location)
         this[_depsQueue].push(link.target)
       }

Now when trying out newer version, I bumped into another "checkpoint" where same thing started to happen again (though this time around instead of getting a silent install, one would actually see the infamous error that is stated at the original report of the issue).

The second change was also done to build-ideal-tree.js, this time within the 8.6.0 release.

       .then(tree => {
+        // search the virtual tree for invalid edges, if any are found add their source to
+        // the depsQueue so that we'll fix it later
+        depth({
+          tree,
+          getChildren: (node) => [...node.edgesOut.values()].map(edge => edge.to),
+          filter: node => node,
+          visit: node => {
+            for (const edge of node.edgesOut.values()) {
+              if (!edge.valid) {
+                this[_depsQueue].push(node)
+                break // no need to continue the loop after the first hit
+              }
+            }
+          },
+        })
         // null the virtual tree, because we're about to hack away at it
         // if you want another one, load another copy.

Sadly enough, neither of the changes have proper test coverage which does not really inject confidence to the reader in terms of knowing that the base functionality of the installer remains the same (second change just modifies existing tests which just shadows potential new bug).

As such, would appreciate if someone else would take it from here.

Conclusion:

Reverting the change 0e58e6f will make npm ci behave expectedly up to version 8.5.5.
Reverting the change bd96ae4 (together with the one above) will make npm ci behave expectedly up to version 10.2.0 (latest).

Whatever these two changes claimed to fix - they introduced new issues. So those fixes should be re-introduced with proper test-coverage or re-reported as issues after reverting the broken fixes (I'd rather have those bugs back than the main intent of lock file being broken).

allanpaiste · 2023-10-17T12:48:16Z

Perhaps @isaacs @nlf @lukekarrys can comment on the changes mentioned above and what they're actually addressing. Could these pieces be re-imagined with provided unit/integration tests that would both guard the pre-7.0.9 functionality and also prove that the changes fix whatever they were supposed to fix.

allanpaiste · 2023-10-25T11:00:29Z

@siemhesda are you still involved in looking into this issue?

siemhesdagit · 2023-11-13T18:49:33Z

@siemhesda are you still involved in looking into this issue?

Hi @allanpaiste I have been away for a while but I'm back on this

…use install dependencies instead of clean build with ci

agdimech · 2024-05-24T01:53:21Z

I am also seeing this issue occuring when installing @aws/pdk.

Repro:

npm init && npm i && npm ci.

running npm i again resolves the issue and npm ci will work.

regseb added Bug thing that needs fixing Needs Triage needs review for next steps Release 10.x labels Sep 10, 2023

lukekarrys added Release 9.x work is associated with a specific npm 9 release and removed Needs Triage needs review for next steps labels Sep 11, 2023

lukekarrys added the Priority 1 high priority issue label Sep 11, 2023

siemhesda self-assigned this Sep 27, 2023

thecodeinfluencer mentioned this issue Nov 29, 2023

[BUG] npm install installs an invalid tree (as validated by npm ci) #5854

Open

2 tasks

nsbarsukov mentioned this issue Dec 26, 2023

ci: fix package-lock.json taiga-family/maskito#820

Merged

6 tasks

christiangalsterer added a commit to christiangalsterer/mongodb-driver-prometheus-exporter that referenced this issue Apr 21, 2024

ci: prevent error in npm ci caused by npm/cli#6787, workaround is to …

26f0642

…use install dependencies instead of clean build with ci

agdimech mentioned this issue May 24, 2024

[BUG] npm ci fails after installing @aws/pdk aws/aws-pdk#755

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] `npm install` creates unsync `package-lock.json` #6787

[BUG] `npm install` creates unsync `package-lock.json` #6787

regseb commented Sep 10, 2023 •

edited

lukekarrys commented Sep 11, 2023

regseb commented Sep 11, 2023

siemhesda commented Sep 28, 2023 •

edited

regseb commented Sep 28, 2023

siemhesda commented Sep 28, 2023

regseb commented Sep 28, 2023

allanpaiste commented Oct 6, 2023 •

edited

allanpaiste commented Oct 17, 2023

allanpaiste commented Oct 25, 2023

siemhesdagit commented Nov 13, 2023

agdimech commented May 24, 2024

[BUG] npm install creates unsync package-lock.json #6787

[BUG] npm install creates unsync package-lock.json #6787

Comments

regseb commented Sep 10, 2023 • edited

Is there an existing issue for this?

This issue exists in the latest npm version

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

Related issues / pull request

lukekarrys commented Sep 11, 2023

regseb commented Sep 11, 2023

siemhesda commented Sep 28, 2023 • edited

regseb commented Sep 28, 2023

siemhesda commented Sep 28, 2023

regseb commented Sep 28, 2023

allanpaiste commented Oct 6, 2023 • edited

allanpaiste commented Oct 17, 2023

allanpaiste commented Oct 25, 2023

siemhesdagit commented Nov 13, 2023

agdimech commented May 24, 2024

[BUG] `npm install` creates unsync `package-lock.json` #6787

[BUG] `npm install` creates unsync `package-lock.json` #6787

regseb commented Sep 10, 2023 •

edited

siemhesda commented Sep 28, 2023 •

edited

allanpaiste commented Oct 6, 2023 •

edited