All Agents offline after Upgrade to 0.17.5 #1751

NetBLOKS · 2024-02-06T14:26:03Z

NetBLOKS
Feb 6, 2024

Server Info (please complete the following information):

OS: [Debian 12]
Browser: [e.g. chrome, safari]
RMM Version (0.17.5):

Installation Method:

[X ] Standard
Docker

Describe the bug
All Agents are marked as "Offline"
Remotecontrol still works
Remotecommands seem not to be working.

To Reproduce
Steps to reproduce the behavior:
Upgrade to 0.17.5
Clear all Caches etc.
Login to TRMM
All Agents offline

Expected behavior
All Agents online and doing update to newest version

mflagler · 2024-02-06T15:11:38Z

mflagler
Feb 6, 2024

Server Info (please complete the following information):

OS: [Ubuntu 20]
Browser: [e.g. chrome, safari]
RMM Version (0.17.5):

Installation Method:

[X ] Standard

I had the same issue, and I suspect this is the root cause possibly:

npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm: [email protected]
npm ERR! notsup Not compatible with your version of node/npm: [email protected]
npm ERR! notsup Required: {"node":"^18.17.0 || >=20.5.0"}
npm ERR! notsup Actual:   {"npm":"9.8.1","node":"v16.20.2"}

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-02-06T15_04_56_595Z-debug-0.log

Then at the end of the upgrade I had this notice:

npm notice
npm notice New major version of npm available! 9.8.1 -> 10.4.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.4.0
npm notice Run npm install -g [email protected] to update!
npm notice
Update finished!

Running the npm update as suggested produces the same error as above.

Here's the output of the log file:

0 verbose cli /usr/bin/node /usr/bin/npm
1 info using [email protected]
2 info using [email protected]
3 timing npm:load:whichnode Completed in 18ms
4 timing config:load:defaults Completed in 4ms
5 timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 6ms
6 timing config:load:builtin Completed in 7ms
7 timing config:load:cli Completed in 3ms
8 timing config:load:env Completed in 1ms
9 timing config:load:project Completed in 1ms
10 timing config:load:file:/home/tactical/.npmrc Completed in 1ms
11 timing config:load:user Completed in 1ms
12 timing config:load:file:/usr/etc/npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:setEnvs Completed in 2ms
15 timing config:load Completed in 19ms
16 timing npm:load:configload Completed in 19ms
17 timing config:load:flatten Completed in 5ms
18 timing npm:load:mkdirpcache Completed in 0ms
19 timing npm:load:mkdirplogs Completed in 0ms
20 verbose title npm install [email protected]
21 verbose argv "install" "--global" "[email protected]"
22 timing npm:load:setTitle Completed in 1ms
23 timing npm:load:display Completed in 1ms
24 verbose logfile logs-max:10 dir:/home/tactical/.npm/_logs/2024-02-06T15_09_49_840Z-
25 verbose logfile /home/tactical/.npm/_logs/2024-02-06T15_09_49_840Z-debug-0.log
26 timing npm:load:logFile Completed in 13ms
27 timing npm:load:timers Completed in 0ms
28 timing npm:load:configScope Completed in 1ms
29 timing npm:load Completed in 84ms
30 timing config:load:flatten Completed in 7ms
31 silly logfile done cleaning log files
32 http fetch GET 200 https://registry.npmjs.org/npm 93ms (cache hit)
33 timing command:install Completed in 175ms
34 verbose stack Error: Unsupported engine
34 verbose stack     at Object.checkEngine (/usr/lib/node_modules/npm/node_modules/npm-install-checks/lib/index.js:14:25)
34 verbose stack     at Install.exec (/usr/lib/node_modules/npm/lib/commands/install.js:112:16)
34 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli-entry.js:61:5)
35 verbose pkgid [email protected]
36 verbose cwd /home/tactical
37 verbose Linux 5.4.0-170-generic
38 verbose node v16.20.2
39 verbose npm  v9.8.1
40 error code EBADENGINE
41 error engine Unsupported engine
42 error engine Not compatible with your version of node/npm: [email protected]
43 error notsup Not compatible with your version of node/npm: [email protected]
43 error notsup Required: {"node":"^18.17.0 || >=20.5.0"}
43 error notsup Actual:   {"npm":"9.8.1","node":"v16.20.2"}
44 verbose exit 1
45 timing npm Completed in 676ms
46 verbose code 1
47 error A complete log of this run can be found in: /home/tactical/.npm/_logs/2024-02-06T15_09_49_840Z-debug-0.log

1 reply

dinger1986 Feb 6, 2024
Collaborator

its not NPM, do you have anything custom?

NetBLOKS · 2024-02-06T15:24:43Z

NetBLOKS
Feb 6, 2024
Author

Hmmm,

I´ve got no Errors on update.sh --force:

0 verbose cli /usr/bin/node /usr/bin/npm
1 info using [email protected]
2 info using [email protected]
3 timing npm:load:whichnode Completed in 0ms
4 timing config:load:defaults Completed in 1ms
5 timing config:load:file:/usr/share/nodejs/npm/npmrc Completed in 2ms
6 timing config:load:builtin Completed in 2ms
7 warn config optional Use --omit=optional to exclude optional dependencies, or
7 warn config --include=optional to include them.
7 warn config
7 warn config Default value does install optional deps unless otherwise omitted.
8 timing config:load:cli Completed in 3ms
9 timing config:load:env Completed in 0ms
10 timing config:load:file:/meshcentral/.npmrc Completed in 0ms
11 timing config:load:project Completed in 1ms
12 timing config:load:file:/home/tactical/.npmrc Completed in 0ms
13 timing config:load:user Completed in 1ms
14 timing config:load:file:/etc/npmrc Completed in 0ms
15 timing config:load:global Completed in 0ms
16 timing config:load:setEnvs Completed in 1ms
17 timing config:load Completed in 9ms
18 timing npm:load:configload Completed in 10ms
19 timing npm:load:mkdirpcache Completed in 0ms
20 timing npm:load:mkdirplogs Completed in 0ms
21 verbose title npm install [email protected]
22 verbose argv "install" "--save-exact" "--no-audit" "--no-optional" "--omit" "optional" "[email protected]"
23 timing npm:load:setTitle Completed in 1ms
24 timing config:load:flatten Completed in 3ms
25 timing npm:load:display Completed in 5ms
26 verbose logfile logs-max:10 dir:/home/tactical/.npm/_logs/2024-02-06T15_21_51_234Z-
27 verbose logfile /home/tactical/.npm/_logs/2024-02-06T15_21_51_234Z-debug-0.log
28 timing npm:load:logFile Completed in 5ms
29 timing npm:load:timers Completed in 0ms
30 timing npm:load:configScope Completed in 0ms
31 timing npm:load Completed in 22ms
32 silly logfile done cleaning log files
33 timing config:load:flatten Completed in 1ms
34 timing arborist:ctor Completed in 1ms
35 timing idealTree:init Completed in 133ms
36 timing idealTree:userRequests Completed in 2ms
37 silly idealTree buildDeps
38 silly fetch manifest [email protected]
39 timing arborist:ctor Completed in 0ms
40 http fetch GET 200 https://registry.npmjs.org/semver 207ms (cache miss)
41 silly placeDep ROOT [email protected] REPLACE for: want: 7.5.4
42 silly fetch manifest lru-cache@^6.0.0
43 http fetch GET 200 https://registry.npmjs.org/lru-cache 51ms (cache miss)
44 timing idealTree:#root Completed in 271ms
45 silly fetch manifest semver@2 || 3 || 4 || 5
46 silly placeDep node_modules/normalize-package-data [email protected] OK for: [email protected] want: 2 || 3 || 4 || 5
47 timing idealTree:node_modules/normalize-package-data Completed in 6ms
48 silly placeDep ROOT [email protected] OK for: [email protected] want: ^6.0.0
49 silly fetch manifest yallist@^4.0.0
50 http fetch GET 200 https://registry.npmjs.org/yallist 32ms (cache miss)
51 timing idealTree:node_modules/semver Completed in 33ms
52 silly placeDep ROOT [email protected] OK for: [email protected] want: ^4.0.0
53 timing idealTree:node_modules/lru-cache Completed in 1ms
54 timing idealTree:node_modules/yallist Completed in 0ms
55 timing idealTree:node_modules/normalize-package-data/node_modules/semver Completed in 0ms
56 timing idealTree:buildDeps Completed in 313ms
57 timing idealTree:fixDepFlags Completed in 2ms
58 timing idealTree Completed in 454ms
59 timing reify:loadTrees Completed in 455ms
60 timing reify:diffTrees Completed in 5ms
61 silly reify mark retired [
61 silly reify '/meshcentral/node_modules/semver',
61 silly reify '/meshcentral/node_modules/.bin/semver'
61 silly reify ]
62 silly reify moves {
62 silly reify '/meshcentral/node_modules/semver': '/meshcentral/node_modules/.semver-pV4TNrh9',
62 silly reify '/meshcentral/node_modules/.bin/semver': '/meshcentral/node_modules/.bin/.semver-La7MnL0Q'
62 silly reify }
63 timing reify:retireShallow Completed in 3ms
64 timing reify:createSparse Completed in 0ms
65 silly reify mark deleted [
65 silly reify '/meshcentral/node_modules/uglify-js',
65 silly reify '/meshcentral/node_modules/.bin/uglifyjs'
65 silly reify ]
66 timing reify:trashOmits Completed in 1ms
67 timing reify:loadBundles Completed in 0ms
68 silly tarball no local data for yallist@https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz. Extracting by manifest.
69 silly tarball no local data for semver@https://registry.npmjs.org/semver/-/semver-7.5.4.tgz. Extracting by manifest.
70 silly tarball no local data for lru-cache@https://registry.npmjs.org/lru-cache/-/lru-cache-6.0.0.tgz. Extracting by manifest.
71 timing reifyNode:node_modules/normalize-package-data/node_modules/semver Completed in 24ms
72 http fetch GET 200 https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz 41ms (cache miss)
73 timing reifyNode:node_modules/yallist Completed in 47ms
74 http fetch GET 200 https://registry.npmjs.org/lru-cache/-/lru-cache-6.0.0.tgz 61ms (cache miss)
75 timing reifyNode:node_modules/lru-cache Completed in 69ms
76 http fetch GET 200 https://registry.npmjs.org/semver/-/semver-7.5.4.tgz 75ms (cache miss)
77 timing reifyNode:node_modules/semver Completed in 81ms
78 timing reify:unpack Completed in 82ms
79 timing reify:unretire Completed in 0ms
80 timing build:queue Completed in 1ms
81 timing build:link:node_modules/semver Completed in 2ms
82 timing build:link:node_modules/normalize-package-data/node_modules/semver Completed in 1ms
83 timing build:link Completed in 2ms
84 timing build:deps Completed in 3ms
85 timing build Completed in 3ms
86 timing reify:build Completed in 3ms
87 timing reify:trash Completed in 3ms
88 timing reify:save Completed in 45ms
89 timing reify Completed in 617ms
90 silly ADD node_modules/yallist
91 silly ADD node_modules/lru-cache
92 silly CHANGE node_modules/semver
93 silly ADD node_modules/normalize-package-data/node_modules/semver
94 timing command:install Completed in 623ms
95 verbose exit 0
96 timing npm Completed in 661ms
97 info ok

0 replies

NetBLOKS · 2024-02-06T15:26:50Z

NetBLOKS
Feb 6, 2024
Author

But I Do get a cross-origin error in Firefox / Chrome fomr time to time and a message about missing backend:

Quellübergreifende (Cross-Origin) Anfrage blockiert: Die Gleiche-Quelle-Regel verbietet das Lesen der externen Ressource auf https://api.netbloks.de/core/dashinfo/. (Grund: CORS-Anfrage schlug fehl). Statuscode: (null).

https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSMissingAllowOrigin?utm_source=devtools&utm_medium=firefox-cors-errors&utm_campaign=default

0 replies

dinger1986 · 2024-02-06T15:30:44Z

dinger1986
Feb 6, 2024
Collaborator

output of cat /etc/hosts please

0 replies

NetBLOKS · 2024-02-06T15:37:59Z

NetBLOKS
Feb 6, 2024
Author

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

--- BEGIN PVE ---

136.243.250.101 rmm.netbloks.de rmm

--- END PVE ---

2a01:4f8:242:ef01::101 rmm.netbloks.de rmm

127.0.1.1 api.netbloks.de rmm.netbloks.de rem.netbloks.de api rmm rem
::1 api.netbloks.de rmm.netbloks.de rem.netbloks.de api rmm rem

Worked like this with 0.17.3

0 replies

NetBLOKS · 2024-02-06T15:51:43Z

NetBLOKS
Feb 6, 2024
Author

There has been a rework of the nats-server:
a12f0fe

A missconfigured nats would explain the symptoms

0 replies

dinger1986 · 2024-02-06T15:55:24Z

dinger1986
Feb 6, 2024
Collaborator

strange,
try changing 127.0.0.1 localhost to

127.0.0.1 localhost api.netbloks.de rmm.netbloks.de rem.netbloks.de

Sorry keep replying with an old account, dont know why!

Correct, nats was reworked to save cert issues and is now bound to 127.0.0.1 but I suspect your dns not set to 127.0.0.1 is effecting it

0 replies

NetBLOKS · 2024-02-06T15:58:13Z

NetBLOKS
Feb 6, 2024
Author

That makes sense.
The manual needs to be corrected as it states to use 127.0.1.1

I´ve restored a backup.
I will reinstall the update and try again with 127.0.0.1 and then report back.

0 replies

NetBLOKS · 2024-02-06T16:15:45Z

NetBLOKS
Feb 6, 2024
Author

Still no connection from Agents.

Host File:

127.0.0.1       localhost api.netbloks.de rmm.netbloks.de rem.netbloks.de api rmm rem
::1             localhost ip6-localhost ip6-loopback api.netbloks.de rmm.netbloks.de rem.netbloks.de api rmm rem
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
136.243.250.101 rmm.netbloks.de rmm
# --- END PVE ---

2a01:4f8:242:ef01::101 rmm.netbloks.de rmm

# 127.0.1.1 api.netbloks.de rmm.netbloks.de rem.netbloks.de
# ::1 api.netbloks.de rmm.netbloks.de rem.netbloks.de

0 replies

NetBLOKS · 2024-02-06T16:21:07Z

NetBLOKS
Feb 6, 2024
Author

For information:
All Agents are online in Meshcentral.
It is just the TRMM side of things not connecting.

0 replies

dinger1986 · 2024-02-06T16:32:28Z

dinger1986
Feb 6, 2024
Collaborator

can you try update.sh --force now you have done that

0 replies

NetBLOKS · 2024-02-06T16:41:12Z

NetBLOKS
Feb 6, 2024
Author

Did that to no avail.
Still all clients red.

0 replies

NetBLOKS · 2024-02-06T16:41:28Z

NetBLOKS
Feb 6, 2024
Author

Is there a method to recreate the nats-Config file?

0 replies

dinger1986 · 2024-02-06T16:46:39Z

dinger1986
Feb 6, 2024
Collaborator

update with force does that, shall get back to you, are you a sponsor?

0 replies

NetBLOKS · 2024-02-06T16:49:51Z

NetBLOKS
Feb 6, 2024
Author

update with force does that, shall get back to you, are you a sponsor?

Yes, I am.
Thanks

0 replies

dinger1986 · 2024-02-06T16:51:03Z

dinger1986
Feb 6, 2024
Collaborator

can you open a ticket please? https://support.amidaware.com

0 replies

NetBLOKS · 2024-02-06T16:54:25Z

NetBLOKS
Feb 6, 2024
Author

Done.
Ticket ID: "880"

0 replies

NetBLOKS · 2024-02-06T17:11:23Z

NetBLOKS
Feb 6, 2024
Author

I´ve found the error:

I hat the /etc/nginx/sites-enabled/rmm.conf File modified:

    location ~ ^/natsws {
#        proxy_pass http://127.0.0.1:9235;
        proxy_pass http://[::1]:9235;
        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

Since nats does not listen to localhost anymore, I had to specify 127.0.0.1 again:

    location ~ ^/natsws {
        proxy_pass http://127.0.0.1:9235;
#        proxy_pass http://[::1]:9235;
        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

1 reply

dinger1986 Feb 6, 2024
Collaborator

what am I missing though? those look identical

dinger1986 · 2024-02-06T17:12:50Z

dinger1986
Feb 6, 2024
Collaborator

ah so doing something custom :)

1 reply

NetBLOKS Feb 6, 2024
Author

ah so doing something custom :)

Exactly :-D
Sorry for the inconvenience and thanks a lot for the help!
I changed the setup for ipv6 support, even internally

wh1te909 · 2024-02-06T17:14:21Z

wh1te909
Feb 6, 2024
Maintainer

to revert to the previous behavior before the nats rework, you can add the following to /rmm/api/tacticalrmm/tacticalrmm/local_settings.py and then run ./update.sh --force

NATS_STD_BIND_HOST = "0.0.0.0"
NATS_WS_BIND_HOST = "0.0.0.0"
NATS_CONNECT_HOST = ALLOWED_HOSTS[0]

just make sure ports 9235 and 4222 are not exposed publicly if you need to do this

but yes this is only needed if you are doing something unsupported or custom.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

All Agents offline after Upgrade to 0.17.5 #1751

{{title}}

Replies: 20 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

All Agents offline after Upgrade to 0.17.5 #1751

NetBLOKS Feb 6, 2024

Replies: 20 comments · 3 replies

mflagler Feb 6, 2024

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

--- BEGIN PVE ---

--- END PVE ---

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

NetBLOKS Feb 6, 2024 Author

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

NetBLOKS Feb 6, 2024 Author

dinger1986 Feb 6, 2024 Collaborator

dinger1986 Feb 6, 2024 Collaborator

NetBLOKS Feb 6, 2024 Author

wh1te909 Feb 6, 2024 Maintainer

NetBLOKS
Feb 6, 2024

Replies: 20 comments 3 replies

mflagler
Feb 6, 2024

dinger1986 Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS
Feb 6, 2024
Author

NetBLOKS
Feb 6, 2024
Author

dinger1986 Feb 6, 2024
Collaborator

dinger1986
Feb 6, 2024
Collaborator

NetBLOKS Feb 6, 2024
Author

wh1te909
Feb 6, 2024
Maintainer