Unable to download the mesh agent from the RMM

[dreamtim]

Server Info:

• OS: Debian 5.10.46-4 (2021-08-03) x86_64
• Browser: NA
• RMM Version: 0.13.3

Installation Method:

• Standard
• [ x ] Docker

Agent Info :

• Agent version : Agent v2.0.3
• Agent OS: Win 10 Pro

Describe the bug
The installation of the agent fail on a specific computer (seems to not occur on other computers) with the following error : "Unable to download the mesh agent from the RMM"

To Reproduce
Steps to reproduce the behavior:

1. Download Dynamically generated exe or Powershell or Manual install script
2. Go to on the computer
3. Launch the install script
4. Installation fail when the RMM is downloaded

Expected behavior
Agent should properly setup and computer display in the RMM

Screenshots

Additional context

• Agent 2.02 was previously installed on this computer and when updating it asked to remove the agent to continue with the installation process.
• I also did check dns resolving and ping to the rmm server and it seems to work properly. I had no issue with other computers.

[wh1te909]

https://docs.tacticalrmm.com/install_agent/#stuck-at-downloading-mesh-agent since you are on docker please run sudo docker-compose exec tactical-backend python manage.py check_mesh redact sensitive info and paste the output here

[dreamtim]

Hi @wh1te909

thank you for your help. Here is the log :

Mesh site: https://mesh.rmm.xxxx.com
Mesh username: tactical-admin
Mesh token: xxxx
Mesh device group: TacticalRMM
Auth token ok:
ccccc
Mesh url ok:
ws://tactical-meshcentral:4443/control.ashx?auth=xxxxx
Error: you are using a custom mesh device group and need to add its name in Global Settings > MeshCentral

Not sure what i'm supposed to do to fix the error of the last line

[wh1te909]

open your meshcentral site, and check screenshot above and you will see your device group. in my screenshot the device group name is "TacticalRMM" and so whatever yours is, you need to copy paste that name and then from Tactical's UI go to Settings > Global Settings > MeshCentral and paste the device group name:

[dreamtim]

That made it. Thanks for your quick assistance !

[multispeed]

Hi i am experiencing the same error "Unable to download the mesh agent from the RMM"

Here are my log files (i deleted some pieces from auth keys and replaced my domain with server.nl)

==============================================================================
time="2022-11-09T21:07:44+01:00" level=fatal msg="Unable to download the mesh agent from the RMM. "
exit status 1

Then i also tried the suggestion by @wh1te909 to run the command python manage.py check_mesh. But all i got was

root@patch:/rmm/api/tacticalrmm# python manage.py
File "manage.py", line 16
) from exc
^
SyntaxError: invalid syntax
root@patch:/rmm/api/tacticalrmm# python manage.py check_mesh
File "manage.py", line 16
) from exc
^
SyntaxError: invalid syntax

[wh1te909]

@multispeed i edited your post and deleted some content, you had sensitive auth tokens in it. please be careful when posting logs and be sure to redact sensitive stuff like tokens, domains etc.

you are not running the check_mesh command correctly. Please copy paste the command from the docs: https://docs.tacticalrmm.com/troubleshooting/#agents-not-installing-or-updating

And be sure to redact sensitive info before posting results

/rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh

[wh1te909]

@multispeed lol sorry just read your comment, missed that you replaced the tokens lmao

[multispeed]

Hi Dan,

No worries haha

I see what mistake i made with the check_mesh command. See output below:

root@patch:/rmm/api/tacticalrmm# /rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh
Mesh site: https://mesh.server.nl
Mesh username: kbppnaua
Mesh token: tokenkeything
Mesh device group: TacticalRMM
Auth token ok:
tokenkeything
Mesh url ok:
wss://mesh.server.nl/control.ashx?auth=tokenkeything
Error getting mesh device id:
server rejected WebSocket connection: HTTP 502

[dinger1986]

Ok, what's the output of sudo systemctl status mongod

[wh1te909]

yea most likely mongo is down, see #1340

[multispeed]

After dinger1986's comment about memory and the memory comment in #1340 i've decided to up the memory on the virtual machine running rmm, and this did the trick.

Solved!

[wh1te909]

@multispeed just curious how many agents do u have? 8GB should be plenty. although lately mesh has been using more ram than usual

[multispeed]

@multispeed just curious how many agents do u have? 8GB should be plenty. although lately mesh has been using more ram than usual

Sure! i've got 129 agents running

[SecureCPU]

I'm having this same issue.
Running it on a Ubuntu 20.04 LTS Virtual Server.
Can anyone help me with it?

[SecureCPU]

I ran the script: /rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh

I got the following response:

Mesh site: https://mesh.******.com
Mesh username: ******
Mesh token: **************************************************************
Mesh device group: TacticalRMM
Auth token ok:
************************************************************************************
Mesh url ok:
wss://mesh.***********.com/control.ashx?auth=******************************************=
**Error getting mesh device id:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1002)**

It seems to have something to do with me Mesh Agent SSL.
But everything is up to date.

[SecureCPU]

I can remote into any of my client agents just fine, but I can't install any new agents.

[silversword411]

certificate has expired (_ssl.c:1002)**

Your SSL certs need renewing every 3 months, you probably haven't done that
https://docs.tacticalrmm.com/update_server/#keeping-your-lets-encrypt-ssl-certificate-up-to-date

[SecureCPU]

I use Nginx Proxy Manager, and its up to date.
Is there something I'm missing?

[SecureCPU]

I followed the directions. It worked.
Thanks!!!!!!

[otuva]

@wh1te909

Hi, even though everything seems to be ok, still getting this error

$ /rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh

Mesh site: https://mesh.*
Mesh username: ****
Mesh token: ****
Mesh device group: TacticalRMM
Auth token ok: ****
Mesh url ok: ****
Mesh device id ok: ****

Debug message from one of the agent installations

==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/meshexe/  HTTP/1.1
HOST   : api.*
HEADERS:
        Accept: application/json
        Authorization: Token *
        Content-Type: application/json
        User-Agent: go-resty/2.10.0 (https://github.com/go-resty/resty)
BODY   :
{
   "goarch": "amd64",
   "plat": "windows"
}

------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-02-18T09:05:53.3716665+03:00
TIME DURATION: 192.612ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 35
        Content-Type: application/json
        Date: Sun, 18 Feb 2024 06:00:23 GMT
        Server: nginx
        Vary: origin
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================
time="2024-02-18T09:06:04+03:00" level=fatal msg="Unable to download the mesh agent from the RMM. "
exit status 1

[dinger1986]

ok, so how did you add the certificates?

[otuva]

@dinger1986

After reading the things I wrote, sorry if I came as an a-hole, man. This specific issue was bothering me so much that I got impatient.

Weird thing because I even made sure venv under rmm dir is using correct certificates via certify locations and sending requests to various sites to make sure self signed certificates are not causing the problem.

Because if I didn't add them, venv python console would spit bunch of ssl errors to any site. And yet even though there were no errors from the console, Mesh just didn't install seemingly no way to troubleshoot or even trace.

[dinger1986]

so is it still not working? can you install mesh agent using the site directly?

[otuva]

no it's working after installing pip-system-certs
thanks

[dinger1986]

Cool. Glad it's working 👍

[MrGoodbody]

why is domain not reachable from outside? That sounds like a DNS issue and nothing to with TRMM

using AD CS for https certificates on local network using AD DS for domains

I have no issues reaching the web interface, api or mesh from any of the clients.

I have the same setup, after a lot of digging i found the problem.
the request command is not using the system ca store.
i enabled the enviroment
source /rmm/api/env/bin/activate
and installed pip-system-certs
pip install pip-system-certs
after service restart the problem is gone
service rmm restart

[wh1te909]

nice catch!

[HauptlingHD]

I'm having the same problem, and I'm getting the same debug message as otuva. I might also have found the issue, but don't know the solution. (running on Debian 12, 1 CPU, 4GB RAM)

Output of sudo systemctl status mongod.service is Unit mongod.service could not be found.

It is not stopped, but not present in the first place. From the issue of multispeed I'm gathering that that should be present and running.

To already rule some things out: I can ping all 3 domains, all websites load correctly in the browser and HTTPS is working, there are no other obvious issues or errors until I try to install my first agent.

Did I possibly do something wrong during installation, or can I check something else to rule out possible issues?

[boonkerz]

nothing special
pve because i use proxmox :)

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
127.0.1.1 rmm mesh.domain.de api.domain.de rmm.domain.de
# --- END PVE ---

[dinger1986]

No other text in the hosts file?

Anything managing it?

[boonkerz]

no i only nano /etc/hosts added 3 domains and works :)

[dinger1986]

Where do you have it installed? Is it Debian or Ubuntu?

[boonkerz]

ubuntu 22

Proxmox
--- Opnsense handles all incoming traffic
--- RMM here added hosts and ubuntu 22

[wh1te909]

I'll have a proper fix for this in the next release, so we don't need to mess with /etc/hosts anymore.

[johnnyq]

I seem to be having the same issue as well but the fixes seem to be unrelated, this started back on the last release
It fails to install the agent

I can access existing my tacticalrmm no problem and I can access mesh.mydomain no problem

When I run this command

I get this error

It states the mesh group mismatch and I checked and they match just fine.

I tried Clearing out the the Mesh Device Group Name and just put in random characters hit save then went back and filled it back in as TacticalRMM then hit save.

• I cleared all Caches
• ran all the server maintence tasks
• restarted the server
• Force Reinstalled Tractical RMM
• Tried to Resync Permissions according to recently released video on resyncing permissions with mesh central.

[wh1te909]

@johnnyq can you try the steps here but only steps 4 and 5 and lmk if fixes your issue

[johnnyq]

@wh1te909 thanks for the fast reply, I actually just found out what the issue was I changed the Username under MeshCentral Settings to be my name instead of the one tactical created initially. Thinking this would be what I change when I connect to users machines so they see my name and not a bunch random letters. I realized this was not the place to change that.

[wh1te909]

https://docs.tacticalrmm.com/mesh_integration/#customize-take-control-username

[johnnyq]

But just an idea to alleviate someone else going through the same thing, the error should state something like make sure the Tactical Username and the MeshCentral Username are a match along with the group Device name.

[herland]

I seem to be having the same issue, It fails to install the agent:

Server Info:

• OS: Debian GNU/Linux 12 (bookworm)
• RMM Version: 0.18.2 - Self signed
• Installation Method: - Standard
• Agent version : tacticalagent-v2.7.0-windows-amd64

c:\Users\administrator\Downloads>trmm-********-workstation-amd64.exe -log debug
ProgramData: C:\ProgramData\TacticalRMM
Installer: C:\ProgramData\TacticalRMM\tacticalagent-v2.7.0-windows-amd64.exe
Tactical Agent: C:\Program Files\TacticalAgent\tacticalrmm.exe
Download URL: https://github.com/amidaware/rmmagent/releases/download/v2.7.0/tacticalagent-v2.7.0-windows-amd64.exe
Install command: C:\Program Files\TacticalAgent\tacticalrmm.exe -m install --api https://api.ope******.com --client-id 1 --site-id 1 --agent-type workstation --auth 1ca459f424d96f65b9cd4b193095f5dd1301e92c43cc04442e656deafb8b2cb1 -log debug
Downloading agent...
Extracting files...
Installation starting.
time="2024-05-14T18:14:32+02:00" level=debug msg="{Hostname:PDL01 Arch: AgentID: BaseURL: ApiURL: Token: AgentPK:0 Cert: ProgramDir:C:\\Program Files\\TacticalAgent EXE:C:\\Program Files\\TacticalAgent\\tacticalrmm.exe SystemDrive:C: WinTmpDir:C:\\ProgramData\\TacticalRMM WinRunAsUserTmpDir:C:\\ProgramData\\TacticalRMM MeshInstaller:meshagent.exe MeshSystemEXE:C:\\Program Files\\Mesh Agent\\MeshAgent.exe MeshSVC:mesh agent PyBin:C:\\Program Files\\TacticalAgent\\py38-x64\\python.exe NuBin:C:\\Program Files\\TacticalAgent\\bin\\nu.exe DenoBin:C:\\Program Files\\TacticalAgent\\bin\\deno.exe AgentHeader:trmm/2.7.0/windows/amd64 Headers:map[] Logger:0xc0000d4b80 Version:2.7.0 Debug:true rClient:0xc0001c8a00 Proxy: LogTo: LogFile:<nil> Platform:windows GoArch:amd64 ServiceConfig:0xc0000ea9a0 NatsServer:wss://:443 NatsProxyPath:natsws NatsProxyPort:443 NatsPingInterval:40 NatsWSCompression:true Insecure:false}\n"
time="2024-05-14T18:14:32+02:00" level=debug msg="Agent ID: ToISgMwWEonVmKcMYuWdrKXYcyBwnqJuvcOXDDrn"
time="2024-05-14T18:14:32+02:00" level=debug msg="API: api.ope******.com"
time="2024-05-14T18:14:32+02:00" level=debug msg="Base URL: https://api.ope******.com"
2024/05/14 18:14:33.002589 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
GET  /api/v3/installer/  HTTP/1.1
HOST   : api.ope******.com
HEADERS:
        Accept: application/json
        Authorization: Token 1ca459f424d96f65b9cd4b193095f5dd1301e92c43cc04442e656deafb8b2cb1
        Content-Type: application/json
        User-Agent: go-resty/2.10.0 (https://github.com/go-resty/resty)
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-05-14T18:14:33.0025898+02:00
TIME DURATION: 757.6504ms
HEADERS      :
        Allow: GET, POST, HEAD, OPTIONS
        Content-Length: 4
        Content-Type: application/json
        Date: Tue, 14 May 2024 16:14:33 GMT
        Server: nginx
        Vary: origin
        X-Content-Type-Options: nosniff
BODY         :
"ok"
==============================================================================
2024/05/14 18:14:33.039741 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/installer/  HTTP/1.1
HOST   : api.ope******.com
HEADERS:
        Accept: application/json
        Authorization: Token 1ca459f424d96f65b9cd4b193095f5dd1301e92c43cc04442e656deafb8b2cb1
        Content-Type: application/json
        User-Agent: go-resty/2.10.0 (https://github.com/go-resty/resty)
BODY   :
{
   "version": "2.7.0"
}

------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-05-14T18:14:33.0397418+02:00
TIME DURATION: 28.307ms
HEADERS      :
        Allow: GET, POST, HEAD, OPTIONS
        Content-Length: 4
        Content-Type: application/json
        Date: Tue, 14 May 2024 16:14:33 GMT
        Server: nginx
        Vary: origin
        X-Content-Type-Options: nosniff
BODY         :
"ok"
==============================================================================
time="2024-05-14T18:14:33+02:00" level=info msg="Downloading mesh agent..."
2024/05/14 18:14:33.252387 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/meshexe/  HTTP/1.1
HOST   : api.ope******.com
HEADERS:
        Accept: application/json
        Authorization: Token 1ca459f424d96f65b9cd4b193095f5dd1301e92c43cc04442e656deafb8b2cb1
        Content-Type: application/json
        User-Agent: go-resty/2.10.0 (https://github.com/go-resty/resty)
BODY   :
{
   "goarch": "amd64",
   "plat": "windows"
}

------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2024-05-14T18:14:33.2523878+02:00
TIME DURATION: 205.5187ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 35
        Content-Type: application/json
        Date: Tue, 14 May 2024 16:14:33 GMT
        Server: nginx
        Vary: origin
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================
time="2024-05-14T18:14:38+02:00" level=fatal msg="Unable to download the mesh agent from the RMM. "
exit status 1

c:\Users\administrator\Downloads>

I tried to change the user administrator name on RMM but this is not reflected on mesh is this normal?

tactical@TacticalRMM:~$ /rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh
Mesh site: https://mesh.open*******.com
Mesh username: dqoszabt
Mesh token: 67c6447479702e115eb056b4418442736d42a603*********************************************
Mesh device group: TacticalRMM
Auth token ok:
4fRQnZeVCT9PVjY7zZwfcAknRqvtbSNn2t8AXA7b3FnT9YhDXM1*********************************************
Mesh url ok:
ws://127.0.0.1:4430/control.ashx?auth=UJcLSBGsw85G4JJnbSElcTpc*********************************************
Mesh device id ok: 5hGmevpLrrO3XuQGM18b*********************************************

file host: 10...11 api.opens******.com rmm.open*****.com mesh.open*****.com

[dinger1986]

You wouldn't necessarily see errors just some weird behaviour if not using the full chain.

Permissions - https://docs.tacticalrmm.com/functions/permissions/

[wh1te909]

@herland I deleted your comment because it's not a good idea to post all your auth tokens ( i know you redacted your domain but still you should also redact auth tokens and other sensitive info). yes the error you had is normal and shouldn't affect functionality.

[herland]

command to Check chain: openssl s_client -showcerts -connect domain.com:443
error: Verify return code: 19 (self signed certificate in certificate chain)

at this point the problem was in the chain, I solved it like this:
the certificates are generated in the local CA (Domain), the server "RMM" and Clients need this "file.crt" installed locally, so that they too can trust it.

Install cert Server:

Use command: sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt
Update the CA store: sudo update-ca-certificates

Install certificate Client windows with "local computer"

thx @dinger1986 and @wh1te909

[dinger1986]

oh lol so it was self signed, I thought it was just a custom cert! anyways glad its working

[wh1te909]

Did you not see this?