Caprover restarts endlessly

[onikienko]

I added a new app to Caprover today and was unable to "Enable HTTPS" for this app (got Domain Verification Failed - Error 1107). All old apps work via https with no problems.

I found that caprover restarts every minute or so.

$ docker service ps captain-captain --no-trunc
ID                          NAME                    IMAGE                      NODE               DESIRED STATE   CURRENT STATE            ERROR                       PORTS
v5dboyk484fbuevhg8581vmj8   captain-captain.1       caprover/caprover:1.11.1   SecondAwesome-VM   Running         Running 28 seconds ago                               *:3000->3000/tcp,*:3000->3000/tcp
sm3krhjk9i3kz68dqgeemeoux    \_ captain-captain.1   caprover/caprover:1.11.1   SecondAwesome-VM   Shutdown        Failed 34 seconds ago    "task: non-zero exit (1)"   
qg3x99aj4gc2ap0ctniq9jhbu    \_ captain-captain.1   caprover/caprover:1.11.1   SecondAwesome-VM   Shutdown        Failed 2 minutes ago     "task: non-zero exit (1)"   
xq00v8xkztk44i38ldkjwcop6    \_ captain-captain.1   caprover/caprover:1.11.1   SecondAwesome-VM   Shutdown        Failed 4 minutes ago     "task: non-zero exit (1)"   
x7rtml4gmem5ei46reqj2lwtq    \_ captain-captain.1   caprover/caprover:1.11.1   SecondAwesome-VM   Shutdown        Failed 5 minutes ago     "task: non-zero exit (1)" 

It looks like there is a problem with domain verification:

$ docker service logs captain-captain --since 60m

captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Captain Starting ...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.319 am    Emptying generated and temp folders.
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.324 am    Ensuring directories are available on host. Started.
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.329 am    Ensuring directories are available on host. Finished.
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.332 am    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.337 am    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.338 am    Migration not needed, skipping.
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.342 am    Copying fake certificates...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.344 am    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.345 am    Locking NGINX configuration reloading...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.355 am    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.358 am    Captain Nginx is already running.. 
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.360 am    Updating NGINX service...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:07.367 am    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:12.371 am    Pruning containers...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:12.372 am    NGINX is fully set up and working...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:12.374 am    Captain Certbot is already running.. 
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:12.376 am    Updating Certbot service...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:15.387 am    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:17.385 am    Pruning containers...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:18.391 am    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:21.397 am    Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:24.403 am    Retrying to get containerId for captain-certbot retry count:3
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:27.406 am    Retrying to get containerId for captain-certbot retry count:4
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:30.410 am    Retrying to get containerId for captain-certbot retry count:5
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:30.413 am    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:31.095 am    **** Captain is initialized and ready to serve you! ****
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.169 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Verification Failed for captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Verification Failed for captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:54:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.170 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Error        Error: getaddrinfo ENOTFOUND captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Error        Error: getaddrinfo ENOTFOUND captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:55:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.170 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: body         undefined
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: body         undefined
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:56:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.170 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: randomUuid   addf132f-e9e2-4554-835f-e779c6308f33
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: randomUuid   addf132f-e9e2-4554-835f-e779c6308f33
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:57:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.170 am    Captain health check failed: #0 at captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:32.170 am    NGINX health check failed: #0
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:41.096 am    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:41.719 am    executeCommand Container: captain-certbot certbot renew --non-interactive 
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:43.146 am    Updating Load Balancer - renewAllCerts
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:43.146 am    Locking NGINX configuration reloading...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:43.154 am    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:43.154 am    sendReloadSignal...
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:43.157 am    Kill HUP Container: f18048d3dfd62d5a78d5e70e70ed1c22ede47601ed755cf619ccb795be0ce53b
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.177 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Verification Failed for captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Verification Failed for captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:54:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.178 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Error        Error: getaddrinfo ENOTFOUND captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: Error        Error: getaddrinfo ENOTFOUND captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:55:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.178 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: body         undefined
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: body         undefined
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:56:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.178 am    
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: randomUuid   b775b04a-ef81-4023-af9f-09a20b2decf9
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | Error: Wrapped: randomUuid   b775b04a-ef81-4023-af9f-09a20b2decf9
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request._callback (/usr/src/app/built/user/system/DomainResolveChecker.js:57:42)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at ClientRequest.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.socketErrorListener (node:_http_client:501:9)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at Socket.emit (node:events:514:28)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    |     at emitErrorNT (node:internal/streams/destroy:151:8)
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.178 am    Captain health check failed: #1 at captain.my-domain-name.com
captain-captain.1.xxrrnmi7pbrd@SecondAwesome-VM    | May 3rd 2024, 1:47:53.178 am    NGINX health check failed: #1

[githubsaturn]

This has nothing to do with the new app. Something has changed about your domain's DNS setting:

getaddrinfo ENOTFOUND captain.my-domain-name.com

Have you checked your DNS settings? Try https://mxtoolbox.com/DNSLookup.aspx

[onikienko]

Yes. I checked via https://mxtoolbox.com/DNSLookup.aspx
Everything looks fine. It points to my server IP.

However, a few months ago my server IP address was changed. I updated DNS records to point to the new IP. All apps worked after this IP update, but now I can't Enable HTTPS for the new app and found that caprover restarts in a loop.

[githubsaturn]

You can run this to avoid the restarts:

echo  "{\"skipVerifyingDomains\":\"true\"}" >  /captain/data/config-override.json
docker service update captain-captain --force

But the fact that you can't enable HTTPS is a bit weird. It again hints that there is a DNS problem. Are you using Cloudflare?

[onikienko]

I do not use Cloudflare.

Once I updated config-override.json like you (and Troubleshooting guide) suggested, caprover stopped to restart, and I was able to "Enable HTTPS" for the new app. That's odd. Why I have this Domain Verification issue at all? It does not look like caprover keeps old server IP somewhere in configs.

Anyway. Thank you for your support and caprover project!

[githubsaturn]

It's likely that your server have the IP address cached.

[onikienko]

I see. Thank you!