enabling https using letsencrypt results in not-trusted certificate from traefik #310

gtomasson · 2019-04-30T08:31:27Z

Hi,
I installed The Littlest Jupyterhu on my own server (ubuntu 18.04). When attempting to enable https using letsencrypt a certificate from traefik (but not letsencrypt) is issued and that certificate is not valid. Guidance would be very appreciated.

yuvipanda · 2019-04-30T17:22:28Z

Thanks for opening the issue! Can you try restarting traefik with 'sudo systemctl restart traefik' and see if that helps fix it? If not, can you show logs from 'sudo journalctl -u traefik'? That'll help us debug. Thank you!

gtomasson · 2019-04-30T18:54:36Z

Thank you very much.
I tried reloading (sudo tljh-config reload proxy) after restarting traefik w/o success
Below is the log

Apr 30 18:45:30 gunnart systemd[1]: Stopping traefik.service...
Apr 30 18:45:30 gunnart traefik[6073]: time="2019-04-30T18:45:30Z" level=info msg="I have to go..."
Apr 30 18:45:30 gunnart traefik[6073]: time="2019-04-30T18:45:30Z" level=info msg="Stopping server gracefully"
Apr 30 18:45:40 gunnart traefik[6073]: time="2019-04-30T18:45:40Z" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: read tcp 130.208.143.74:443->130.208.204.7:158
Apr 30 18:45:40 gunnart traefik[6073]: time="2019-04-30T18:45:40Z" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: read tcp 130.208.143.74:443->130.208.204.7:289
Apr 30 18:45:40 gunnart traefik[6073]: time="2019-04-30T18:45:40Z" level=info msg="Server stopped"
Apr 30 18:45:40 gunnart traefik[6073]: time="2019-04-30T18:45:40Z" level=info msg="Shutting down"
Apr 30 18:45:40 gunnart systemd[1]: Stopped traefik.service.
Apr 30 18:45:40 gunnart systemd[1]: Started traefik.service.
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Apr 30 18:45:40 gunnart traefik[3287]: time="2019-04-30T18:45:40Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.tra
Apr 30 18:45:41 gunnart traefik[3287]: time="2019-04-30T18:45:41Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc00019c340 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Co
Apr 30 18:45:41 gunnart traefik[3287]: time="2019-04-30T18:45:41Z" level=info msg="Preparing server https &{Address::443 TLS:0xc000219050 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> 
Apr 30 18:45:41 gunnart traefik[3287]: time="2019-04-30T18:45:41Z" level=info msg="Starting server on :80"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc000333a10 WhitelistSourceRange:[] Whi
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Starting server on :443"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Starting server on 127.0.0.1:8099"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"rules.toml\",\"Constraints\":null,\"Trace\":false,\"Templ
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Starting provider *acme.Provider {\"Email\":\"[email protected]\",\"ACMELogging\":false,\"CAServer\":\"https://ac
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \"n\" is duplicated in the configuration or validated by the domain {n []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \"r\" is duplicated in the configuration or validated by the domain {r []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \".\" is duplicated in the configuration or validated by the domain {. []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \"h\" is duplicated in the configuration or validated by the domain {h []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \"i\" is duplicated in the configuration or validated by the domain {i []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \".\" is duplicated in the configuration or validated by the domain {. []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="Domain \"i\" is duplicated in the configuration or validated by the domain {i []}. It will be processed once."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Testing certificate renew..."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Server configuration reloaded on :443"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:45:42 gunnart traefik[3287]: time="2019-04-30T18:45:42Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:45:43 gunnart traefik[3287]: time="2019-04-30T18:45:43Z" level=error msg="Unable to obtain ACME certificate for domains \"g\" : unable to generate a certificate for the domains [g]: acme: Error 
Apr 30 18:45:43 gunnart traefik[3287]: time="2019-04-30T18:45:43Z" level=error msg="Unable to obtain ACME certificate for domains \"u\" : unable to generate a certificate for the domains [u]: acme: Error 
Apr 30 18:45:43 gunnart traefik[3287]: time="2019-04-30T18:45:43Z" level=error msg="Unable to obtain ACME certificate for domains \"n\" : unable to generate a certificate for the domains [n]: acme: Error 
Apr 30 18:45:43 gunnart traefik[3287]: time="2019-04-30T18:45:43Z" level=error msg="Unable to obtain ACME certificate for domains \"i\" : unable to generate a certificate for the domains [i]: acme: Error 
Apr 30 18:45:43 gunnart traefik[3287]: time="2019-04-30T18:45:43Z" level=error msg="Unable to obtain ACME certificate for domains \"r\" : unable to generate a certificate for the domains [r]: acme: Error 
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=error msg="Unable to obtain ACME certificate for domains \"t\" : unable to generate a certificate for the domains [t]: acme: Error 
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=error msg="Unable to obtain ACME certificate for domains \"h\" : unable to generate a certificate for the domains [h]: acme: Error 
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=info msg="Server configuration reloaded on :443"
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=error msg="Unable to obtain ACME certificate for domains \".\" : unable to generate a certificate for the domains []: acme: Error 4
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=error msg="Unable to obtain ACME certificate for domains \"s\" : unable to generate a certificate for the domains [s]: acme: Error 
Apr 30 18:45:44 gunnart traefik[3287]: time="2019-04-30T18:45:44Z" level=error msg="Unable to obtain ACME certificate for domains \"a\" : unable to generate a certificate for the domains [a]: acme: Error 
Apr 30 18:46:55 gunnart systemd[1]: Stopping traefik.service...
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Server configuration reloaded on :443"
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="I have to go..."
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Stopping server gracefully"
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Server stopped"
Apr 30 18:46:55 gunnart traefik[3287]: time="2019-04-30T18:46:55Z" level=info msg="Shutting down"
Apr 30 18:46:55 gunnart systemd[1]: Stopped traefik.service.
Apr 30 18:46:55 gunnart systemd[1]: Started traefik.service.
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
Apr 30 18:46:56 gunnart traefik[3361]: time="2019-04-30T18:46:56Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.tra
Apr 30 18:46:57 gunnart traefik[3361]: time="2019-04-30T18:46:57Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc0003283c0 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Co
Apr 30 18:46:57 gunnart traefik[3361]: time="2019-04-30T18:46:57Z" level=info msg="Preparing server https &{Address::443 TLS:0xc000219050 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> 
Apr 30 18:46:57 gunnart traefik[3361]: time="2019-04-30T18:46:57Z" level=info msg="Starting server on :80"
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc0002942d0 WhitelistSourceRange:[] Whi
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Starting server on :443"
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Starting server on 127.0.0.1:8099"
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"rules.toml\",\"Constraints\":null,\"Trace\":false,\"Templ
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Starting provider *acme.Provider {\"Email\":\"[email protected]\",\"ACMELogging\":false,\"CAServer\":\"https://ac
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \"n\" is duplicated in the configuration or validated by the domain {n []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \"r\" is duplicated in the configuration or validated by the domain {r []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \".\" is duplicated in the configuration or validated by the domain {. []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \"h\" is duplicated in the configuration or validated by the domain {h []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \"i\" is duplicated in the configuration or validated by the domain {i []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \".\" is duplicated in the configuration or validated by the domain {. []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="Domain \"i\" is duplicated in the configuration or validated by the domain {i []}. It will be processed once."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=info msg="Testing certificate renew..."
Apr 30 18:46:58 gunnart traefik[3361]: time="2019-04-30T18:46:58Z" level=warning msg="FQDN detected, please remove the trailing dot: ."
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=info msg="Server configuration reloaded on :443"
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=error msg="Unable to obtain ACME certificate for domains \"g\" : unable to generate a certificate for the domains [g]: acme: Error 
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=error msg="Unable to obtain ACME certificate for domains \"h\" : unable to generate a certificate for the domains [h]: acme: Error 
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=error msg="Unable to obtain ACME certificate for domains \"t\" : unable to generate a certificate for the domains [t]: acme: Error 
Apr 30 18:46:59 gunnart traefik[3361]: time="2019-04-30T18:46:59Z" level=error msg="Unable to obtain ACME certificate for domains \"n\" : unable to generate a certificate for the domains [n]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \"i\" : unable to generate a certificate for the domains [i]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on :443"
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \"r\" : unable to generate a certificate for the domains [r]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \"u\" : unable to generate a certificate for the domains [u]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \"a\" : unable to generate a certificate for the domains [a]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \".\" : unable to generate a certificate for the domains []: acme: Error 4
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=error msg="Unable to obtain ACME certificate for domains \"s\" : unable to generate a certificate for the domains [s]: acme: Error 
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on :80"
Apr 30 18:47:00 gunnart traefik[3361]: time="2019-04-30T18:47:00Z" level=info msg="Server configuration reloaded on :443"

yuvipanda · 2019-04-30T19:04:24Z

It looks like the domain is not being read properly. Can you look at the contents of the file /opt/tljh/config.yaml, redact any sensitive info, and paste it here?

gtomasson · 2019-04-30T19:12:44Z

yea, it looks strange (only one letter at the time) in the error log. Here are the contents of the config file

root@gunnart:/opt/tljh/config# sudo cat config.yaml
users:
  admin:
  - adgunnar
https:
  enabled: true
  letsencrypt:
    email: [email protected]
    domains: gunnart.rhi.hi.is
root@gunnart:/opt/tljh/config# ^C

yuvipanda · 2019-04-30T19:15:29Z

@gtomasson not sure why 'domains' isn't a list but a string. Can you show me the tljh command you used to set the value?

Either way, if you change the file to read:

users:
  admin:
  - adgunnar
https:
  enabled: true
  letsencrypt:
    email: [email protected]
    domains:
      - gunnart.rhi.hi.is

And do the reload, it should work.

gtomasson · 2019-04-30T19:38:43Z

Great. thank you!!. It worked!!
I used the command gunnart@gunnart:~$ sudo tljh-config add-item https.letsencrypt.domains gunnart.rhi.hi.is

but now it gave me an error,

I manually corrected the file per your instructions and it worked fine.

yuvipanda · 2019-04-30T19:58:20Z

yw, @gtomasson. Can you tell us what error it gave you when you ran the command? That might help fix this at its root.

gtomasson · 2019-04-30T20:45:25Z

sure, here is the report

gunnart@gunnart:~$ sudo tljh-config add-item https.letsencrypt.domains gunnart.rhi.hi.is
Traceback (most recent call last):
  File "/usr/bin/tljh-config", line 11, in <module>
    load_entry_point('the-littlest-jupyterhub==0.1', 'console_scripts', 'tljh-config')()
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/config.py", line 321, in main
    add_config_value(args.config_path, args.key_path, parse_value(args.value))
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/config.py", line 156, in add_config_value
    config = add_item_to_config(config, key_path, value)
  File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/config.py", line 79, in add_item_to_config
    cur_part.append(value)
AttributeError: 'str' object has no attribute 'append'

yuvipanda · 2019-05-19T02:00:20Z

Thanks @gtomasson. I think there might have accidentally been a 'set' command first. I think we should use this issue to provide more meaningful error messages in this case.

consideRatio · 2021-10-22T13:25:51Z

I'm closing this as something that would be resolved by having schema validation, which is represented by #725.

yuvipanda assigned yuvipanda and unassigned yuvipanda May 19, 2019

yuvipanda added bug Something isn't working support Support questions (should be on discourse.jupyter.org instead) and removed bug Something isn't working labels May 20, 2019

consideRatio closed this as completed Oct 22, 2021

consideRatio mentioned this issue Oct 22, 2021

JSONSchema validation of our config #725

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enabling https using letsencrypt results in not-trusted certificate from traefik #310

enabling https using letsencrypt results in not-trusted certificate from traefik #310

gtomasson commented Apr 30, 2019

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 •

edited by consideRatio

Loading

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 •

edited by yuvipanda

Loading

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 •

edited by consideRatio

Loading

yuvipanda commented May 19, 2019

consideRatio commented Oct 22, 2021

enabling https using letsencrypt results in not-trusted certificate from traefik #310

enabling https using letsencrypt results in not-trusted certificate from traefik #310

Comments

gtomasson commented Apr 30, 2019

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 • edited by consideRatio Loading

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 • edited by yuvipanda Loading

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019

yuvipanda commented Apr 30, 2019

gtomasson commented Apr 30, 2019 • edited by consideRatio Loading

yuvipanda commented May 19, 2019

consideRatio commented Oct 22, 2021

gtomasson commented Apr 30, 2019 •

edited by consideRatio

Loading

gtomasson commented Apr 30, 2019 •

edited by yuvipanda

Loading

gtomasson commented Apr 30, 2019 •

edited by consideRatio

Loading