Failed to connect to 127.0.0.1 port 8099: Connection refused

[jtpio]

After installing Traefik with:

python3 -m jupyterhub_traefik_proxy.install --output=/usr/local/bin

Using the following config:

from jupyterhub.auth import DummyAuthenticator
from jupyterhub.spawner import SimpleLocalProcessSpawner
from jupyterhub_traefik_proxy import TraefikTomlProxy

c = get_config()  # noqa

c.JupyterHub.authenticator_class = DummyAuthenticator
c.JupyterHub.spawner_class = SimpleLocalProcessSpawner
c.JupyterHub.proxy_class = TraefikTomlProxy
c.JupyterHub.allow_named_servers = True

And starting JupyterHub with:

python -m jupyterhub -f jupyterhub_config.py --debug

The hub is sometimes unable to reach the Traefik API:

[D 2020-02-28 12:41:03.057 JupyterHub proxy:153] Fetching traefik api http://127.0.0.1:8099/api/providers/file
[E 2020-02-28 12:41:03.059 JupyterHub proxy:172] Error checking for traefik static configuration
    Traceback (most recent call last):
      File "/home/jtp/miniconda/envs/jhub-traefik/lib/python3.8/site-packages/jupyterhub_traefik_proxy/proxy.py", line 170, in _check_traefik_static_conf_ready
        resp = await self._traefik_api_request("/api/providers/" + provider)
      File "/home/jtp/miniconda/envs/jhub-traefik/lib/python3.8/site-packages/jupyterhub_traefik_proxy/proxy.py", line 154, in _traefik_api_request
        resp = await AsyncHTTPClient().fetch(
    tornado.curl_httpclient.CurlError: HTTP 599: Failed to connect to 127.0.0.1 port 8099: Connection refused

Restarting the hub process fixes it sometimes, which makes it look like some kind of race condition.

More complete example in this gist: https://gist.github.com/jtpio/88d98746ed8577403d4e4497370c6950

[GeorgianaElena]

@jtpio, as long as the hub process doesn't end with "Traefik static configuration not available" this error only means that traefik needs more time to load its static config (traefik.toml in this case) and this is called here.

We allow up to 30s by default(configurable) for traefik to load its static / dynamic config. During this time, we are pinging traefik to see if it's ready.

Traefik was started by the hub in this case, right?

[jtpio]

Yes Traefik was started by the hub in this case (using this config: https://gist.github.com/jtpio/88d98746ed8577403d4e4497370c6950).

Sometimes this message is printed only once and the hub continues as normal. But sometimes this is what happens:

After the 30s, the hub process terminates.

[GeorgianaElena]

I don't seem to be able to reproduce this :( Is the traefik process running when this is happening?

If yes, try setting TraefikTomlProxy.traefik_api_username and TraefikTomlProxy.traefik_api_password and then login with these credentials into traefik's dashboard (on port 8099) when the errors pop. If the dashboard is accessible then it means traefik is working ok and it might be something wrong with the communication between the hub and traefik.

[jtpio]

It appears to be running:

I'll check with an explicit config (username and password). But it looks like they are generated if not specified?

traefik-proxy/jupyterhub_traefik_proxy/proxy.py

Lines 74 to 103 in a968d81

	@default("traefik_api_username")
	def _warn_empty_username(self):
	self.log.warning("Traefik API username was not set.")
	if self.should_start:
	self.log.warning('Defaulting traefik_api_username to "jupyterhub"')
	return "jupyterhub"
	self.log.warning(
	"Please set c.TraefikProxy.traefik_api_username to authenticate with traefik"
	" if the proxy was not started by the Hub."
	)
	return ""
	traefik_api_hashed_password = Unicode()
	check_route_timeout = Integer(
	30,
	config=True,
	help="""Timeout (in seconds) when waiting for traefik to register an updated route.""",
	)
	static_config = Dict()
	def _generate_htpassword(self):
	from passlib.apache import HtpasswdFile
	ht = HtpasswdFile()
	ht.set_password(self.traefik_api_username, self.traefik_api_password)
	self.traefik_api_hashed_password = str(ht.to_string()).split(":")[1][:-3]

[jtpio]

Using this config:

import os

from jupyterhub.auth import DummyAuthenticator
from jupyterhub.spawner import SimpleLocalProcessSpawner
from jupyterhub_traefik_proxy import TraefikTomlProxy

c = get_config()  # noqa

c.JupyterHub.authenticator_class = DummyAuthenticator
c.JupyterHub.spawner_class = SimpleLocalProcessSpawner
c.JupyterHub.proxy_class = TraefikTomlProxy
c.JupyterHub.allow_named_servers = True

c.TraefikTomlProxy.traefik_api_username='hub'
c.TraefikTomlProxy.traefik_api_password='hub'

rules.tomland traefik.toml are generated in the current working directory with the following content:

$ cat traefik.toml
defaultentrypoints = [ "http",]
debug = true
logLevel = "ERROR"

[api]
dashboard = true
entrypoint = "auth_api"

[wss]
protocol = "http"

[file]
filename = "rules.toml"
watch = true

[entryPoints.http]
address = ":8000"

[entryPoints.auth_api]
address = ":8099"

[entryPoints.auth_api.auth.basic]
users = [ "hub:$apr1$hsgg9D/Q$X9EYYofZhk5e7ASFkhjOW1",]

[frontends.frontend__2F]
backend = "backend__2F"
passHostHeader = true

[backends.backend__2F.servers.server1]
url = "http://127.0.0.1:8081"
weight = 1

[frontends.frontend__2F.routes.test]
rule = "PathPrefix:/"
data = "{\"hub\": true}"

It really seems to be random. Most of the time the hub would start, print Failed to connect to 127.0.0.1 port 8099: Connection refused and continue normally.

[GeorgianaElena]

I'll check with an explicit config (username and password). But it looks like they are generated if not specified?

Yup, those are generated but I don't believe you can find out what's the password that got generated. With your own configured username and password, you can login into traefik's dashboard:

Also, what changed in the last gif (the one that shows that traefik proc is running)? It seems the error is 401 and not 599 like in the previous one.

[jtpio]

Thanks @GeorgianaElena for checking!

Yes I get that one too, and in that case the auth works well.

The 599 happens after several restarts (which maybe stops the hub too fast).

[manics]

Does this occur with the current dev version (install from main) which has recently had a big refactoring to work with Traefik v2?

[Erikhksquad]

Ok