[SOLVED] Error when attaching Sablier middleware to a Traefik router config on docker swarm stack/service.

[evmolpos]

I'm experiansing issue behaviour on Traefik when attaching sablier middleware to a Traefik router configuration as a middleware in a stack/service.

error: invalid middleware "it-tool-sablier-plugin-mid@file" configuration: invalid middleware type or middleware does not exist

Weird is that If the sablier middleware not attached to any traefik router, it is recognized normally by traefik as a middleware.

Has anyone experience the same ?

Environment is:
Docker swarm Server Version: 26.1.0 CE
Traefik 2.11.2 CE

Traefik static config for sablier is:

experimental:
  plugins:
    sablier:
      moduleName: "github.com/acouvreur/sablier"
      version: "v1.7.0-beta.7"

Traefik dynamic config (file provider) for sablier middleware of it-tools specific service is:

http:
  middlewares:
    it-tool-sablier-plugin-mid:
      plugin:
        sablier:
          sablierUrl: http://sablier:10000  # The sablier URL service, must be reachable from the Traefik instance
          names: it-tools_it-tools               # name of service. **Does it needs to be traefik service it-tools-svc ?** 
          sessionDuration: 5m               # The session duration after which containers/services/deployments instances are shutdown
          group: default
          # You can only use one strategy at a time
          # To do so, only declare `dynamic` or `blocking`

          # Dynamic strategy, provides the waiting webui
          dynamic:
            displayName: HomeLab       # (Optional) Defaults to the middleware name
            showDetails: true           
            theme: hacker-terminal      # (Optional) The theme to use
            refreshFrequency: 5s        # (Optional) The loading page refresh frequency

          # Blocking strategy, waits until services are up and running
          # but will not wait more than `timeout`
          # blocking: 
          #   timeout: 1m

At this point Traefik accepts and recognizes above middleware configuration, as it is not attached to any traefik router yet.

The error behavior comes when I attach the "it-tool-sablier-plugin-mid@file" as middleware in a traefik router of a traefik service.
error: invalid middleware "it-tool-sablier-plugin-mid@file" configuration: invalid middleware type or middleware does not exist.

Normally it shouldn't. Middleware really exists and recognized by traefik before attached to it-tools service traefik router.

I use the following docker compose definition for it-tools service/stack:

version: "3.9"
services:
  it-tools:
    container_name: it-tools
    image: 'corentinth/it-tools:latest'
    networks:
      - traefik_traefik-net
    .
    .
    deploy:
      mode: replicated # or global:exactly one container per swarm node.
      replicas: 2 # 1 container instance replica for the service in swarm.
      endpoint_mode: vip 
      placement:
        max_replicas_per_node: 1
      labels:
        - traefik.enable=true   
        - traefik.docker.network=traefik_traefik-net
        - traefik.docker.lbswarm=true
        ## Sablier
        - sablier.enable=true
        - sablier.group=default
        ## ROUTERS
        - traefik.http.routers.it-tools-secure-external-rtr.entrypoints=websecure-external
        - traefik.http.routers.it-tools-secure-external-rtr.rule=Host(`**************`)
        - traefik.http.routers.it-tools-secure-external-rtr.tls=true
        - traefik.http.routers.it-tools-secure-external-rtr.tls.certresolver=production-dns-challenge
        - traefik.http.routers.it-tools-secure-rtr.entrypoints=websecure
        - traefik.http.routers.it-tools-secure-rtr.rule=Host(`**************`)
        - traefik.http.routers.it-tools-secure-external-rtr.tls=true
        .
        .
       ## MIDDLEWARES
       .
       .
        - traefik.http.routers.it-tools-secure-external-rtr.middlewares=it-tool-sablier-plugin-mid@file
        ## SERVICE
        - traefik.http.routers.it-tools-secure-external-rtr.service=it-tools-svc
        - traefik.http.routers.it-tools-secure.service=it-tools-svc
        - traefik.http.services.it-tools-svc.loadbalancer.server.port=80
        .
        .
        .

networks:
  traefik_traefik-net:
    external: true

And finally, Sablier API service it self:

version: "3.9"
services:
  sablier:
    image: ghcr.io/acouvreur/sablier:beta # ghcr.io/acouvreur/sablier:beta # acouvreur/sablier:beta #main # beta
    command:
      - start
    container_name: sablier
    security_opt:
      - no-new-privileges:true
    stdin_open: true # docker run -i
    tty: true # docker run -t
    networks:
      - traefik_traefik-net
    environment:
      - TZ=Europe/Athens
      - PROVIDER_NAME=swarm
      - SERVER_PORT=10000
      # The base path for the API
      - SERVER_BASE_PATH=/
      # File path to save the state (default stateless)
      - SERVER_STORAGE_FILE=stateless
      # The default session duration (default 5m)
      - SESSIONS_DEFAULT_DURATION=5m
      # The expiration checking interval. 
      # Higher duration gives less stress on CPU. 
      # If you only use sessions of 1h, setting this to 5m is a good trade-off.
      - EXPIRATION_INTERVAL=20s
      - LOGGING_LEVEL=trace
   
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
    restart: unless-stopped
    deploy:
      # the following deploy configs applies only in swarm mode.
      mode: replicated # or global:exactly one container per swarm node.
      replicas: 2 # 1 container instance replica for the service in swarm.
      endpoint_mode: vip 
      placement:
        max_replicas_per_node: 1
.
.
.

networks:
  traefik_traefik-net:
    external: true

Both Sablier and It-tools services are attached to traefik_traefik-net and capable to resolve / ping each other including traefik.

F|or theshake of testing also created a docker provider version of sablier middleware "it-tool-sablier1-plugin-mid@docker" using tarefik labels in docker compose ...

version: "3.9"
services:
  it-tools:
    container_name: it-tools
    image: 'corentinth/it-tools:latest'
    networks:
      - traefik_traefik-net
    .
    .
    deploy:
      mode: replicated # or global:exactly one container per swarm node.
      replicas: 2 # 1 container instance replica for the service in swarm.
      endpoint_mode: vip 
      placement:
        max_replicas_per_node: 1
      labels:
        - traefik.enable=true   
        - traefik.docker.network=traefik_traefik-net
        - traefik.docker.lbswarm=true
        ## Sablier
        - sablier.enable=true
        - sablier.group=default
        ## ROUTERS
        - traefik.http.routers.it-tools-secure-external-rtr.entrypoints=websecure-external
        - traefik.http.routers.it-tools-secure-external-rtr.rule=Host(`**************`)
        - traefik.http.routers.it-tools-secure-external-rtr.tls=true
        - traefik.http.routers.it-tools-secure-external-rtr.tls.certresolver=production-dns-challenge
        - traefik.http.routers.it-tools-secure-rtr.entrypoints=websecure
        - traefik.http.routers.it-tools-secure-rtr.rule=Host(`**************`)
        - traefik.http.routers.it-tools-secure-external-rtr.tls=true
        .
        .
       ## MIDDLEWARES

        - traefik.http.routers.it-tools-secure-external-rtr.middlewares=it-tool-sablier1-plugin-mid@docker
        .
        .
        
       # Sablier Dynamic Middleware
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.sablierUrl=http://sablier:10000
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.names=it-tools_it-tools
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.sessionDuration=5m
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.group=default        
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.dynamic.displayName=HomeLab
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.dynamic.showDetails=true        
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.dynamic.theme=hacker-terminal
        - traefik.http.middlewares.it-tool-sablier1-plugin-mid.plugin.sablier.dynamic.refreshFrequency=5s
       
        ## SERVICE
        - traefik.http.routers.it-tools-secure-external-rtr.service=it-tools-svc
        - traefik.http.routers.it-tools-secure.service=it-tools-svc
        - traefik.http.services.it-tools-svc.loadbalancer.server.port=80
        .
        .
        .

networks:
  traefik_traefik-net:
    external: true

... but also behaves with the same error when attaching to the traefik router. Really wander is there any sablier middleware configuration error by my side ?

[evmolpos]

if Traefik.log report back the following error, it's because Traefik has no permissions to create the directory plugins-storage.

time="2022-11-03T16:09:42+01:00" level=error msg="Plugins are disabled because an error has occurred." error="mkdir plugins-storage: permission denied"

Explanation:
Traefik tries to create directory /plugins-storage on the filesystem root.

If you run Traefik container using non root user for security reasons, then Traefik can not create the directory . permission-denied.

Solution:

Inspired by
Can’t use plugins - error=“mkdir plugins-storage: permission denied”

Steps:

1. Bind mount the plugins-storage directory to the root (/) of the traefik container. You need it to survive on every container restart, and finally restart traefik service.
   e.g. in my case added the following:

services:
  traefik:
  .
  .
  .
    volumes:
      - /<local path>/plugins-storage:/plugins-storage

2. After restart verify sablier plugin successfully loaded on traefik. You should find sablier plugin content data inside /plugins-storage folder.