Argo workflow triggering through Github - 503 Error

[Matthias64]

Hello all,

We are facing an issue while trying to trigger an Argo Workflow through a Github webhook with Argo Events.
The endpoint is exposed using Ambassador with the configuration :

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: argoworkflows-wf-start
  namespace: argoworkflows
spec:
  host: argoevents.XXXXXX.com
  prefix: /
  service: github-eventsource-svc.argoworkflows:12000
  precedence: 100

Our event flow looks like the following :

The webhook is created successfully on Github and points to the expected location but it's facing the following error :

503 Service Unavailable - upstream connect error or disconnect/reset before headers. reset reason: connection failure

We feel like this is an issue with the event source pod because we are not able to access it on port 12000, event from inside the cluster.

ka describe pod github-eventsource-ctbw8-5b4594b4d5-4lfvm

Name:         github-eventsource-ctbw8-5b4594b4d5-4lfvm
Namespace:    argoworkflows
Priority:     0
Node:         XXXXXX
Start Time:   Wed, 23 Jun 2021 10:45:00 +0200
Labels:       controller=eventsource-controller
              eventsource-name=github
              owner-name=github
              pod-template-hash=5b4594b4d5
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.0.2.196
IPs:
  IP:           10.0.2.196
Controlled By:  ReplicaSet/github-eventsource-ctbw8-5b4594b4d5
Containers:
  main:
    Container ID:   docker://7609659c5d95d92a1312c016bdee273cd93ed85bdd6b9224840e389dd8f8eb25
    Image:          quay.io/argoproj/eventsource:v1.3.1
    Image ID:       docker-pullable://quay.io/argoproj/eventsource@sha256:1cb769971cdc1e1c447c8aaa81e01963170e1cc9f48da25c333fdbd343002ef2
    Port:           7777/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 23 Jun 2021 10:45:01 +0200
    Ready:          True
    Restart Count:  0
    Environment:
...
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m18s  default-scheduler  Successfully assigned argoworkflows/github-eventsource-ctbw8-5b4594b4d5-4lfvm to XXXXX
  Normal  Pulling    8m18s  kubelet            Pulling image "quay.io/argoproj/eventsource:v1.3.1"
  Normal  Pulled     8m17s  kubelet            Successfully pulled image "quay.io/argoproj/eventsource:v1.3.1" in 885.568572ms
  Normal  Created    8m17s  kubelet            Created container main
  Normal  Started    8m17s  kubelet            Started container main


--------------------------

 ka exec -it  webhook-sensor-bprxd-5759bfff4-6brqs -- sh

/ # ping 10.0.2.196
PING 10.0.2.196 (10.0.2.196): 56 data bytes
64 bytes from 10.0.2.196: seq=0 ttl=253 time=0.487 ms
64 bytes from 10.0.2.196: seq=1 ttl=253 time=0.434 ms
^C
--- 10.0.2.196 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.434/0.460/0.487 ms

/ # curl -k 10.0.2.196:12000
curl: (7) Failed to connect to 10.0.2.196 port 12000: Connection refused

/ # curl -k 10.0.2.196:7777/metrics
# HELP argo_events_event_service_running_total How many configured events in the EventSource object are actively running. https://argoproj.github.io/argo-events/metrics/#argo_events_event_service_running_total
# TYPE argo_events_event_service_running_total gauge
argo_events_event_service_running_total{eventsource_name="github",namespace="argoworkflows"} 1

/ # nslookup github-eventsource-svc
Server:		172.20.0.10
Address:	172.20.0.10:53
Name:	github-eventsource-svc.argoworkflows.svc.cluster.local
Address: 172.20.205.125

/ # curl -k 172.20.0.125:12000
curl: (28) Failed to connect to 172.20.0.125 port 12000: Operation timed out

We must be missing something quite simple but after playing with it for several days we do not succeed to fix it.
Would you have any idea?

We have tried to modify the ports, to create an Ingress to the service and so on but nothing solved our issue.
We also thought it maybe a namespace issue as we had argoevents and argoworkflows in different namespaces but we now have all in the same.

Thank you for your help !!

Full configuration

event_bus.yaml

apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
spec:
  nats:
    native:
      replicas: 3
      auth: token

event_source.yaml

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: github
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  github:
    argo-workflow-creation:
      owner: XXXXX-git
      repository: XXXXX-backend
      webhook:
        # Endpoint listening to events
        endpoint: /
        port: "443"
        method: GET
        url: https://argoevents.XXXXX.com
      events:
        - "*"
      apiToken:
        name: github-access
        key: token
      insecure: true
      active: true
      contentType: json

sensor.yaml

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: webhook
spec:
  template:
    serviceAccountName: argoevents-account # Argo events account created through Terraform
  dependencies:
  - name: github # metadata.name in event-source.yaml
    eventSourceName: github
    eventName: argo-workflow-creation # spec.github.X in event-source.yaml
  triggers:
  - template:
      name: trigger
      argoWorkflow:
        namespace: argoworkflows
        group: argoproj.io
        version: v1alpha1
        resource: workflows
        operation: submit
        source:
          resource:
            apiVersion: argoproj.io/v1alpha1
            kind: Workflow
            metadata:
              namespace: argoworkflows
              generateName: build-workflow-    # Workflow Name
            spec:
              .............
        parameters:
        - src:
            dependencyName: github
            dataKey: body.repository.git_url
          dest: spec.arguments.parameters.0.value
        - src:
            dependencyName: github
            dataKey: body.ref
          dest: spec.arguments.parameters.1.value

[whynowy]

Change 443 below to 12000.

      webhook:
        # Endpoint listening to events
        endpoint: /
        port: "443"

[Matthias64]

Thanks for your reply but that results in the same :

[Matthias64]

Well you were right actually, the rerun worked. Thank you very much !! I can't believe it was actually such a simple fix