A loadBalancer service were failing on port 80 only

[edikmkoyan]

Before I switched to metallb(it doesn't solve all my problems), I was trying the kube-vip.
nginx-ingress were getting the hardcoded IP address, and I was able to connect to the exposed IP on port 443 for argocd, however it was randomly entering to failing state for port 80.
I mac address change were correlating with that, I could freely access the service on all the nodePort ports on all hosts, forwarding the ingress were also working fine. ipvsadm -ln were showing no listener changes. I hardly can image what could be the reason. Here is my configuration and the looped curl output.

apiVersion: v2
name: kube-vip
type: application
version: 0.4.4
appVersion: "0.4.1"
dependencies:
- name: kube-vip
  version: 0.4.4
  repository: https://kube-vip.github.io/helm-charts
kube-vip:
  affinity: {}
  config:
    address: "192.168.113.113"
  env:
    cp_enable: "false"
    lb_enable: "false"
    lb_port: "6443"
    svc_enable: "true"
    vip_arp: "true"
    vip_cidr: "192.168.113.0/24"
    vip_interface: ""
    vip_leaderelection: "true"
  envFrom: []
  envValueFrom: {}
  fullnameOverride: ""
  image:
    pullPolicy: IfNotPresent
    repository: ghcr.io/kube-vip/kube-vip
    tag: v0.5.11 #I also tried v0.6.2
  imagePullSecrets: []
  nameOverride: ""
  nodeSelector: {}
  podAnnotations: {}
  podSecurityContext: {}
  resources: {}
  securityContext:
    capabilities:
      add:
      - NET_ADMIN
      - NET_RAW
  serviceAccount:
    annotations: {}
    create: true
    name: ""
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists

curl: (7) Failed to connect to 192.168.113.200 port 80 after 1021 ms: Couldn't connect to server
HTTP/1.1 308 Permanent Redirect
Date: Tue, 05 Sep 2023 10:26:08 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://example.com/argo

HTTP/2 301 
date: Tue, 05 Sep 2023 10:26:09 GMT
content-type: text/html; charset=utf-8
location: /argo/
strict-transport-security: max-age=15724800; includeSubDomains

HTTP/2 200 
date: Tue, 05 Sep 2023 10:26:09 GMT
content-type: text/html; charset=utf-8
content-length: 788
accept-ranges: bytes
content-security-policy: frame-ancestors 'self';
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
strict-transport-security: max-age=15724800; includeSubDomains

curl: (7) Failed to connect to 192.168.113.200 port 80 after 13 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 14 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 7 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1007 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1010 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1015 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1010 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1010 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1016 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1011 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1008 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1011 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1010 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 8 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1008 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1011 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1008 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1012 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1018 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1014 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1008 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1010 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1015 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1009 ms: Couldn't connect to server
curl: (7) Failed to connect to 192.168.113.200 port 80 after 1012 ms: Couldn't connect to server
HTTP/1.1 308 Permanent Redirect
Date: Tue, 05 Sep 2023 10:26:42 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://example.com/argo

HTTP/2 301 
date: Tue, 05 Sep 2023 10:26:42 GMT
content-type: text/html; charset=utf-8
location: /argo/
strict-transport-security: max-age=15724800; includeSubDomains

HTTP/2 200 
date: Tue, 05 Sep 2023 10:26:42 GMT
content-type: text/html; charset=utf-8
content-length: 788
accept-ranges: bytes
content-security-policy: frame-ancestors 'self';
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
strict-transport-security: max-age=15724800; includeSubDomains