Make Original Client IP Available to Ingress-Nginx

[artntek]

Goal

We want ingress-nginx to have access to the original client IP address for each request, so we can audit these in the access logs, and use them for IP filtering (e.g. adc /wp-admin whitelist), etc.

Current State

I could not make IP filtering work with ingress-nginx, because nginx sees internal IP addresses (from 192.168.0.0/16) as the "source" of each incoming request - example from nginx logs:

# in prod-k8s context:
$ kubectl logs -n ingress-nginx -f $(kc get pods -n ingress-nginx | grep "ingress-nginx" | awk '{print $1}')
[...]
192.168.86.128 - - [24/May/2025:15:47:25 +0000] "GET /metacat/d1/mn/v2/object?fromDate=2025-03-27T21:02:11.053%2B00:00&toDate=2025-05-24T15:47:25.025%2B00:00&replicaStatus=0&start=0&count=1000 HTTP/1.1" 200 948 "-" "Apache-HttpClient/4.5.14 (Java/17.0.7)" 1061 0.367 [opc-metacatopc-hl-8080] [] 192.168.175.215:8080 948 0.367 200 940b41a67e62553d86f77ebed279bda4
192.168.23.192 - - [24/May/2025:15:48:00 +0000] "GET /metacat/d1/mn/v2/object?count=1 HTTP/2.0" 200 563 "-" "checkmk-active-httpv2/2.3.0/chrome" 71 0.244 [opc-metacatopc-hl-8080] [] 192.168.175.215:8080 563 0.243 200 c51d07feadcf7493c83d6acbc6acd683
192.168.23.192 - - [24/May/2025:15:48:06 +0000] "GET / HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" 320 0.000 [goa-mcuigoa-metacatui-80] [] - - - - 7932d75a1962bd1971c878b727f85f1a
192.168.23.192 - - [24/May/2025:15:48:07 +0000] "" 400 0 "-" "-" 0 0.241 [] [] - - - - 181a722f7716cf946ec333633379d8a4
192.168.23.192 - - [24/May/2025:15:48:16 +0000] "GET / HTTP/1.1" 404 146 "-" "check_http/v2.4.0 (monitoring-plugins 2.4.0)" 118 0.001 [upstream-default-backend] [] 127.0.0.1:8181 146 0.001 404 14a322dfaa7273eed1b88b66fca8856a
[...]

The documentation says:

By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address

...but I couldn't get this to work. Tried all the following annotations (see k8s-cluster-config/[prod_cluster/metacatarctic/wordpress/values-prod-cluster-wordpress-arctic.yaml)

nginx.ingress.kubernetes.io/enable-real-ip: "true"
nginx.ingress.kubernetes.io/proxy-real-ip-cidr: "0.0.0.0/0"
nginx.ingress.kubernetes.io/set_real_ip_from: "0.0.0.0/0"
# also tried: "128.111.85.128/25" and "192.168.0.0/16"
nginx.ingress.kubernetes.io/real_ip_header: "X-Forwarded-For"
# also tried: "X-Real-IP" and "CF-Connecting-IP"
nginx.ingress.kubernetes.io/real_ip_recursive: "on"
nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
nginx.ingress.kubernetes.io/use-proxy-protocol: "false"
nginx.ingress.kubernetes.io/whitelist-source-range: |
  128.111.61.0/24,
  128.111.64.0/22,
  128.111.85.0/24,
  128.111.180.0/22,
  128.111.188.0/22,
  128.111.196.0/23,
  207.71.230.208/29
# Only way I could get filtering to "work" is by adding the internal IPs:
#  , 192.168.0.0/16

Next steps

• Talk to @mbjones about current setup - what is upstream?
  • Can we configure it to set one of the headers X-Forwarded-For, X-Real-IP. or CF-Connecting-IP?
  • Alternatively, can we config it to work with use-proxy-protocol?
• Try installing ngrep on a container that's behind the ingress to inspect incoming headers (ngrep -d any -W byline port 8080)
• Also see:
• How do I enable PROXY protocol when my load balancer sends requests to the NGINX Ingress Controller?
  
• Is there a way to enable proxy-protocol on Ingress for only one service?

[artntek]

according to this SE thread, we need to set:

externalTrafficPolicy: Local

docs: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

.spec.externalTrafficPolicy - denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type Services, but risks potentially imbalanced traffic spreading.

Caution

With externalTrafficPolicy set to Local, traffic will only be served if the ingress-nginx pod is running on the same node as the public IP receiving the traffic (e.g. on dev cluster: docker-dev-ucsb-1).

In our case: ingress-nginx is on k8s-dev-node-3, NOT docker-dev-ucsb-1, so traffic will stop working unless we move the ingress:

nodeSelector:
  kubernetes.io/hostname: docker-dev-ucsb-1

...or better, convert the controller to a DaemonSet, which runs one pod on every node automatically.

[mbjones]

...or better, convert the controller to a DaemonSet, which runs one pod on every node automatically

This still wouldn't guarantee that you'd use the ingress on the same node as the pod -- any ingress can serve any pod in the cluster.

[mbjones]

This is a useful page for understanding source IP config: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer

[artntek]

Thanks for the page you posted above - it's really helpful. Following up on the comment that a DaemonSet...

...wouldn't guarantee that you'd use the ingress on the same node as the pod -- any ingress can serve any pod in the cluster.

...that page indicates this may not be correct:

"If you set service.spec.externalTrafficPolicy to the value Local, kube-proxy only proxies proxy requests to local endpoints, and does not forward traffic to other nodes. If there are no local endpoints, packets sent to the node are dropped, so you can rely on the correct source-ip in any packet processing rules you might apply a packet that make it through to the endpoint."

However, it's not clear to me how this would work in practice, for our clusters. I think we'd need to have a component upstream, to direct incoming requests to the correct nodes, or to all nodes (otherwise a request might just be dropped, if the target node doesn't have a local endpoint)?

[artntek]

Problem: This Client IP Issue May Cause Metric Service to Under-Count Concurrent Views and Downloads

The metrics service does a couple of things, mainly related to IP:

1. The IP filtering by logstash parses a dataone_IP.txt file (last updated in 2020), and adds a tag dataone_ip to each log entry if there is a hit with our system IPs. The metrics API automatically discards all events tagged with dataone_ip while aggregating the query metrics.

2. Sessions calculation - Based on Counter Code of Practice (Counter COP) developed during the Make Data Count project (back in 2018), if we have more than one visit for a dataset/object by the same IP address within a certain timeframe (45 minutes to an hour??), it falls under the same session. So, if we're not tracking "real" client IPs, then it would also change how metrics are calculated and displayed on the UI.

   • for example: if two different IPs (a.a.a.a and b.b.b.b) are viewing/downloading same dataset within this session time frame, they will be assigned same session ID and counted as a single download/view event, because both events will have the same K8s cluster IP instead of their real ones.