Cannot get clusters to talk to each other across the public IP network setup

[malcolmtye-optiva]

Hi, hopefully this is the correct place to ask for help.

I have 2 Openshift 4.10.57 clusters setup.
Each worker node has a second network interface which is designated as the public_ip
I can netcat on ports 500,4500,4900 between the workers on the 2 different clusters

When the connection gets made, it all looks good :
2023-11-20T13:27:48.245Z INF ../datastoresyncer.go:320 DSSyncer Creating local submariner Cluster: types.SubmarinerCluster{ID:"mtyeaiotelcs01", Spec:v1.ClusterSpec{ClusterID:"mtyeaiotelcs01", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"172.30.0.0/16"}, ClusterCIDR:[]string{"10.128.0.0/16"}, GlobalCIDR:[]string{}}} 2023-11-20T13:27:48.262Z INF ../datastoresyncer.go:333 DSSyncer Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"mtyeaiotelcs01", CableName:"submariner-cable-mtyeaiotelcs01-10-164-218-12", HealthCheckIP:"10.128.14.1", Hostname:"wat-rtcs-ocpwrk02p-p.rtcs-prod.otg.om", Subnets:[]string{"172.30.0.0/16", "10.128.0.0/16"}, PrivateIP:"10.164.218.12", PublicIP:"10.164.220.12", NATEnabled:false, Backend:"libreswan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "public-ip":"ipv4:10.164.220.12", "udp-port":"4500"}}} 2023-11-20T13:27:48.274Z DBG ..ery/request_send.go:115 NAT Sending request - REQUEST_NUMBER: 0xe010312019cd3b02, SENDER: "submariner-cable-mtyeaiotelcs01-10-164-218-12", RECEIVER: "submariner-cable-mtyeaiotelcs02-10-72-218-12", USING_SRC: 10.164.218.12:4490, USING_DST: 10.72.218.12:4490 2023-11-20T13:27:48.274Z DBG ..ery/request_send.go:115 NAT Sending request - REQUEST_NUMBER: 0xe010312019cd3b03, SENDER: "submariner-cable-mtyeaiotelcs01-10-164-218-12", RECEIVER: "submariner-cable-mtyeaiotelcs02-10-72-218-12", USING_SRC: 10.164.220.12:4490, USING_DST: 10.72.220.12:4490 2023-11-20T13:27:48.275Z DBG ..y/response_handle.go:31 NAT Received response from 10.72.220.12:4490 - REQUEST_NUMBER: 0xe010312019cd3b03, RESPONSE: NAT_DETECTED, SENDER: "submariner-cable-mtyeaiotelcs02-10-72-218-12", RECEIVER: "submariner-cable-mtyeaiotelcs01-10-164-218-12" 2023-11-20T13:27:48.275Z DBG ../remote_endpoint.go:164 NAT selected public IP "10.72.220.12" for endpoint "submariner-cable-mtyeaiotelcs02-10-72-218-12" 2023-11-20T13:27:48.275Z INF ..gine/cableengine.go:195 CableEngine Installing Endpoint cable "submariner-cable-mtyeaiotelcs02-10-72-218-12"

So, it looks like it's selected the public_ip for the endpoint as the 10.72.220.12 address (which is correct)

But it then fails to create a connection :
2023-11-20T13:27:48.276Z INF ..reswan/libreswan.go:359 libreswan Creating connection(s) for {"metadata":{"name":"mtyeaiotelcs02-submariner-cable-mtyeaiotelcs02-10-72-218-12","namespace":"submariner-operator","uid":"d085e1f4-8d53-46c0-997e-b22d6bd5df04","resourceVersion":"944682158","generation":1,"creationTimestamp":"2023-11-20T04:26:24Z","labels":{"submariner-io/clusterID":"mtyeaiotelcs02"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2023-11-20T04:26:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:public-ip":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"mtyeaiotelcs02","cable_name":"submariner-cable-mtyeaiotelcs02-10-72-218-12","healthCheckIP":"10.252.4.1","hostname":"tcc-rtcs-ocpwrk02r-p.rtcs-dr.otg.om","subnets":["172.31.0.0/16","10.252.0.0/14"],"private_ip":"10.72.218.12","public_ip":"10.72.220.12","nat_enabled":false,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","public-ip":"ipv4:10.72.220.12","udp-port":"4500"}}} in bi-directional mode 2023-11-20T13:27:48.277Z INF ..reswan/libreswan.go:422 libreswan Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0 --id 10.164.218.12 --host 10.164.218.12 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.72.218.12 --host 10.72.220.12 --client 172.31.0.0/16 --ikeport 4500 --dpdaction=hold] 002 "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0": added IKEv2 connection 2023-11-20T13:27:48.294Z INF ../datastoresyncer.go:105 DSSyncer Datastore syncer started 181 "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0" #1: initiating IKEv2 connection 2023-11-20T13:27:48.304Z INF ..reswan/libreswan.go:422 libreswan Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-mtyeaiotelcs02-10-72-218-12-0-1 --id 10.164.218.12 --host 10.164.218.12 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.72.218.12 --host 10.72.220.12 --client 10.252.0.0/14 --ikeport 4500 --dpdaction=hold] 002 "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-1": added IKEv2 connection 2023-11-20T13:27:48.330Z INF ..reswan/libreswan.go:422 libreswan Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-mtyeaiotelcs02-10-72-218-12-1-0 --id 10.164.218.12 --host 10.164.218.12 --client 10.128.0.0/16 --ikeport 4500 --to --id 10.72.218.12 --host 10.72.220.12 --client 172.31.0.0/16 --ikeport 4500 --dpdaction=hold] 002 "submariner-cable-mtyeaiotelcs02-10-72-218-12-1-0": added IKEv2 connection 2023-11-20T13:27:48.333Z INF ..reswan/libreswan.go:422 libreswan Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-mtyeaiotelcs02-10-72-218-12-1-1 --id 10.164.218.12 --host 10.164.218.12 --client 10.128.0.0/16 --ikeport 4500 --to --id 10.72.218.12 --host 10.72.220.12 --client 10.252.0.0/14 --ikeport 4500 --dpdaction=hold] 002 "submariner-cable-mtyeaiotelcs02-10-72-218-12-1-1": added IKEv2 connection 2023-11-20T13:27:48.335Z INF ..gine/cableengine.go:202 CableEngine Successfully installed Endpoint cable "submariner-cable-mtyeaiotelcs02-10-72-218-12" with remote IP 10.72.220.12 2023-11-20T13:27:50.304Z DBG ..reswan/libreswan.go:257 libreswan Connection "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0" not found in active connections obtained from whack: map[], map[] 2023-11-20T13:27:50.304Z DBG ..reswan/libreswan.go:257 libreswan Connection "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-1" not found in active connections obtained from whack: map[], map[] 2023-11-20T13:27:50.304Z DBG ..reswan/libreswan.go:257 libreswan Connection "submariner-cable-mtyeaiotelcs02-10-72-218-12-1-0" not found in active connections obtained from whack: map[], map[] 2023-11-20T13:27:50.304Z DBG ..reswan/libreswan.go:257 libreswan Connection "submariner-cable-mtyeaiotelcs02-10-72-218-12-1-1" not found in active connections obtained from whack: map[], map[] 2023-11-20T13:27:50.304Z DBG ..reswan/libreswan.go:271 libreswan Connection "submariner-cable-mtyeaiotelcs02-10-72-218-12" not found in active connections obtained from whack: map[], map[]

EDIT :
After looking further, the problem I have is that the test that is successful when the gateway first starts uses the SRC IP address as the public_ip which is what I want :
2023-11-20T13:27:48.275Z DBG ..y/response_handle.go:31 NAT Received response from 10.72.220.12:4490 - REQUEST_NUMBER: 0xe010312019cd3b03, RESPONSE: NAT_DETECTED, SENDER: "submariner-cable-mtyeaiotelcs02-10-72-218-12", RECEIVER: "submariner-cable-mtyeaiotelcs01-10-164-218-12"

But when it creates the connection, it uses the 10.164.218.12 private_ip as the source address
2023-11-20T13:27:48.277Z INF ..reswan/libreswan.go:422 libreswan Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0 --id 10.164.218.12 --host 10.164.218.12 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.72.218.12 --host 10.72.220.12 --client 172.31.0.0/16 --ikeport 4500 --dpdaction=hold] 002 "submariner-cable-mtyeaiotelcs02-10-72-218-12-0-0": added IKEv2 connection

Thanks

Malc