[operator] gardenctl ssh not working on openstack

[benedikt-haug]

How to categorize this issue?

/area useability
/kind bug

What happened:

Bastion node creation fails on openstack due to not being unable to reserve a public IP via openstack's neutron:

gardenctl ssh
...
E0228 11:26:40.342618    9616 options.go:1054] "Still waiting" err="Error while waiting for Bastion shoot--m-seed--m-seed-0/cli-zklnlmfz to become ready: error during reconciliation: Error reconciling Bastion: failed to get (create) public ip address: Bad request with: [POST https://api.<snip>.de:9696/v2.0/floatingips], error message: {\"NeutronError\": {\"type\": \"ExternalIpAddressExhausted\", \"message\": \"Unable to find any IP address on external network 68a806ee-4eb8-4b50-ae49-c06bde9baf06.\", \"detail\": \"\"}}"

the same also happens for shoots:
E0228 11:37:46.647309    9780 options.go:1054] "Still waiting" err="Error while waiting for Bastion shoot--dev--single-small/cli-epitut1a to become ready: error during reconciliation: Error reconciling Bastion: failed to get (create) public ip address: Bad request with: [POST https://api.<snip>.de:9696/v2.0/floatingips], error message: {\"NeutronError\": {\"type\": \"ExternalIpAddressExhausted\", \"message\": \"Unable to find any IP address on external network 68a806ee-4eb8-4b50-ae49-c06bde9baf06.\", \"detail\": \"\"}}"

I tested these application credentials, they are valid and allow reserving floating IP's and the pool is not empty, as the same floating_network_id was used:

What you expected to happen:

Being able to ssh to a bastion host.

How to reproduce it (as minimally and precisely as possible):

Try to use ssh on any openstack environment. I think this is a general issue.

Anything else we need to know?:

I know I'm currently opening a lot of issues. This is the least important one, feel free to de prioritize it if that helps <3

Environment:

• Gardener version: v1.113.0
• Kubernetes version (use kubectl version): v1.31.6
• Cloud provider or hardware configuration: gardener-extension-provider-openstack v1.45.1
• Others:

[gardener-prow]

@benedikt-haug: The label(s) area/useability cannot be applied, because the repository doesn't have them.

In response to this:

How to categorize this issue?

/area useability
/kind bug

What happened:

Bastion node creation fails on openstack due to not being unable to reserve a public IP via openstack's neutron:

gardenctl ssh
...
E0228 11:26:40.342618    9616 options.go:1054] "Still waiting" err="Error while waiting for Bastion shoot--m-seed--m-seed-0/cli-zklnlmfz to become ready: error during reconciliation: Error reconciling Bastion: failed to get (create) public ip address: Bad request with: [POST https://api.<snip>.de:9696/v2.0/floatingips], error message: {\"NeutronError\": {\"type\": \"ExternalIpAddressExhausted\", \"message\": \"Unable to find any IP address on external network 68a806ee-4eb8-4b50-ae49-c06bde9baf06.\", \"detail\": \"\"}}"

the same also happens for shoots:
E0228 11:37:46.647309    9780 options.go:1054] "Still waiting" err="Error while waiting for Bastion shoot--dev--single-small/cli-epitut1a to become ready: error during reconciliation: Error reconciling Bastion: failed to get (create) public ip address: Bad request with: [POST https://api.<snip>.de:9696/v2.0/floatingips], error message: {\"NeutronError\": {\"type\": \"ExternalIpAddressExhausted\", \"message\": \"Unable to find any IP address on external network 68a806ee-4eb8-4b50-ae49-c06bde9baf06.\", \"detail\": \"\"}}"

I tested these application credentials, they are valid and allow reserving floating IP's and the pool is not empty, as the same floating_network_id was used:

![Image](https://github.com/user-attachments/assets/fa4b7840-ed5b-4be5-a054-694d0712b5c8

What you expected to happen:

Being able to ssh to a bastion host.

How to reproduce it (as minimally and precisely as possible):

Try to use ssh on any openstack environment. I think this is a general issue.

Anything else we need to know?:

I know I'm currently opening a lot of issues. This is the least important one, feel free to de prioritize it if that helps <3

Environment:

• Gardener version: v1.113.0
• Kubernetes version (use kubectl version): v1.31.6
• Cloud provider or hardware configuration: gardener-extension-provider-openstack v1.45.1
• Others:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[gardener-robot]

@benedikt-haug Label area/useability does not exist.

[kon-angelo]

Hi @benedikt-haug I am not sure we can do much for this. The error is coming directly from neutron that it cannot allocate an IP. The extension cannot do much at this point, unless you suspect an issue with the request.

[benedikt-haug]

The extension cannot do much at this point, unless you suspect an issue with the request.

Fair.

I tried to look around but couldn't find the actual neutron request being used anywhere. Could you help me find it? <3
Because if I create a floatingIP by any other means with the same application credentials (example above), reserving a floatingIP works just fine.

So I'd assume either an issue with the request itself or with how our infrastructure handles that this specific one.

[kon-angelo]

@benedikt-haug

gardener-extension-provider-openstack/pkg/controller/bastion/actuator_reconcile.go

Line 218 in eca510d

createOpts := floatingips.CreateOpts{

the request is created here.

[benedikt-haug]

Thank you! I'll try to beg someone with more go knowledge to further debug this.

Something which instantly intrigued me when I look at this code, is that it just assumes the first subnet to be valid. In our case its an IPv6 network:

Could this be an issue? (I'm asking, because we recently had issue's with IPv6 subnet ordering here: #897)