How to categorize this issue?

/area networking
/area performance
/kind enhancement

What would you like to be added:
I would like to enable BPF masquerading for scenarios where direct routing (tunnel: disabled) is used.

Currently, BPF masquerading is disabled by default in this configuration, which forces the use of IPTables for masquerading and disables BPF host routing.

The request is to modify the configuration so that BPF masquerading can be used by default, unless SNAT masquerading (snatToUpstreamDNS & snatToUpstreamDNS) is enabled which creates IPTables rules in cilium init containers.

Why is this needed:
Enabling BPF masquerading allows the use of BPF host routing, which offers performance benefits and better leverages the advanced networking capabilities of BPF. The current behavior of disabling BPF masquerading requires using the legacy IPTables host routing mode.

Logs from cilium when enable-bpf-masquerade is missing in cilium-config

time="2024-06-20T08:35:33Z" level=info msg="Direct routing device detected" direct-routing-device=eth0 subsys=linux-datapath
time="2024-06-20T08:35:33Z" level=info msg="BPF host routing requires enable-bpf-masquerade. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon