Weird behaviour of iPXE on Dell r730 server with Qlogic BCM57800s NIC.

[0megam]

Hello.
I'm building iPXE with following flags from master branch:

make -j 8 bin-x86_64-efi/ipxe.efi bin/undionly.kpxe bin-x86_64-efi/snponly.efi EMBED=embed.ipxe LOG_LEVEL=LOG_ALL -C src

And then trying to use ipxe.efi or snponly.efi to boot on Dell r730 in UEFI mode I'm getting following errors:

But in case I'm building iPXE from same source base with these flags:

make -j 8 bin-x86_64-efi/ipxe.efi bin/undionly.kpxe bin-x86_64-efi/snponly.efi EMBED=embed.ipxe DEBUG=efi_driver,snpnet LOG_LEVEL=LOG_ALL -C src

And then using ipxe.efi or snponly.efi to boot the server, it able to get ip from DHCP and execute embed.ipxe script.

embed.ipxe:

embed.ipxe

set user-class Tinkerbell
echo Welcome to Neverland!

prompt --key 0x02 --timeout 2000 Press Ctrl-B for the iPXE command line... && shell ||

set vlan-id ${43.116:string}
isset ${vlan-id} && goto boot-with-vlan ||

:iterate-interfaces
set idx:int32 0
:interfaces-loop
isset ${net${idx}/ip} && goto interfaces-loop-done || iseq ${idx} 50 && goto autoboot || inc idx && goto interfaces-loop

:interfaces-loop-done
echo Booting from net${idx}...
autoboot net${idx}

:autoboot
autoboot

:boot-with-vlan
set idx:int32 0
:loop isset ${net${idx}-${vlan-id}/ip} && goto loop_done ||
iseq ${idx} 50 && goto error ||
inc idx && goto loop

:loop_done
echo Booting from net${idx}-${vlan-id}...
autoboot net${idx}-${vlan-id}

:error
echo Failed to find the vlan interface
shell

Any ideas why with DEBUG=efi_driver,snpnet parameter same code are able to work with this hardware and without it not ?
Thanks.

[NiKiZe]

Most likely cause I can think of is the delay added fixes a race condition... But have you visited the linked page and followed the suggestions there?

What is net3 connected to? And what is the configuration on that end?

You should grab an ifstat for both the working and non working case after the attempted DHCP - enter shell and run autoboot followed by ifstat.

[0megam]

Hi @NiKiZe
Thank you for fast reply.
net3 is connected to a regular access port on Juniper switch. I don't have configuration snippet of the port in my hands, but if its really needed I can obtain it.

With same switch port configuration HPE server booting without any problems with non debug iPXE build.
Here is what I got if I run dhcp and ifstat then autoboot commands on non debug iPXE build:


On DHCP server side I don't see any DHCP request packets for the server.

Here is what I got if I run dhcp and ifstat then autoboot commands on debug iPXE build:

[NiKiZe]

Run ifstat after failed autoboot.

The "no DHCP packets on the server" does suggest that the port might be blocked maybe with STP really really do check the URL in the error message. https://ipxe.org/040ee186

Here it tells you to get the ifstat data, but I would especially check the last paragraph about STP.

We have seen that the iPXE NII driver is used, and you really do seem to have different behaviour due to debug flags, but are these booted in the exacts same way? Are they both booted from cold start? Or is one after a reboot but not the other?

[0megam]

Here is the screenshot with ifstat after autoboot. This is different server but same hardware and network config. Debug binary also works on this server without problems:

I checked https://ipxe.org/040ee186 error, as I mentioned I don't see any DHCPDISCOVER packets that coming from the server.
I can't check right now if STP is enabled on a switch port. I will check tomorrow. But still, if problem is on switch side I expect iPXE not to work no matter with or without debug flags.
Another thing, at first I followed this discussion #248, similar hardware with similar problem, but it's not clear how problem was resolved for the author. I guess he also compiled the binary with debug flag and just left it be that way..

Server are cold booted in both cases(powercycle). Turned off and then turned on.

Another interesting matter. Before we started to use Tinkerbell for server provisioning, we used xCAT. xCAT has their own iPXE code repo: https://github.com/xcat2/xNBA
iPXE binary compiled from xNBA codebase without any debug flags is working on this server model, the only problem it's lack on EFI watchdog functionality and this is no go for us.

[NiKiZe]

If you can find a version of ipxe where this worked, you could then use instructions at https://ipxe.org/howto/bisect for how to find when this might have broken.

You are using the NII driver, which means that the real driver is the one from firmware, what we can see from the ifstat output is that you have TX but no RX.

You need both debugs to get working boot, or can you reduce it more?