stable.release.flatcar-linux.net DNS records are missing

[handcraftedbits]

Description

It appears the DNS records for stable.release.flatcar-linux.net are missing according to most major providers. Interestingly, there is an A record known to OpenDNS, but when I attempt to locally resolve stable.release.flatcar-linux.net to that IP address (136.144.58.113), the server returns a 502.

Impact

Can't download Flatcar Linux ISOs and perform installation.

Environment and steps to reproduce

Try to access stable.release.flatcar-linux.net.

Expected behavior

Should have been able to access stable.release.flatcar-linux.net, but instead the hostname doesn't resolve.

Additional information

This was working as of two days ago.

[rgeraskin]

actually, not only 'stable' but all dns zone flatcar-linux.net

[tormath1]

Hi @handcraftedbits and thanks for the report. The team is currently investigating the issue.

[t-lo]

Update:
flatcar-linux.net, flatcar-linux.com and flatcar-linux.de are currently not DNS resolved. This is after a zone transfer to CNCF / LF. Most impactful here is that update.flatcar-linux.net is not reachable

We expect this issue to be transient and we are working with the CNCF to fix the DNS records.

[marcgweg]

Can you give an update? 4 hours for a DNS issue seems a long time to me. Specially considering you just released new releases, which nobody can download right now.

[chewi]

Unfortunately, we have to wait for someone we don't really know to wake up in a totally different time zone. We had to transfer these domains as part of joining the CNCF, and we no longer have direct control over them.

[handcraftedbits]

Are there any host -> IP address mappings you could provide here? So perhaps we can do /etc/hosts mappings in the meantime.

[chewi]

It's a good idea, but I'm afraid it won't help because this has also brought our CDN down. I already tried it. 😢

[tormath1]

Are there any host -> IP address mappings you could provide here? So perhaps we can do /etc/hosts mappings in the meantime.

As a stop gap solution one can use this entries:

147.75.87.17 stable.release.flatcar-linux.net
147.75.87.17 lts.release.flatcar-linux.net
147.75.87.17 beta.release.flatcar-linux.net
147.75.87.17 alpha.release.flatcar-linux.net

It's far from ideal but still better than nothing. Please note that the whole situation is out of our direct control as chewi mentioned and that we're actively working on fixing this.

[tormath1]

Things seems to get back to the normal since a few minutes now.

[lefpiz]

My nvdia driver is still not installing do to download issues after reboot (PXE boot) giving the following error message in journalctl -u nvidia -f:

Feb 03 16:18:05 xwabss.home.arpa systemd[1]: Starting nvidia.service - NVIDIA Configure Service...
Feb 03 16:18:05 xwabss.home.arpa sh[1543]: rm: cannot remove '/run/extensions/nvidia-driver': No such file or directory
Feb 03 16:18:05 xwabss.home.arpa setup-nvidia[1545]: Downloading Flatcar Container Linux Developer Container for version: 4186.1.0
Feb 03 16:19:05 xwabss.home.arpa setup-nvidia[1559]: % Total % Received % Xferd Average Speed Time Time Time Current
Feb 03 16:19:05 xwabss.home.arpa setup-nvidia[1559]: Dload Upload Total Spent Left Speed
Feb 03 16:19:06 xwabss.home.arpa setup-nvidia[1559]: [139B blob data]
Feb 03 16:19:06 xwabss.home.arpa systemd[1]: nvidia.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Feb 03 16:19:06 xwabss.home.arpa rmmod[1560]: rmmod: ERROR: Module nvidia_uvm is not currently loaded
Feb 03 16:19:06 xwabss.home.arpa rmmod[1560]: rmmod: ERROR: Module nvidia_modeset is not currently loaded
Feb 03 16:19:06 xwabss.home.arpa rmmod[1560]: rmmod: ERROR: Module nvidia is not currently loaded
Feb 03 16:19:06 xwabss.home.arpa systemd[1]: nvidia.service: Control process exited, code=exited, status=1/FAILURE
Feb 03 16:19:06 xwabss.home.arpa systemd[1]: nvidia.service: Failed with result 'exit-code'.
Feb 03 16:19:06 xwabss.home.arpa systemd[1]: Failed to start nvidia.service - NVIDIA Configure Service.

Can anyone tell me how to pass a local http server for flatcar_developer_container.bin.bz2 (e.g. http://boot.home.arpa:8080/flatcar_developer_container.bin.bz2) in butane?

[chewi]

It's taking some time for the DNS changes to propagate. I'm still not seeing them here. I need to go now, hopefully someone else can help with your question.

[lefpiz]

Thank you for yout reply. Unfortunately the release server seems to be down again. Does anyone have a hint how to make flatcar use a local source for developer container download during nvidia.service installation?

[t-lo]

@lefpiz I can reach https://beta.release.flatcar-linux.net/amd64-usr/current/ from here; can you test from your node if you can download https://beta.release.flatcar-linux.net/amd64-usr/4186.1.0/flatcar_developer_container.bin.bz2 ?

[lefpiz]

After restart of my opnsense I can now reach the URL and installation of nvidia drivers was working. But I would still be interested if there is an option to set a local server address for the files needed by nvidia.service. I also host the kermel and boot image locally for PXE boot. Thank you for your help!

[chewi]

I've checked the code, and I'm afraid the setup-nvidia script is hardcoded to try CHANNEL.release.flatcar-linux.net and bincache.flatcar-linux.net. The good news is that we are planning to provide prebuilt NVIDIA modules, so this process won't be necessary anymore.

[lefpiz]

Excellent! Thank you for the help. I am looking forward to the new module.

[chewi]

Reopening for more visibility until we post a write-up later.