Phased updates can cause build regressions on Ubuntu Jammy

[Blast545]

Bug Report Track

Phased updates are available on apt since Ubuntu 21.04. This has caused build regressions events on our buildfarms at ci.ros2.org

Phased updates knowledge base:
https://wiki.ubuntu.com/PhasedUpdates

Events

June 7th, first time appearance,

Linux CI was totally broken for that day
Internal discussion: https://osrf.slack.com/archives/C5UEHBHD2/p1654626755337259
https://people.canonical.com/~ubuntu-archive/phased-updates.html
Fix for the problem: #667

June 21st, second time appearance,

Linux CI was partially affected, the problem "fixed itself" and there was no CI downtime because the nodes were using the ccache.
This wasn't expected to occur because with the previously attached PR we opted out of phased updates.
https://osrf.slack.com/archives/C5UEHBHD2/p1655789723051689

[Blast545]

See: ros-infrastructure/ros_buildfarm#969

And discussion on event happening: https://osrf.slack.com/archives/C5UEHBHD2/p1659110534293569

[claraberendsen]

Last week we had an issue related to this. The issue is not related necessarily with phased updates but rather with how phased updates work vs our current workflow with docker.
Phased updates where introduced in Ubuntu 21 and have form part of LTS Jammy. The way they work is explained here.
The issue that arose affected most builds of aarch64 agents and was related to libosmesa6 having a mismatch with their installed dependencies.
This sprung the development of two fixes #708 which after the review phase was deemed that the change did not fix the issue even though empirical evidence showed something different and #709 that changed the logic to always prefer phased updates and update machine-id to ensure all of our containers belong to the same phase (thus preventing the original issue that made us opt-out of phased updates) .