mac.eight was building jobs unusually slow compared to previous days.
For example what usually took 3-4 min (https://build.osrfoundation.org/job/gz_physics8-install_bottle-homebrew-arm64/325/) it started taking 30min (https://build.osrfoundation.org/job/gz_physics8-install_bottle-homebrew-arm64/332/)
Fix
The machine was restarted to fix it.
Cause analysis
-
Disk I/O Bottleneck (iostat)
-
Data: The SSD sustained transfer rates between 940 MB/s and 1,020 MB/s, executing over 15,000 operations per second.
-
CPU Impact: CPU user processing time dropped to 5%–8%, while idle time increased to 65%–70%. This indicates the CPU was largely inactive, waiting for the SSD to process queued data.
-
Virtual Memory Swap (vm_stat)
-
Data: The system executed up to 30,000 swap-out operations per second.
-
Analysis: Based on the system's 16 KB page size, 30,000 pages equates to approximately 480 MB of data written to the SSD per second (with a proportional amount read back in). This accounts for the high disk traffic observed in the iostat logs and confirms that the SSD was primarily managing memory overflow.
-
Process Analysis (ps and top)
-
Data: Static process snapshots revealed that baseline system daemons (e.g., Wi-Fi services) were the highest standard memory consumers, while compiler processes were absent.
-
Analysis: Compiler tasks are transient; they utilize significant memory allocations briefly before terminating. Snapshots taken outside of peak compilation windows capture the locked kernel memory managing the swap files, rather than the temporary compiler tasks themselves.
Conclusion
Memory exhaustion during build lead to severe thrashing on the system that left the host in an unstable state. It's unclear if this is simply due to age of the machine or a systematic xcode bug where due to the increase speed of Silicon based machines lead to memory exhaustion or a combination of both. Further investigation when the agent is building is needed to understand the behavior.
Limiting concurrent xcode compilation tasks might avoid this problem in the future.
mac.eight was building jobs unusually slow compared to previous days.
For example what usually took 3-4 min (https://build.osrfoundation.org/job/gz_physics8-install_bottle-homebrew-arm64/325/) it started taking 30min (https://build.osrfoundation.org/job/gz_physics8-install_bottle-homebrew-arm64/332/)
Fix
The machine was restarted to fix it.
Cause analysis
Disk I/O Bottleneck (iostat)
Data: The SSD sustained transfer rates between 940 MB/s and 1,020 MB/s, executing over 15,000 operations per second.
CPU Impact: CPU user processing time dropped to 5%–8%, while idle time increased to 65%–70%. This indicates the CPU was largely inactive, waiting for the SSD to process queued data.
Virtual Memory Swap (vm_stat)
Data: The system executed up to 30,000 swap-out operations per second.
Analysis: Based on the system's 16 KB page size, 30,000 pages equates to approximately 480 MB of data written to the SSD per second (with a proportional amount read back in). This accounts for the high disk traffic observed in the iostat logs and confirms that the SSD was primarily managing memory overflow.
Process Analysis (ps and top)
Data: Static process snapshots revealed that baseline system daemons (e.g., Wi-Fi services) were the highest standard memory consumers, while compiler processes were absent.
Analysis: Compiler tasks are transient; they utilize significant memory allocations briefly before terminating. Snapshots taken outside of peak compilation windows capture the locked kernel memory managing the swap files, rather than the temporary compiler tasks themselves.
Conclusion
Memory exhaustion during build lead to severe thrashing on the system that left the host in an unstable state. It's unclear if this is simply due to age of the machine or a systematic xcode bug where due to the increase speed of Silicon based machines lead to memory exhaustion or a combination of both. Further investigation when the agent is building is needed to understand the behavior.
Limiting concurrent xcode compilation tasks might avoid this problem in the future.