Support concurrent access to shared PLATFORMIO_PACKAGES_DIR and PLATFORMIO_PLATFORMS_DIR for CI builds

[nomis]

• Feature Request.
  Start by telling us what problem you’re trying to solve. Often a solution
  already exists! Don’t send pull requests to implement new features without first getting our
  support. Sometimes we leave features out on purpose to keep the project small.

---

Configuration

Operating system: Debian Testing

PlatformIO Version (platformio --version): 6.1.x

Description of problem

To avoid unnecessary HTTP downloads and package extractions on every CI build, I have shared the following directories across builds (which may run concurrently for different projects):

• PLATFORMIO_CACHE_DIR
• PLATFORMIO_PACKAGES_DIR
• PLATFORMIO_PLATFORMS_DIR

The problem with this approach is that PlatformIO uses unversioned directory names for platforms and packages by default, and under some conditions it will only keep one unversioned name of a package (e.g. tool-scons), so concurrent builds will fail if the unversioned directory name is renamed or replaced with a different version while the build is running. This can happen if different projects use different versions of the same dependencies, or one or more projects use a version range and a new version becomes available.

I'd like to be able to configure PlatformIO to:

• Always use versioned directory names so that concurrent builds won't interfere with each other
• Keep old versions of platforms and packages indefinitely (I think check_prune_system_threshold will do some of this, but I observe it immediately deleting tool-scons when changing PlatformIO version)

I'm assuming the HTTP cache is already safe for concurrent access?

I realise that there may not be 100% compatibility between different versions of PlatformIO using the same unpacked files, but the content appears to be mostly read-only (and Python cache files are already versioned and should support concurrent access).

Additional info

The directories are shared in containers as bind mounts on Linux so locking of concurrent modifications using files inside the directories will work as expected.

Projects are never built concurrently with the same project directory, Jenkins will make a copy of the workspace for this, so the PLATFORMIO_WORKSPACE_DIR directory is always used sequentially.

I feel like there's a downside to caching HTTP downloads in that the registry will consider the content "unused" because it's not being downloaded on every build and some kind of opt-in "cache hit" notification may help mitigate this.