[CI] Set environment variables correctly

[kaimast]

Motivation

I noticed a few issues with the CircleCI setups:

• We didn't set the environment variables properly, so sccache seems to not run at all. Maybe CircleCI's behavior changed and this used to work.
• The windows workflows used different keys to load and restore its cache
• We only regenerated the cache for Linux when they expired.

Proposed Changes

• Set the environment variables for sccache correctly
• Generate a daily cache for all Linux jobs
• Update from sccache 0.3 to 0.10
• Use xlarge for the router tests as they sometimes fail due to lack of memory

Notes

• While we now get cache hits, compilation still takes some time. I have a bigger PR that aims to share build artifacts using CircleCI workspaces, but this smaller one should be merged first.

• It would be good if we could have the cache files expire quicker, maybe after five days or so. There should be a setting in CircleCI, but I might not have permission to change it.

• I did not change the prefix for the cache keys, but it might still be good to remove the version number, as the keys are getting quite long with this change.

• Windows builds seem to generate a new cache entry whenever the Cargo.lock file changes. That might be too infrequent, so opted for daily updates. However, it might make sense to unify the behavior between the two.

[vicsn]

Thx! Should some of this also be moved to snarkVM?

In snarkVM we use the cache to store gigabytes of parameters, so it would be nice to not change daily

[kaimast]

In snarkVM we use the cache to store gigabytes of parameters, so it would be nice to not change daily

It should be possible to have multiple caches. So we can update the sccache one more frequently.

[kaimast]

Rebased this on the Rust version upgrade (#3699) which also has some CI improvements.

[kaimast]

I updated this PR to use Google Cloud Storage as a backend for sccache. There is a single bucket shared among all jobs. We now get a decent number of cache hits and do not have to rely on CircleCI's cumbersome caching mechanism.

These are the stats for building snarkos-node-tcp:

Compile requests                      47
Compile requests executed             31
Cache hits                            29
Cache hits (Rust)                     29
Cache misses                           0
Cache hits rate                   100.00 %
Cache hits rate (Rust)            100.00 %
Cache timeouts                         0
Cache read errors                      0
Forced recaches                        0
Cache write errors                     0
Cache errors                           0
Compilations                           0
Compilation failures                   2
Non-cacheable compilations             0
Non-cacheable calls                   14
Non-compilation calls                  2
Unsupported compiler calls             0
Average cache write                0.000 s
Average compiler                   0.000 s
Average cache read hit             0.391 s
Failed distributed compilations        0

Non-cacheable reasons:
crate-type                            10
-                                      2
missing input                          2

Cache location                  gcs, name: ci_sccache, prefix: /
Version (client)                0.10.0

All jobs, except for node and node-bft now finish in less than eight minutes.

There are still a few downsides to this approach:

• I noticed higher memory usage and had to reduce the number of build jobs for some packages to not run out of memory.
• sccache can also only cache libraries targets, so all executables still need to be built and linked. Most of our code is library crates, though.
• Cache reads are slower now as they have to talk to remote storage.

[vicsn]

How was the environment variable set? If it leaks, is the damage contained? Should we add a mini-manual/doc for how to set the environment variable?

[kaimast]

It's stored in CircleCI similar to how we store the instance types right now. The more complicated thing is to set up the service key with the needed permissions, but I can add documentation for that.