fix: separate golang license caches from mod dir

[kzantow]

Previously, license handling in Golang had a few quirks and may not function if the go mod directory did not exist.

This PR makes the following changes:

• Separate local go mod cache directory from remote download cache
• Use known cache directory using XDG if not specified in config; create the directory if it does not exist
• No longer download files into a go mod directory, only the syft cache

TODO:

• ensure this cache directory is excluded during directory scans (e.g. syft /)
• ascertain if we need a syft cache command to view/manage the cache as part of this PR (answer: no, we are caching significantly less data than syft was previously)
• add more tests
• remove writable fileresolver usage and just directly use fs.FS implementations

Fixes: #2798 #1933

[wagoodman]

This is great! I feel like this is something we can build on over time too.

Since this PR doesn't introduce mechanisms for cache invalidation, users will start to have new cache directories that will continue to grow (this is different than grype, where new DBs get replaced).

I feel like before merging this there should be an answer to how a user would delete the cache or how syft would automatically do this... this could be:

• like you brought up on the call, allow for TTL configuration and track in the cache dir track last write date so we can clean up periodically.
• maybe add a cache list and cache delete subcommands to syft

[wagoodman]

This cache foundation is awesome ❤️ 🚀

[Joerki]

Hi @wagoodman, @kzantow

This is great! I feel like this is something we can build on over time too.

Since this PR doesn't introduce mechanisms for cache invalidation, users will start to have new cache directories that will continue to grow (this is different than grype, where new DBs get replaced).

I feel like before merging this there should be an answer to how a user would delete the cache or how syft would automatically do this... this could be:

• like you brought up on the call, allow for TTL configuration and track in the cache dir track last write date so we can clean up periodically.
• maybe add a cache list and cache delete subcommands to syft

Thanks for the info,
FYI: Good to know. In my case I don't need to do an explicit clean-up step, since my CI pipeline agent throws away all local data after doing its jobs and becomes unused for some minutes.

For other users it might be good to have a choice

• to discard the cache (at the end of execution "I really don't want to have it anymore")
• to keep it after execution (e.g. as default, "I want to use possible cached modules and collect further on")
• or to reset/remove it (at the beginning of execution, "I want to forget cached modules up to now, but want consider cached modules in the future until my next reset")
  during certain build steps.
  I might prefer a command line option or config setting that set one of possible option above.
  With a single command (cache delete) I see a possible, unwanted effect when
• e.g. a pipeline stops because of a failing step
• or a final cleanup step is skipped or gets forgotton

BR,
Jörg

[kzantow]

Hey @Joerki -- we are going to improve the configuration in the near future. The current configuration options allows you to achieve the different options you mentioned. There is a cache directory (SYFT_CACHE_DIR) configuration option (you can always see this now using syft config). So you can give it an explicit dir that gets cleaned up or cached between CI runs or a temporary directory that gets deleted. Or you can just cache/delete the default syft cache directory.

There is also a cache TTL, specifying the time-to-live of cache items, but this does not delete them currently. We plan to add a syft cache purge or similar command to do this explicitly.

And there's an obvious CACHE_ENABLED we could add; I'll get that done right away :) EDIT: I chose not to add this, since the different behavior options can be covered by logical values in the Dir / TTL configuration and we may want to add different types, such as remote cache.

[kzantow]

@Joerki I made another PR to make a couple configuration options work in a more expected manner and be more explicit to support the cases you've mentioned: #2966