Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't cache failed generators #737

Merged
merged 2 commits into from
Mar 3, 2025

Conversation

jfrimmel
Copy link
Contributor

@jfrimmel jfrimmel commented Mar 2, 2025

When FuseSoC runs generators, the output is cached if the generator requests so. While this is generally a good idea, the cached output must not be used in subsequent runs if the generator fails with a non-zero exit code. In that case, the generator should be re-invoked in the next run to not need to user to manually run fusesoc gen clean to fix a faulty cache entry. Therefore this pull request adds in a check, if the generator call was successful and if it isn't the caching is prevented.

The first commit adds a test for a failing generator called in two subsequent FuseSoC-runs, expecting it to be called twice. The second commit changes the code calling generators to check for errors, taking action and re-raising that error. Currently the aforementioned action is to delete the entire working directory of the generator, since this will fail the current check for a cached generator result:

if os.path.lexists(generator_cwd) and not os.path.isdir(generator_cwd):

As an alternative, one could store a file .success (or similar) in the generator directory and only use the cache directory if that file is present there. I'm happy to implement this instead, if deemed better.

jfrimmel added 2 commits March 2, 2025 12:09
When FuseSoC runs generators, the output is cached if the generator re-
quests so. While this is generally a good idea, the cached output must
not be used in subsequent runs if the generator fails with a non-zero
exit code. In that case, the generator should be re-invoked in the next
run to not need to user to manually run `fusesoc gen clean` to fix a
faulty cache entry.
This commit adds a test for that exact behavior, which shows, that the
expectation outlined above is not upheld. Instead a failed generator run
is still cached and thus never re-executed until its input changes.
If the generator invocation failed for any reason, its output directory
is removed. While this is bad for debugging failing generators, it at
least prevents the next FuseSoC-run to assume, that the generator was
successful and hence a failed generator is retried on the next FuseSoC
run. Since the stderr of the executed command is printed to the user,
the harm for debugging is not that large, so this should be fine.

As an alternative, one could store a file `.success` (or similar) in the
generator directory and only use the cache directory if that file is
present there.
@olofk
Copy link
Owner

olofk commented Mar 3, 2025

Ah, you're right. I didn't think of that. Agree that this implementation makes it harder to debug failed generators, but it seems like the safest option also, so let's start with this and consider other options if we need to.

Maybe we could extend this to just move the failed output dir somewhere else, (e.g. $work_root/failed_generators) but taking it in as is to solve the original problem. Thank you for your contributions!

@olofk olofk merged commit 663b348 into olofk:main Mar 3, 2025
12 checks passed
@jfrimmel jfrimmel deleted the dont-cache-failed-generators branch March 3, 2025 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants