Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When the compiler get's killed by OOM/SIGTERM the cache-db.json file is not updated #3995

Open
gorbak25 opened this issue Jan 21, 2021 · 1 comment · May be fixed by #3996
Open

When the compiler get's killed by OOM/SIGTERM the cache-db.json file is not updated #3995

gorbak25 opened this issue Jan 21, 2021 · 1 comment · May be fixed by #3996

Comments

@gorbak25
Copy link

gorbak25 commented Jan 21, 2021

Description

When building large purescript codebases such as https://github.com/erlscripten/erlps-stdlib it's very easy to get killed by OOM - The modules are compiled in parallel usually exceeding the amount of RAM available on the system(I will open separate issues with the results of my investigation/profiling data - for instance the lexer right now uses up a lot of memory on large files). What is really troublesome is that if the compilation gets killed by OOM after compiling a lot of heavy modules then on the next run they will get recompiled although they compiled successfully. After investivating the purescript codebase the cache-db.json file is only updated after all of the compiler tasks either succeed or failed - this is not a problem for small projects but when the compilation times become significant(https://github.com/erlscripten/erlps-stdlib) it's really troublesome :(

To Reproduce

  1. Compile a project with modules M_1, M_2, ..., M_N
  2. The compiler compiles M_1, M_2, ..., M_N-1
  3. While compiling M_N the compiler gets killed either by SIGTERM or OOM
  4. After restarting the build modules M_1, M_2, ..., M_N-1 are being recompiled again although the generated code is already present on the drive

Expected behavior

  1. Compile a project with modules M_1, M_2, ..., M_N
  2. The compiler compiles M_1, M_2, ..., M_N-1
  3. While compiling M_N the compiler gets killed either by SIGTERM or OOM
  4. After restarting the build only M_N gets build

PureScript version

Tried both on 0.13.8 and on the current master branch :(

@gorbak25 gorbak25 changed the title When the compiler get's killed by OOM the cache-db.json file is not updated When the compiler get's killed by OOM/SIGTERM the cache-db.json file is not updated Jan 21, 2021
@hdgarrood
Copy link
Contributor

This sounds sensible to me, but we'll have to be really careful to avoid race conditions if we implement this, since modules are compiled in parallel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants