rsp2-python uses up to 8.1 GB (!!) of RAM when being built

[colin-daniels]

Yeah not sure what's going on here, rustc seems to be using a ton of memory compiling rsp2_python. Incidentally, clion likes to auto-compile things when you open it and does both release + debug, and since I only have 16 GB ram it doesn't end well.

Edit: this is the specific command for the plot above

/home/cc/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/rustc \
    --crate-name rsp2_python src/python/lib.rs \
    --crate-type lib \
    --emit=dep-info,link \
    -C debuginfo=2 \
    -C metadata=264cf8b5352636b9 \
    -C extra-filename=-264cf8b5352636b9 \
    --out-dir /run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps \
    -C incremental=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/incremental \
    -L dependency=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps \
    --extern failure=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps/libfailure-eb5f24025fc850d4.rlib \
    --extern include_dir=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps/libinclude_dir-8706cc4b7efba23e.rlib \
    --extern log=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps/liblog-14a287c9015a87f1.rlib \
    --extern rsp2_fs_util=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/deps/librsp2_fs_util-c06cc223e93fb9c6.rlib \
    -L native=/run/media/cc/31b7ba32-4e18-4c56-bcc5-c904490446b5/build/rsp2/target/debug/build/backtrace-sys-6fb5aad937e11ada/out

[ExpHP]

Holy cheeseballs!

Do you have a target folder in src/python? (or anything else besides the Cargo.toml, build.rs, and rsp2 directory?) I should probably make my usage of include_dir! a bit more precise to avoid including unnecessary files.

[colin-daniels]

Nothing other than this:

cc@engine ~/build/rsp2/src/python (git)-[mpi] % tree -a .
.
├── build.rs
├── Cargo.toml
├── lib.rs
├── rsp2
│   ├── cli
│   │   ├── __init__.py
│   │   ├── negative_modes.py
│   │   └── __pycache__
│   │       ├── __init__.cpython-37.pyc
│   │       └── negative_modes.cpython-37.pyc
│   ├── __init__.py
│   ├── internals
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-37.pyc
│   │   └── scipy_eigsh
│   │       ├── call.py
│   │       ├── __init__.py
│   │       ├── negative.py
│   │       └── __pycache__
│   │           ├── __init__.cpython-37.pyc
│   │           └── negative.cpython-37.pyc
│   ├── io
│   │   ├── dwim.py
│   │   ├── dynmat.py
│   │   ├── eigensols.py
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── dwim.cpython-37.pyc
│   │   │   ├── dynmat.cpython-37.pyc
│   │   │   ├── eigensols.cpython-37.pyc
│   │   │   └── __init__.cpython-37.pyc
│   │   └── _yaml_shim.py
│   └── __pycache__
│       └── __init__.cpython-37.pyc
└── thing.json

10 directories, 26 files

I killed my target directory in the main rsp2 folder, still happens. I'm pretty sure it does this on my work computer too.

[ExpHP]

how large is thing.json?

[colin-daniels]

12MB

Edit: just went ahead and killed it, building again.

[colin-daniels]

...that was actually the issue. Takes 11s to compile now. Ok. Whatever.

[ExpHP]

I haven't looked at how include_dir! works but it clearly isn't very efficient...

There doesn't seem to be an easy solution to this; I'm stuck between patching include_dir to add some knobs, or finding some other way to keep the right version of the python code tied to the binary. I really figured including it at compile time would be the most effective :/

Or having rsp2-python's build script look for stuff that doesn't belong there and complain, but that's just dumb.

[colin-daniels]

Well, apparently as long as you don't accidentally leave any large json files (that it tries to parse or something?) in the source tree it seems to be fine. It's probably fine as it is, unless a simple warning can be added or similar

[ExpHP]

I believe it takes every file in the directory, reads it as binary, and generates binary literals like this. I assumed it used the compiler's own include_bytes! to do this, but searching the source I'm not so sure now. Generating the token stream for that is probably the cause of the memory trouble.

I could easily add a check for files without python extensions, but that's more or less it.

[ExpHP]

Should be addressed in b135a65 (now on the mpi branch head)

[ExpHP]

I submitted a PR to include_dir which reduces maximum memory usage by a factor of at least 10.
It has been merged and, with any luck, will be present in the v0.2.2 release. Michael-F-Bryan/include_dir#32

[ExpHP]

triage: Still no release of the upstream crate. I'll go bug the maintainer.