Skip to content
This repository has been archived by the owner on Mar 2, 2022. It is now read-only.

Excessive memory usage #84

Open
ln5 opened this issue Feb 6, 2018 · 2 comments
Open

Excessive memory usage #84

ln5 opened this issue Feb 6, 2018 · 2 comments

Comments

@ln5
Copy link

ln5 commented Feb 6, 2018

Possibly related to unhandled errors (#70, #81), a scan eats huge amounts of memory. Here's an example:

2018-02-06T00:04:41+0100 [INFO]: Performing a measurement scan with 6216 relays.

10h later, the python process consumes about 5GB of RAM (RSS):

F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0  1000   996   583  20   0 5156300 5035808 SyS_ep Sl+ pts/1    2:30 /usr/bin/python /home/bwscanner/.local/bin/bwscan --launch-tor --loglevel debug scan
@ln5
Copy link
Author

ln5 commented Feb 6, 2018

I'm running branch nn_revert_remove_hashing_readbody since 9h now and am seeing a very different memory usage pattern. Even with the Unhandled errors (both of them mentioned above) RSS has never exceeded 360 MB and is shrinking now and then (current RSS is 272 MB and VSZ 472 MB).

@DonnchaC
Copy link
Member

DonnchaC commented Feb 8, 2018

@ln5 Well spotted. We need to reintroduce the hashingReadBody code to minimize the memory usage when running the scans.

@juga0 juga0 added this to the simple_working milestone Mar 20, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants