Memory Issue in the VHI preprocessor

[gabrieltseng]

Trying to preprocess all the VHI data leads to the process being killed:

Starting work on VHP.G04.C07.NJ.P1997031.VH.nc
Killed

Looking into the logs of the google cloud instance indicates it was caused by a memory error:

Jun 30 20:18:15 instance-1 kernel: [2687694.642439] oom kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/session-289.scope,task=python,pid=11241,uid=1001
Jun 30 20:18:15 instance-1 kernel: [2687694.642466] Out of memory: Kill process 11241 (python) score 772 or sacrifice child
Jun 30 20:18:15 instance-1 kernel: [2687694.651553] Killed process 11241 (python) total-vm:3070856kB, anon-rss:2911760kB, file-rss:2768kB, shmem-rss:0kB

The (hackey) solution is to just restart it from where it failed (i.e. only process files > 1997 if it failed at 1997, using a statement like if int(file.parents[0].stem) >= 1997:), but it would be nice to also fix it properly.

cc @tommylees112

[gabrieltseng]

Limit is 14 years