Clever caching policy involving RemoteData

[unkcpz]

Next action

Have a discussion on the issue and potential solution.

Description

• add by @jusongyu

Calculations which pass (part of) their outputs on as a RemoteData input are broken once their scratch directory is cleaned. It becomes problematic however when caching is introduced: When trying to re-run the calculation (to regenerate the remote folder), caching will happily use the existing calculation (without a remote folder). As a result, subsequent calculations which use the remote folder will fail. Moreover, the cached Calculation node point to the same remote folder which is a 'shallow copy' of the original calculation node that causes the issue when cleaning the remote workdir of cached node, the original node is unable to used for parent calculation to pass their outputs on as a RemoteData. The caching policy needs to be redesigned a bit to overcome these issues and balance with not increasing the connection burden and remote disk storage too much.

See #3735 and #5178 for more details.

AEP open at aiidateam/AEP#35

[sphuber]

See aiidateam/team-compass#13