[SYSTEMDS-3891] New out-of-core instructions and improvements / bugfixes

[janniklinde]

This patch adds new out-of-core instructions such as rand (only constant support using matrix(v, ...)) and rsort, which are required for builtin PCA. Further, the patch adds support for broadcasting binary operations (on matrices where dimensions are known) and improves performance for TSMM by enabling parallel processing. Finally, bugfixes are provided for CachingStream (fixing error propagation), for DataOp (to propagate dimensions in the HOP DAG when having Tee nodes), and for MatrixObject (to support acquireRead on unknown dimensions).

[janniklinde]

Removed the invalid blocksize propagation which I added for completeness. Now the io tests should pass @mboehm7

[janniklinde]

Re-ran the failing tests locally but could not reproduce

[codecov]

Codecov Report

❌ Patch coverage is 46.95341% with 148 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.28%. Comparing base (b090e49) to head (dd0d7f2).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...sysds/runtime/instructions/ooc/OOCInstruction.java	0.00%	51 Missing ⚠️
...untime/instructions/ooc/DataGenOOCInstruction.java	65.62%	17 Missing and 16 partials ⚠️
...runtime/instructions/ooc/BinaryOOCInstruction.java	14.28%	24 Missing and 6 partials ⚠️
...s/runtime/controlprogram/caching/MatrixObject.java	17.64%	9 Missing and 5 partials ⚠️
.../sysds/runtime/instructions/ooc/CachingStream.java	7.14%	12 Missing and 1 partial ⚠️
.../runtime/instructions/ooc/ReorgOOCInstruction.java	88.46%	2 Missing and 4 partials ⚠️
src/main/java/org/apache/sysds/lops/Transform.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2362      +/-   ##
============================================
- Coverage     72.33%   72.28%   -0.05%     
- Complexity    46909    46933      +24     
============================================
  Files          1513     1513              
  Lines        178198   178429     +231     
  Branches      34984    35037      +53     
============================================
+ Hits         128902   128982      +80     
- Misses        39545    39681     +136     
- Partials       9751     9766      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mboehm7]

LGTM - thanks @janniklinde. Sometimes, there are spurious failures and we have to trigger these tests again. During the merge, I only removed some unused variables from the OOC datagen instruction.