Delta GPUs results give NaNs in the viscous sub-grid bubble benchmark case whereas Phoenix GPUs do not

[sbryngelson]

Delta GPUs results give NaNs in the viscous sub-grid bubble benchmark case whereas Phoenix GPUs do not.

This is regardless of "memory size" (I've checked 4gb).

I've tested A100s and A40s on Delta, both give the issue discussed further on Slack.

I tested A100s and V100s on Phoenix, both of which do not give the issue.

Both computers use NVHPC 22.11.

Error is this:

 [ 40%]  Time step      358 of 901 @ t_step = 357
 [ 40%]  Time step      359 of 901 @ t_step = 358
 [ 40%]  Time step      360 of 901 @ t_step = 359
Warning: ieee_inexact is signaling
ERROR STOP NaN(s) in timestep output.
 NaN(s) in timestep output.            0            0            0            1
             0          360          198           99           99
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

One can run this case via something like
./mfc.sh run benchmarks/viscous_weno5_sgb_mono/case.py 4 -t pre_process simulation -c delta --gpu
if you are already on a node with GPUs and have loaded the appropriate modules.

[sbryngelson]

Update: This issue is associated with parallel_IO = 'T'. I witnessed it again on a Rogues Gallery GH200 chip w/ NVHPC 24.1.

[sbryngelson]

I'm not sure if this is still "broken" or not.

[sbryngelson]

Update: This is still broken. Related to PR #425

Update 2: This does not fail when case optimization is disabled. It only fails with case optimization enabled (on non-Phoenix computers).

I get the feeling that this line is not actually invoking case optimization....

MFC/.github/workflows/phoenix/bench.sh

Line 12 in 4f89f33

./mfc.sh bench --mem 8 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix $device_opts -n $n_ranks

@henryleberre

Update 3: Update 2 is incorrect and case optimization is not relevant

[henryleberre]

@sbryngelson I'm pretty sure --case-optimization is enabled by default in bench.

[wilfonba]

The logs indicate that case optimization is enabled on Phoenix for the benchmarking. There's recompilation of code in cases that I would expect to see recompilation due to case optimization.

[sbryngelson]

Nevermind, you're both right and it fails with and without case optimization on Delta (and presumably other computers).