Closed
Description
Delta GPUs results give NaNs in the viscous sub-grid bubble benchmark case whereas Phoenix GPUs do not.
This is regardless of "memory size" (I've checked 4gb).
I've tested A100s and A40s on Delta, both give the issue discussed further on Slack.
I tested A100s and V100s on Phoenix, both of which do not give the issue.
Both computers use NVHPC 22.11.
Error is this:
[ 40%] Time step 358 of 901 @ t_step = 357
[ 40%] Time step 359 of 901 @ t_step = 358
[ 40%] Time step 360 of 901 @ t_step = 359
Warning: ieee_inexact is signaling
ERROR STOP NaN(s) in timestep output.
NaN(s) in timestep output. 0 0 0 1
0 360 198 99 99
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------One can run this case via something like
./mfc.sh run benchmarks/viscous_weno5_sgb_mono/case.py 4 -t pre_process simulation -c delta --gpu
if you are already on a node with GPUs and have loaded the appropriate modules.