Simultaneous model building and running

[craig-warren]

In the case where you have multiple models to simulate and are using the GPU-based solver:

• is it possible to run the build phase (which always happens on CPU) of the next model in the series whilst the previous model is still executing on GPU?

This could improve overall simulation times.

[mzmmoazam]

We can do this using a GPU based FIFO queue. There was an issue leading to GPU starvation in Tensorflow v.1, they have fixed it using QueueRunner and a coordinator (due to their graph architecture). I believe FIFO will achieve the goal we are trying to reach.
Using this approach we can get a minimum of 35% performance enhancement.

[rsettlage]

In some cases, it would be even more beneficial to completely de-couple these two steps. On our system, we can span more cores/nodes on the CPU side through different HPC queues and then only use the (expensive) GPU nodes for the last step of the compute.

[rsettlage]

I should have added some details. In our system, for the CPU queues, I can span 64 nodes each having 128 cores (cost factor=0.0023/core). On the GPU side, I am limited to 8 a100s (cost factor=155/GPU). In my ideal case, we would rip through the CPU tasks on the "cheap" nodes and only hit the GPU nodes for the GPU task.

[Harshsaini001]

import gprMax

models = [
gprMax.Model(data1),
gprMax.Model(data2),
gprMax.Model(data3),
]

results = gprMax.runModelsAsync(models)

for result in results:
print(result.predictions)
print(result.error)

[Jay-sanjay]

Hello everyone , I wanted to contribute to the gprMax project , can someone assist me further for it.