Nontensor gen operators

[jeremylt]

Non-tensor gen operators 🎉

Closes #839

• Testing
• HIP
• [ ] Support tensor + non-tensor mix (follow-up) Gen Tensor/NonTensor Mixed #1736
• [ ] Duplicate mat loads (follow-up) De-duplicate Gen Basis Matrices #1737

Note: IBM Power test failure is unrelated to this PR

[jedbrown]

Cool! Any preliminary performance comparison?

[jeremylt]

Not yet, still chasing some odd bugs here and there

[jeremylt]

Ok, pure non-tensor works now in 1, 2, and 3D. I need to get operators with a mix of tensor and non-tensor elements working, or put in bypass for now

[jrwrigh]

I need to get operators with a mix of tensor and non-tensor elements working, or put in bypass for now

You mean as a single operator or as suboperators? I had imagined that each operators would have to work with a single element type. So a composite operator would have 1 suboperator per element type.

Edit: Continuing that thought, a CeedBasis would only be valid for a single element type anyways, so I think the single-element-type-per-operator approach makes sense from that limitation as well.

[jeremylt]

I'm talking about within a single operator. Operators must share a quadrature space but can have different element types feeding into the same quadrature space. Some of our operators on faces are like that in Ratel.

[jrwrigh]

As in surface-based CeedBasis being used for a volume operator?

[jeremylt]

One instance off the top of my head is computing geometric factors on the face with some full cell data

[jeremylt]

ToDo: Write up issues on follow-up tasks, run test suite on my ROCm machine

[jeremylt]

on my local machine, I'm seeing for shared

    DoFs/Sec in CG                          : 29.1799 (29.1799) million

but for gen

    DoFs/Sec in CG                          : 9.70631 (9.70631) million

when testing BP3. Digging around currently

With the same mesh though, gen can run BP4 and shared cannot, so we are getting a big memory savings here.

[jeremylt]

Yup, gen goes up to

    DoFs/Sec in CG                          : 33.0333 (33.0333) million

if I used the thread block strategy from shared bases. So a slight improvement in perf but the real win is the big decrease in memory used.

[jeremylt]

I'm investigating why the ARM/IBM Power jobs are misbehaving but its unrelated to this PR so we can merge it without those passing