How to completely update an AmrCore's boxArray and distributeMap?

[pkufourier]

Right now in my application code for multilevels with the AMReX tutorial framework that derived from AmrCore. And the code generated the following refined boxArrrays:
Level= 0
CPUid=0, box No.= 0, i=( 0, 99), j=( 0, 49), k=( 0, 3)
CPUid=0, box No.= 1, i=( 0, 99), j=( 50, 99), k=( 0, 3)
Level= 1
CPUid=0, box No.= 0, i=( 64, 169), j=( 64, 103), k=( 0, 7)
CPUid=0, box No.= 1, i=( 64, 169), j=( 104, 135), k=( 0, 7)
Level= 2
CPUid=0, box No.= 0, i=( 158, 281), j=( 158, 199), k=( 0, 15)
CPUid=0, box No.= 1, i=( 158, 281), j=( 200, 241), k=( 0, 15)

I would like to equally devide in y direction with 2 boxes. But we can see that, the level=1's y split is not satisfactory. The perfect split should be j=(64, 99) and j=(100,135), and I am not sure what happened for this strange box split. But the level=0 and lev=2 are perfect.

Considering this, I manually redefine the BoxArray and DistributingMap in the MakeNewLevelFromScratch by split the j=(64, 103) into j=(64,99) and j=(100,104) and the code is:

    SplitForContainment(boxArray(lev - 1), DistributionMap(lev - 1),
        ba, refRatio(lev - 1), new_ba, new_dm);
    ClearDistributionMap(lev);
    ClearBoxArray(lev);
    SetBoxArray(lev, new_ba);
    SetDistributionMap(lev, new_dm);
    AllocateMultiFab(lev, new_ba, new_dm);

Then I print the new BoxArray and DistributinMap as:

BoxArray ba_test = boxArray(lev);
DistributionMapping dm_test = DistributionMap(lev);

if (ParallelDescriptor::MyProc() == 0) {
Print(0) << "\n id=0 ba=" << ba_test;
Print(0) << "\n id=0 new dm=" << dm_test;

  Print(0) << "Screatch complete  on id=0\n";

}

if (ParallelDescriptor::MyProc() == 1) {
Print(1) << "id=1 new ba=" << ba_test;
Print(1) << "id = 1 new dm = " << dm_test;

  Print(1) << "Screatch complete  on id=1\n";

}
The output is:

id=0 ba=(BoxArray maxbox(3)
m_ref->m_hash_sig(0)
((64,64,0) (169,99,7) (0,0,0)) ((64,100,0) (169,103,7) (0,0,0)) ((64,104,0) (169,135,7) (0,0,0)) )

id=0 new dm=(DistributionMapping
m_pmap[0] = 0
m_pmap[1] = 1
m_pmap[2] = 1
)
Screatch complete on id=0
id=1 new ba=(BoxArray maxbox(3)
m_ref->m_hash_sig(0)
((64,64,0) (169,99,7) (0,0,0)) ((64,100,0) (169,103,7) (0,0,0)) ((64,104,0) (169,135,7) (0,0,0)) )
id = 1 new dm = (DistributionMapping
m_pmap[0] = 0
m_pmap[1] = 1
m_pmap[2] = 1
)

Indeed they are as expected. But now crash happens. Before Makescratch on level=2, the assert error report:
0::Assertion dm.ProcessorMap().size() == bxs.size()' failed, file "/home/yangkun/amrex-25.08/Src/Base/AMReX_FabArrayBase.cpp", line 202 !!! SIGABRT 1::Assertion dm.ProcessorMap().size() == bxs.size()' failed, file "/home/yangkun/amrex-25.08/Src/Base/AMReX_FabArrayBase.cpp", line 202 !!!
SIGABRT

I am wondering wether the AmrCore's InitFromScratch() is still using the old BA and DM somewhere, and conflict with my updated BA&DM. Perhapes it is the errorTag that needs to be called in the ErrorEst to generate the level=2 mesh.

[WeiqunZhang]

I don't know exactly what has happened. You can probably find it out by looking the backtraces. Nevertheless, you should NOT change the BoxArrays provided by amrex. There is a reason why a cut in y-direction chops (64,135) into (64,103) and (104, 135), not (64, 99) and (100, 135). There are a total of 72 cells. So one might think the best way is to cut them two equal parts of 36 cells. But there is also the constraint of blocking factor. It's likely that you are using a blocking factor of 8. So 72 cells are 5 blocks. We can only chop them into 3 and 2 blocks, not two 2.5 blocks.

[pkufourier]

Well, the divide of the inequal of boxes might depend on the load-balance policy. But it is not a critial point. What I am trying to do is, ensure the fine level boxes are covered by the coarse level boxes in the same MPI process.

In this case, the level=1 box that j=64~103 would cross the bound of the lev=0 mesh in id=0.
That is, the j=100~103 is covered by the id=1 process of lev0 box.
So I am trying to break the j=64~103 into two boxes: j=64~99 and j=100~103,
and rearrange the j=100~103 box of lev=1 into id=1 process.
In this way, some operations, like the average down will not need to send a large piece of data from other processes. If we can set some option for amrex to enforce "the same prcess contain the fine and coarse level boxes", that would be better.

[WeiqunZhang]

You are allowed to set your own DistributionMapping.

If we can set some option for amrex to enforce "the same prcess contain the fine and coarse level boxes", that would be better.

Usually that will create a big load imbalance. Many people have tried to do that. But it never seemed to work well for general cases.