Assert no nested MFIters

[WeiqunZhang]

Summary

Add assertion to catch nested MFIters (e.g., MultiFab functions are called inside MFIter).

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• are described in the proposed changes to the AMReX documentation, if appropriate

[drummerdoc]

Cool idea!

[maximumcats]

This is indeed a good idea but we need to fix the implementation. This breaks AmrLevel (at least) because FillPatchIterator is derived from MFIter

FillPatchIterator::FillPatchIterator (AmrLevel& amrlevel,
                                      MultiFab& leveldata,
                                      int       boxGrow,
                                      Real      time,
                                      int       idx,
                                      int       scomp,
                                      int       ncomp)
    :
    MFIter(leveldata),
    m_amrlevel(amrlevel),
    m_leveldata(leveldata),
    m_ncomp(ncomp)
{
    BL_ASSERT(scomp >= 0);
    BL_ASSERT(ncomp >= 1);
    BL_ASSERT(AmrLevel::desc_lst[idx].inRange(scomp,ncomp));
    BL_ASSERT(0 <= idx && idx < AmrLevel::desc_lst.size());

    Initialize(boxGrow,time,idx,scomp,ncomp);

#ifdef BL_USE_TEAM
    ParallelDescriptor::MyTeam().MemoryBarrier();
#endif
}

and so the constructor will increment depth here -- if Initialize has any MFIter loops this will cause a false positive. And it does, the call to setDomainBndry (at least).

[WeiqunZhang]

Thanks for catching this. This is tricky. There is another case that can cause problems. I think some codes do

for (FillPatchIterator A(...), B(...); ....)

I think we can fix it by setting MFIter::depth = 0 in FillPatchIterator ctor.