[Carpet] Regridding and scheduling in new version of Carpet

[Ian Hinder]

Erik Schnetter wrote:

> I notice that our three groups have three different regridding thorns
> which essentially all do the same thing.
> 
> CarpetRegrid2::min_fraction specifies the minimum fraction of refined
> points that need to be present in a single, cuboid box that replaces two
> individual overlapping boxes.  If the fraction is lower, there is no
> single cuboid boxes; instead a concave region is used.  By setting this
> parameter to 1 there is never a single, cuboid box; by setting it to 0,
> there are never concave regions.
> 
> Using a single,cuboid box can increase the memory requirements by a
> factor of two.

But typically only on the finest grids, which make up a small fraction
of the overall memory usage, especially when you consider that the grids
for wave extraction usually have 8 times as many points as the grids for
the punctures in a BBH simulation.  Also, we do not tend to be limited
currently by available memory, and if the new version of Carpet scales
as well as we hope, this will become even less of an issue.

> Note that this condition is applied before the hierarchy is made
> self-consistent and consistent with the symmetry conditions, so that a
> concave region can still result later on.

A concave region makes it very difficult to adapt the finite
differencing stencil relative to the refinement boundary without
additional support from Carpet.  It also makes me nervous from
stability and complexity perspectives.

> Note also that CarpetRegrid performs none of these actions and none of
> these checks, so that it is possible to end up with an inconsistent grid
> hierarchy, or a grid hierarchy where many more grid points are
> interpolated from the coarse grid than intended.

I'm not sure I understand.  What do you mean by the grids being
'self-consistent'?  Are you referring to proper nesting?  I would rather
that the code aborted (as it does in the new version) in the case of
improper nesting than silently giving me a different grid structure,
which might have implications for convergence.  Consistency with the
symmetry conditions is ensured by my regridding thorn (at least for
rotating symmetry).  In what case can more grid points be interpolated
than intended?

I am not in principle averse to using CarpetRegrid2, and it would be
beneficial to have this code in one place rather than three.  For this,
we would need support from Carpet to make sure that the refined regions
were actually rectangular boxes and not L-shaped.  I think I have the
regridding working now with the new version, but we can work on adding
functionality to CarpetRegrid2 and think about a transition eventually.

On the other hand, remember that CarpetRegrid2 is a very specialized
thorn which is pretty much only of use in the particular simulations
that many of us are running, where we have a small number of centres
about which we want refined regions of certain radii.  CarpetRegrid
allows the user to easily specify arbitrary refined regions, which is
obviously much more general, so I think it should still be supported.

-- 
Ian Hinder
hinder at gravity.psu.edu
http://www.gravity.psu.edu/~hinder