[Carpet] WaveMoL in parallel

[Bela Szilagyi]

On Wednesday 15 March 2006 18:50, Erik Schnetter wrote:
...
>
> Near the boundary, this stencil has a width of 3 ghost zones.  The
> dissipation may also use one more ghost zone.  Peter?  If that is so,
> then there should be a check and a parameter warning for that.  Béla,
> I assume that there are no warnings printed when SummationByParts is
> initialised?
...
> > As a note, I had also tried to add my own dissipation right into
> > the WaveMoL
> > RHS routine and got the same result.
>
> What stencil width did you use for that?

SummationByParts is apparently happy with my parfile. 

What I notice (comparing HDF5 files) is a difference between the runs that is 
localized to the (outer) buffer zone's intersection with the ghostzones.

The reason SummationByParts may make a difference is that it actually uses 
more than just one ghost point, while the evolution code itself does not.  
Also, the near-boundary stencils for the dissipation are turned on, I 
believe, ONLY for penalty type boundaries, from what I understand:

in SummationByParts/src/dissipation.c "bbox" appears to be set to one for 
non-ghost boundaries that have to do something with multipach

  pen_sym_handle = SymmetryHandleOfName ( "multipatch" );

  for (d=0; d<6; ++d) {
    if ( symbnd[d] == pen_sym_handle && cctk_bbox[d] == 1 ) {
      bbox[d] = 1;
    } else {
      bbox[d] = 0;
    }
  }

while the Dissipation_*.F90 files only use modified SBP near-boundary stencils 
if this bbox is non-zero.

That is, I am not using anything else than centered (D_+D_-)^2 that is applied 
evenly at all points that are not too close to the boundary.

This is, by the way, the stencil I had also manually insert into WaveMoL, at 
points not too close to the boundary, to repeat the test.  

There's no place where I see a need for three ghost points.  Or does Carpet 
need that somewhere?  It should complain then.


I eliminated the problem by setting things as below.  My buffer_width setting 
is completely heuristic, but seems necessary.  Which I don't understand, 
given the "smart_outer_boundaries" feature being turned on.  Here I am using 
3 ghostzones and a higher order dissipation, but that is non-essential.  
Though may influence the minimum necessary buffer width.

Whatever the outcome, Erik, if you give me enough info (let it be some threads 
on the mailing list or a sketch of yours) I am willing to compile it into a 
section on "how to set up your grid with carpet" in the documentation.  Or, 
is there such a description already available?

# grid
#-----------------------------------------

Carpet::domain_from_coordbase = yes
Carpet::max_refinement_levels = 10

driver::ghost_size              = 3
Carpet::num_integrator_substeps = 3

Carpet::prolongation_order_space = 3
Carpet::prolongation_order_time  = 2

Carpet::convergence_level = 0

# Carpet::init_3_timelevels = yes

CoordBase::domainsize = minmax

CoordBase::xmin = -5.0
CoordBase::ymin = -5.0
CoordBase::zmin = -5.0
CoordBase::xmax =  5.0
CoordBase::ymax =  5.0
CoordBase::zmax =  5.0
CoordBase::dx   =  0.2
CoordBase::dy   =  0.2
CoordBase::dz   =  0.2

CartGrid3D::type         = coordbase
CartGrid3D::avoid_origin = no

CarpetRegrid::refinement_levels = 2
CarpetRegrid::refined_regions = manual-coordinate-list
CarpetRegrid::smart_outer_boundaries = yes
CarpetRegrid::coordinates = "
        [ [ ([-3.0,-3.0,-3.0]:[+3.0,+3.0,+3.0]:[0.10,0.10,0.10]) ] ]
"
CarpetRegrid::keep_same_grid_structure = yes
Carpet::buffer_width          = 2
CarpetLib::save_memory_during_regridding = yes



-- 
Bela Szilagyi
----------------------------------------------------
Max-Planck-Institut für Gravitationsphysik
Albert-Einstein-Institut
Tel: +49 331 567 7632
Fax: +49 331 567 7649
----------------------------------------------------