[Carpet] direction of mglevel?

[Erik Schnetter]

On Aug 24, 2006, at 11:55:43, Jonathan Thornburg wrote:

> Hi, Erik,
>
> Thanks for clarifying mglevel.  Meanwhile, Thomas and I have some
> more questions for you, mostly inspired by trying to fix my incorrect
> description of when global mode routines are executed:
> * I thought the standard loop order has mglevel as the outer loop,
>   and reflevel nested inside this.  But I see that the following 3
>   parts of the main time evolution loop invert this loop order,
>   i.e. do reflevel as outer loop and mglevel nested inside this:
>   * PRESTEP + EVOL
>   * Restrict
>   * POSTRESTRICT + POSTSTEP + CHECKPOINT + ANALYSIS + OutputGH
>   Why is the loop order "inside-out" for these?

When the simulation evolves physics, then different convergence  
levels are treated independently.  Since the refinement levels that  
should be evolved at a given iteration depend on the convergence  
level, the mglevel loop is the outer loop.  One could interchange the  
loops.  One reason for not interchanging them is that people may want  
to copy the fine grid result into the coarse grid result at certain  
times to keep the results from diverging, and therefore I wanted to  
evolve the finer convergence levels first.  However, I have not  
experimented with that.

Internally, in CarpetLib, mglevels are actually the last element of  
the hierarchy!  The internal CarpetLib hierarchy is: map, reflevel,  
component, mglevel.  This reflects that patches are virtually  
independent of each other, while the grid structure of different  
convergence levels has to be the same.

When the grid structure changes, then all convergence levels need to  
have the same grid structure.  This is why the grid structure should  
be set up once, and then applied to all convergence levels.  (Regrid  
has not even an mglevel loop.)

By the way, Evolution_Restrict has mglevel outside reflevel.

Initialisation is different from evolution.  In the initialisation  
loops, I generally have mglevel inside reflevel.  The grid structure  
is initialised reflevel by reflevel, making it possible to add finer  
reflevels until a certain accuracy criterion is satisfied.  On each  
reflevel, all mglevels are initialised together (in a tighter loop).

> * In EvolutionI(), are Thomas and I correct in our belief that
>     do_global_mode  ==  (rl == 0) ?  If so, why did you write it
>    in the more complicated way you did, using the   
> have_done_global_mode
>   flag?

This is not correct.  The coarse grid is evolved much more rarely  
than the finer grids.  do_every is often false for rl=0, but true for  
larger rl.

timereffacts for smaller rl are smaller (these are refinement  
factors).  Therefore the quotient that you look at is larger for  
smaller rl.

> * In EvolutionII() you write the  do_global_mode  computation  in
>   terms of  finest_active_reflevel .  We're confused about why
>   you do this (when everywhere else it's either  rl == 0   or
>   rl == maximum possible rl).  Can you clarify?

do_global_mode is only equivalent to rl==reflevels-1 during  
initialisation.  Initialisation is different from evolution, since  
the levels are initialised one by one.  That is, all levels are  
"active" at t=0.

During evolution, at any given time, not all levels may be active,  
since some levels may be skipped at that time.  Therefore  
finest_active_reflevel can be different from reflevels-1.

The test is nowhere rl == maximum possible rl.  This would be an  
error, since changing the maximum possible rl must not have any  
effect on the physics.

EvolutionII proceeds from coarsest to finest to allow people to apply  
boundary conditions, which interpolate from coarser levels.  In my  
experience, people prefer to have global mode routines run after  
local mode routines in the analysis bin, which motivated my choice.   
(This allows people e.g. to call reduction routines.)  During  
EvolutionI, in my experience people prefer to run global mode  
routines before local mode routines.  (This allows people e.g. to  
choose a gauge condition.)

In the end, it turned out that these "preferences" do not work in  
general, which is why there are now global loop-local routines.  This  
mechanism allows people to choose the order between global and local  
routines, and it allows people to have more complicated schedules as  
well.  At the moment, I think that relying on the order between  
global and local routines is almost always a mistake, and I keep this  
order only for backwards compatibility.  If you need a certain order,  
then use global and global loop-local routines instead.



By the way, I find more and more that the Cactus scheduler in its  
current form is too restrictive.  This is why I try to advertise  
"schedule scripts", written by hand, which allow people to switch  
modes, loop over levels, and activate/deactivate storage at will.   
There are no working examples yet, and I'm sure there are bugs that  
need to be corrected, but in the end I think that such scripts will  
be easier to understand than the scheduler gymnastics found e.g. in  
MoL.  Tom Goodale has mentioned schedule scripts as a likely future  
feature of Cactus, meaning that it would be a good thing to  
experiment now with prototypes of such scripts for mesh refinement.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/developers/attachments/20060825/5d1e6767/attachment.pgp