[Carpet] Carpet scheduling question

[Erik Schnetter]

On Aug 23, 2006, at 02:52:11, Frank Loeffler wrote:

> Hi,
>
> Jonathan Thornburg wrote:
>> and neither of them knows everything.  In particular:
>> * the scheduler doesn't know anything about Carpet modes or  
>> refinement
>>   levels (only Carpet knows that)
>> * Carpet doesn't know which routines are scheduled before/after which
>>   other routines (only the flesh knows that)
>
> Let my try to understand this, which might be also enlightening to
> others, therefore to the list.
> If only Cactus knows when and in which order to schedule functions and
> Carpet cannot do something about this, how does it come that the
> functions are not always executed in the order specified?
>
> In particular, in our example:
>
>     schedule A in CCTK_POSTINITIAL
>      {
>      } "routine A"
>     schedule B in CCTK_POSTINITIAL after A
>      {
>      options: global
>      } "routine B"
>
> Why does this end up as:
>
>     call A in local mode on level 2, component 0
>     call B in global mode
>     call A in local mode on level 1, component 0
>     call A in local mode on level 0, component 0
>
> and not as first all calls of A and after that the call to B?

The Berger-Oliger time evolution scheme, and similarly the Berger- 
Oliger way of setting up initial data, proceeds through the levels  
sequentially.  It is necessary to set up initial data for the coarse  
level before beginning to work on the finer levels.  Otherwise, one  
cannot prolongate to the fine grid boundaries.

The standard way to set up initial data, after which I modelled  
Carpet, is the following:

1. Set up coordinates x, y, z
2. Set up ADM variables from coordinates
3. Calculate BSSN variables from ADM variables

Since people may want to apply boundary conditions during this  
procedure (e.g. to set up the Gamma^i on the boundaries, or to  
overwrite the infinities in the excision region), things have to be  
initialised level-wise.

Also, you are assuming that you have data given on all levels.  You  
give the postinitial bin in your problem, where this is the case.   
During time evolution, you will usually not have data on all levels,  
since the coarse grid takes larger time steps.  In this case a simple  
scheme will not work; it will work only when all levels are in sync.

> I suspect, that Cactus does not do all the calls to A, because it does
> not know about refinement levels.
> If so, Carpet must be the piece that does these calls, possibly
> 'motivated' by a request from Cactus to call A.
> But if this is the case, why not calling A on all levels immediately?

The way in which things are handled by Carpet is not dictated by  
Cactus.  If Cactus is too restrictive in some way, then Cactus can be  
changed.  The real reasons why Carpet doesn't always use a global/ 
loop-local scheme is that this doesn't work in general.

Maybe there could be a simpler way to examine a snapshot of a  
simulation.  This would involve allocating storage for a grid  
function for all levels, and calculating it on all levels (involving  
in general time interpolation on coarser levels).  Such a scheme  
would naturally use a global/loop-local scheduling.  However, such a  
scheme would be expensive, since it calculates quantities on all  
levels on each fine grid time step.  If there are n levels, then this  
increases the cost by a factor n/2 (just the factor that the Berger- 
Oliger scheme saves).

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/developers/attachments/20060823/07305299/attachment.pgp