[Carpet] ApplyBCs and global mode scheduling

[Bela Szilagyi]

It seems that the two "types" of scheduling (i.e., "global" and "level") are 
introducing all kinds of subtle inconsistencies in the Carpet user's code.  
Some of these may be fixed by editing the local schedule.ccl files, but 
others are a harder nut.

Take, e.g., a thorn of mine which wants to convert from the "harmonic" 
variables into the ADM variables at post-step, before AHFinderDirect finds 
its horizons.

From what I gather AHFidnerDirect's finder routine itself must be in "global" 
mode so that it can interpolate data across all refinement levels.  By 
implication, if I want my conversion routine scheduled before AHFinderDirect, 
I must schedule it in "global loop-local" mode.  Next comes the issue of 
symmetries.  Some of them  get applied to the gridfunctions by registering my 
variables with cactus and then scheduling the "ApplyBCs" group. If one peaks 
at CactusBase/Boundaries the routines Boundary_ApplyPHysicalBCs as well as 
Boundary_ClearSelection are both scheduled in loop-level mode.  That is, I 
will not get my boundary conditions applied correctly to the variables 
computed in global loop- modes.

I guess the "best" hack is to scedule my own global loop-local routine from 
within which I call directly all the routines I can find scheduled into 
ApplyBCs, presumably keeping the right order.  But, by standards is this a 
clean solution.




-- 
Bela Szilagyi

Max-Planck-Institut für Gravitationsphysik
Albert-Einstein-Institut
Tel: +49 331 567 7632
Fax: +49 331 567 7649