[Carpet] memory allocation in global loop-local mode?

[Erik Schnetter]

On Aug 23, 2006, at 13:59:13, Bela Szilagyi wrote:

> In one of my thorns (taken after AHFinderDirect's mask-setter  
> routine schedule
> bing) I had
>
>
> schedule my_routine in my_group
>         {
>         lang: C
>        storage: my_var
>         options: global loop-local
>         } "..."
>
>
> while the parfile says
>
> Carpet::enable_all_storage = no
>
> The routine ended up being called, at least once within the first  
> course
> time-step worth of time, with my_var being set to NULL.
>
> A bit of peaking into it with gdb has shown that not all calls to  
> the routine
> were lacking the memory, but, in the one case this variable would  
> have been
> used, the memory was not there.
>
> Is this a bug or a feature? In other terms -- does "global loop- 
> local" imply
> that one should not assume memory associated with tmp_var ?

Béla,

thanks for the screen output in your bug report.  The funny pattern  
triggered a few sluggish neurons in my brain, and this output is  
actually to be expected.  It is a "feature".

Storage is switched on and off by the scheduler in the flesh, not by  
Carpet.  The iteration over all levels happens later in Carpet.  The  
overall sequence of events is thus:

	1. Scheduler activates storage for the current level
	2. Carpet loops over all levels, calling the function
	3. Scheduler deactivates storage for the current level

Scheduling happens always in level mode; there is no convenient way  
around this.  Thus only exactly one level has storage switched on.  I  
can think of two ways to circumvent this "feature":

- Activate storage for the temporary variable globally
- Switch storage on and off manually

If you switch storage of a grid function in level mode, then only the  
current level is affected.  If you switch storage in global mode,  
then all levels are affected.  Thus you could, depending on your  
needs, either

BEGIN_GLOBAL_MODE {
	activate storage;
	BEGIN_REFLEVEL_LOOP {
		do stuff;
	} END_REFLEVEL_LOOP;
	deactivate storage;
} END_GLOBAL_MODE;

or

BEGIN_GLOBAL_MODE {
	BEGIN_REFLEVEL_LOOP {
		activate storage;
		do stuff;
		deactivate storage;
	} END_REFLEVEL_LOOP;
} END_GLOBAL_MODE;



The first scheme can be written in the scheduler (if you prefer that;  
it is slightly complicated)

SCHEDULE activate_storage AT evol BEFORE dostuff
{
	OPTIONS: global
}
SCHEDULE dostuff AT evol
{
	OPTIONS: global loop-local
}
SCHEDULE deactivate_storage AT evol AFTER dostuff
{
	OPTIONS: global
}

where activate_storage and deactivate_storage are routines that you  
have to write which call CCTK_GroupStorageIncrease and -Decrease,  
respectively.

The second scheme can be written as a single scheduled routine, as  
you did.  Your routine would need to be slightly modified:

void
dostuff (CCTK_ARGUMENTS)
{
	CCTK_GroupStorageIncrease;
	{
		DECLARE_CCTK_ARGUMENTS;
		do stuff;
	}
	CCTK_GroupStorageDecrease;
}

You have to enable storage before declaring your grid functions, so  
that the grid function pointers point to the newly allocated  
storage.  This does not work in Fortran.



I am sorry about this complication.  If my description above is too  
complicated, I'd be happy to help you implement this, hoping to  
create a clean example for others to look at.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/developers/attachments/20060825/b8bc6c84/attachment.pgp