[Carpet] cost of repeated calls on CCTK_SyncGroup()

[Jonathan Thornburg]

Hi, Erik,

Some algorithms Bela and I are working on typically need to make
on the order of 10 sweeps-over-the-grid per time step, each of which
requires calling CCTK_SyncGroup() on 20 or more grid functions.
Together, all these synchronizations are taking a *lot* of time
(over half the total run time in some of Bela's tests).

My basic question is, how could this be optimized?

The flesh provides a  CCTK_SyncGroupsI()  function to synchronize a
whole set of groups, but this is implemented as a sequential loop over
the groups, calling  CCTK_SyncGroup()  on each one.

Looking in the Carpet source code, I see a  SyncProlongateGroups()
function in  Carpet/Carpet/src/Comm.cc , but it also prolongates
the boundaries, which I don't want here.

Question:
Is there some deep reason this function *must* prolongate as well as
synchronize?  Looking at the code, it looks easy enough to turn off
the prolongation where needed.  That is, suppose I were to add a
Boolean parameter 'prolongate' to this function, defaulting to true
for backwards compatability, and have the function only prolongate
if this parameter is true.

I could then call this function from my thorn
	[either as an aliased fn or as an overload of
	 CCTK_SyncGroupsI() ]
to (hopefully) more efficiently synchronize my entire set of grid
functions all at once.

Would this be a reasonable thing to do?


ciao,

-- 
-- Jonathan Thornburg <jthorn at aei.mpg.de>      
   Max-Planck-Institut fuer Gravitationsphysik (Albert-Einstein-Institut),
   Golm, Germany, "Old Europe"     http://www.aei.mpg.de/~jthorn/home.html      
   "Washing one's hands of the conflict between the powerful and the
    powerless means to side with the powerful, not to be neutral."
                                      -- quote by Freire / poster by Oxfam