[Carpet] component geometry for multi-core processor chips

[Steve White]

Erik,

On 22.08.06, Erik Schnetter wrote:
> On Aug 16, 2006, at 07:23:16, Steve White wrote:
> 
> >Erik & al.
> >
> >Does Carpet do anything regarding clumping together geometrically
> >processes that are running on the same node?
> >
> >If 8 processes are running per node, it makes a very big performance
> >difference if the 8 on each node are arranged in a cube, because much
> >less network communication need be done.
> >
> >We already have low cost clusters with dual-core dual-chip  
> >motherboards,
> >which would benefit, and the quad core chips are soon to come.
> >
> >Any ideas on this?
> 
> Carpet splits the domain onto the processors approximately in the  
> following way: First the domain is cut in the z direction, the  
> resulting slabs are cut in their y directions, and the resulting  
> bricks are cut in the x direction.  These regions are the numbered  
> according to that hierarchy.  This leads to a numbering scheme that  
> is very similar to how Fortran lays out a 3D array in memory.
> 
That's what I thought.

> In particular, there are no 2x2x2 blocks that would be arranged in a  
> cube.
> 
> A question comes to my mind: How can one find out from PBS and from  
> MPI which processors are "close" to each other? 
>
I don't know, but I'll look into it.

I think there are resource flags to request that each node be
dedicated to the job.

> You seem to assume  
> that the processors within a node are numbered sequentially. 
>
I do not assume this.

> That does not seem to be so on Supermike, which enumerates first the first  
> processor on each node, then the second.  That seems to be a PBS  
> property, and I don't know whether MPI is clever enough to rearrange  
> the processors.
> 
> In order to experiment with this, I would first play with manual  
> processor decompositions and measure the speed differences.
> 
OK, I'll spend a little time with it this week

Thanks!

-- 
Steve White : Programmer
Max-Planck-Institut für Gravitationsphysik      Albert-Einstein-Institut
Am Mühlenberg 1, D-14476 Golm, Germany                  +49-331-567-7625