[Carpet] Too many HDF5 files?

[Erik Schnetter]

On Apr 25, 2007, at 11:34:31, Thomas Radke wrote:

> Erik Schnetter wrote:

>> Should we implement another  method
>> to reduce the number of files?  We could e.g. write one output   
>> file per
>> group, per thorn, or even per iteration.  Do you have  another  
>> suggestion?
>
> One file per group shouldn't be difficult to implement, one just needs
> to copy the logic which is already there in IOScalar or IOASCII,  
> right ?
> One file per thorn sounds a little strange to me. One per iteration
> basically amount to writing a checkpoint.
>
>> If we combine different variables into the same file, is there   
>> anything
>> special that needs to be added?  I think it should be  possible to  
>> just
>> write several variables into the same file, adding  the meta- 
>> information
>> (grid structure etc.) only once.  Is that correct?
>
> Yes, I think so. I'll take a closer look on the one_file_per_group
> implementation of IOScalar.

If it is not too difficult, then I can also try that myself.

> Would that be enough in order to reduce the total number of HDF5  
> output
> files ?

I don't know whether this will be sufficient, but it should help.

However, I think the basic problem is that there is one file per  
processor -- on 1000 processors the output directory will just  
overflow.  Maybe the correct approach would be to have different  
output subdirectories on different processors?  This should also  
improve performance a bit, since changes to the subdirectories are  
then interesting only for a single processor, whereas with a global  
output directory each processor has to be informed about newly  
created files, resulting in lock contention.  (If I understood Maciek  
correctly.)

Would that work with the current tools?  The current tools accept the  
file name of the root processor, and then automatically construct the  
file names for the other processors.  This would then need to be  
augmented.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/developers/attachments/20070425/af750377/attachment.pgp