[Carpet] Lower-dimensional HDF5 output?

[Thomas Radke]

Let me ask a few questions here which, I admit, are driven by the fear 
that a big pile of work is about to land on my desk :-/

Erik Schnetter wrote:
> We are currently using ASCII output for 0D and 1D output of grid  
> functions.  This has several disadvantages: There is much overhead,  

Is there really ?

> there are too many files, 

Which can be reduced by setting {IOScalar,IOASCII}::one_file_per_group 
to 'yes'

> output happens on a single processor only,  

Again, how much does this overhead contribute to the total simulation time ?

> and the output is currently chunked.

True. Is this a problem for postprocessing ?

> One way out would be to implement lower-dimensional HDF5 output.  I  
> imagine the same possibilities as for ASCII output, as in
> 
> IOHDF5::out1D_every = 16
> IOHDF5::out1D_vars = "..."
> IOHDF5::out1D_x = yes
> IOHDF5::out1D_x = no
> 
> After recovering, it would be much easier to eliminate overlapping  
> datasets.

Yes, HDF5 output solves that problem of overlapping timesteps 
internally. But there are these scripts mergeCarpetIOASCII.pl and 
mergeCarpetIOScalar.pl available now which do the same, however, as a 
postprocessing utility only.

> In addition, we could implement IOHDF5::one_file_per_group, which  would 
> reduce the number of output files.

Already there for CarpetIOScalar and CarpetIOASCII.

> There would be post-processing tools to create unchunked ASCII files  
> from these HDF5 files.
> 
> Is there interest for such a feature?  Or would people prefer  
> incremental improvements to ASCII output?

I believe the real benefits of Carpet HDF5 output could be exploited 
when such data files are analysed/postprocessed/visualised directly. So 
far I see many Cactus users not doing this and use standard ASCII-based 
tools instead. As soon as some format conversion, extraction, 
recombination etc. is involved, the question (to me) becomes: what's 
more efficient, implement these features in CarpetIOHDF5 or as 
stand-alone postprocessing utility programs/scripts. So far I found the 
latter to be quicker.

Don't get me wrong, I'm the last who will advocate against using HDF5. 
It's the non-HDF5 parts, eg. (to answer Bela's question) allowing HDF5 
output files being used and compared in testsuites, which may cause 
quite some extra coding work.

-- 
Cheers, Thomas.