[Carpet] Lower-dimensional HDF5 output?

[Ian Hinder]

> On Wed, 30 Aug 2006, Erik Schnetter wrote:
> 
> > We are currently using ASCII output for 0D and 1D output of grid functions.
> > This has several disadvantages: There is much overhead, there are too many
> > files, output happens on a single processor only, and the output is currently
> > chunked.
> > 
> > One way out would be to implement lower-dimensional HDF5 output.  I imagine
> > the same possibilities as for ASCII output, as in
> > 
> > IOHDF5::out1D_every = 16
> > IOHDF5::out1D_vars = "..."
> > IOHDF5::out1D_x = yes
> > IOHDF5::out1D_x = no
> > 
> > After recovering, it would be much easier to eliminate overlapping datasets.
> > 
> > In addition, we could implement IOHDF5::one_file_per_group, which would reduce
> > the number of output files.
> > 
> > There would be post-processing tools to create unchunked ASCII files from
> > these HDF5 files.
> > 
> > Is there interest for such a feature?  Or would people prefer incremental
> > improvements to ASCII output?

This is a feature I think would be very useful.  As far as I know, if
you don't want to output full 3D data, the only option is to output
ASCII, and has the problems you mentioned.  Either a standard utility
to postprocess the ASCII files, or separate 1D and 2D HDF5 output,
would solve these problems.  

If postprocessing, the utility should do the following:

* Eliminate data that has been output twice (was this ever fixed?)

* Unchunk the output

* Have the option of stripping ghost zones and buffer zones for
visualization purposes, or keeping them for debugging (the number of
zones would probably have to be a parameter, as it is not stored in
the output file)

* Work with both 1D and 2D data

There are then the separate issues of converting the 1D data into a
format suitable for ygraph, and of merging output from
checkpoint/recover runs, which in principle are covered by existing
utilities.  But maybe it would be nice to merge these features into
one utility?

I have not measured the overhead associated with the ASCII output; you
say there is a lot?  In that case, for performance reasons it would
make sense to use HDF5. Also, there is the file size aspect to
consider.  ASCII is much less efficient in terms of storage than HDF5.

-- 
Ian Hinder
hinder at gravity.psu.edu
http://www.gravity.psu.edu/~hinder