[Carpet] memory leak while running with carpet
Erik Schnetter
schnetter at cct.lsu.edu
Tue Jul 10 11:05:25 CEST 2007
Hi Yosef,
we found that the HDF5 library allocates some memory for internal
purposes, and continues to hold on to it, probably buffering
something. This is much worse for 1.6.x than for 1.8.x. You could
give 1.8.x a try.
Christian Ott found a misbehaviour of the GNU libc which is triggered
by CarpetIOASCII (and probably also CarpetIOScalar and
CarpetIOBasic.) Can you try running without them (or without
outputting anything with them) to see whether this helps?
Apart from this a memory leak is difficult to track down. It could
be anywhere -- Cactus, Carpet, your code, or it could be some
interaction between them. There are no "relevant sections" of a
parameter file; there can only be such section if you know already
where the memory leak occurs.
Can you try to remove thorns from your thorn list, trying to make the
memory leak go away? After that it is necessary to watch the
simulation, probably in a debugger, and catch it increasing its
memory usage. If you can point me to a routine we can go on from there.
I'm afraid that your section of a parameter file doesn't ring any
bell. Your parameter settings look pretty normal, and I don't see
such a drastic problem.
-erik
On Jul 10, 2007, at 03:54:45, Yosef Zlochower wrote:
> Hi,
>
> I forgot to mention that the memory leak is quite significant. The
> memory load tends
> to double after a few hours of runtime.
>
> Yosef
>
> Yosef Zlochower wrote:
>> Hi
>>
>> I am trying to locate the source of a memory leak in my code. I
>> am using
>> stable (version 3) version of carpet and I noticed that the memory
>> usage
>> continuously increases with time. If I checkpoint and restart, then
>> the memory usage goes back to its original size, but then increases
>> with time.
>>
>> I attached the relevant sections of the par file. Any suggestion on
>> how I can
>> track down the source of this leak. I am running the code on a
>> cluster
>> consisting
>> of dual core AMD X86_64 procs with HTX Infiniband interconnect and
>> the
>> pathscale compiler.
>>
>> Thanks
>>
>> Yosef
>> ---------------------------------------------------------------------
>> ---
>>
>> #====================================================================
>> ==========
>>
>> ActiveThorns = "admanalysis admbase admcoupling admmacros
>> coordgauge spacemask StaticConformal boundary time cartgrid3d
>> ioutil iobasic nanchecker CoordBase aeilocalinterp SymBase Slab
>> SphericalSurface LocalReduce MoL Carpet CarpetInterp CarpetIOASCII
>> CarpetLib CarpetReduce CarpetSlab CarpetRegrid2 AHFinderDirect
>> SphericalSurface CarpetTracker CarpetIOHDF5 CarpetIOScalar"
>>
>> #--------------------------------------------------------------------
>> ----------
>>
>> # Run parameters
>> #--------------------------------------------------------------------
>> ----------
>>
>> # Grid
>>
>> mol::ode_method = "RK4"
>> mol::mol_intermediate_steps = 4
>> mol::mol_num_scratch_levels = 1
>>
>>
>> driver::global_nx = 101
>> driver::global_ny = 101
>> driver::global_nz = 101
>>
>>
>> grid::type = "byrange"
>> grid::xmin = -319.9875
>> grid::ymin = -319.9875
>> grid::zmin = -319.9875
>> grid::xmax = 320.0125
>> grid::ymax = 320.0125
>> grid::zmax = 320.0125
>>
>>
>>
>> grid::dxyz = 6.4
>> grid::domain = "full"
>>
>> SymHelp::grid_domain = "cartgrid3d"
>> SymHelp::grid_avoid_origin = "no"
>>
>>
>> driver::ghost_size = 3
>>
>> #--------------------------------------------------------------------
>> -------
>>
>> #cactus::cctk_itlast = 128
>>
>> CoordBase::xmin = -319.9875
>> CoordBase::ymin = -319.9875
>> CoordBase::zmin = -319.9875
>> CoordBase::xmax = 320.0125
>> CoordBase::ymax = 320.0125
>> CoordBase::zmax = 320.0125
>> CoordBase::dx = 6.4
>> CoordBase::dy = 6.4
>> CoordBase::dz = 6.4
>>
>> time::dtfac = .03125
>> Carpet::prolongation_order_space = 5
>> Carpet::prolongation_order_time = 2
>> Carpet::max_refinement_levels = 10 #5
>> Carpet::buffer_width = 3
>> Carpet::time_refinement_factors = "[1,1,1,1,2,4,8,16,32,64]"
>> Carpet::init_3_timelevels = "no"
>> Carpet::init_each_timelevel = "no"
>> Carpet::enable_all_storage = no
>> Carpet::regrid_during_recovery = no
>>
>> CarpetLib::fast_recomposing = yes
>>
>>
>> CarpetRegrid2::regrid_every = 32
>>
>> CarpetRegrid2::num_centres = 2
>>
>> CarpetRegrid2::num_levels_1 = 10
>> CarpetRegrid2::position_x_1 = 1.7604572
>> CarpetRegrid2::position_y_1 = 0.0
>> CarpetRegrid2::position_z_1 = 0.0
>> CarpetRegrid2::radius_1[ 1] = 160
>> CarpetRegrid2::radius_1[ 2] = 80
>> CarpetRegrid2::radius_1[ 3] = 50
>> CarpetRegrid2::radius_1[ 4] = 13
>> CarpetRegrid2::radius_1[ 5] = 6.5
>> CarpetRegrid2::radius_1[ 6] = 3.25
>> CarpetRegrid2::radius_1[ 7] = 1.625
>> CarpetRegrid2::radius_1[ 8] = .8125
>> CarpetRegrid2::radius_1[ 9] = .40625
>>
>> CarpetRegrid2::num_levels_2 = 10
>> CarpetRegrid2::position_x_2 = -4.7455652
>> CarpetRegrid2::position_y_2 = 0.0
>> CarpetRegrid2::position_z_2 = 0.0
>> CarpetRegrid2::radius_2[ 1] = 160
>> CarpetRegrid2::radius_2[ 2] = 80
>> CarpetRegrid2::radius_2[ 3] = 50
>> CarpetRegrid2::radius_2[ 4] = 13
>> CarpetRegrid2::radius_2[ 5] = 6.5
>> CarpetRegrid2::radius_2[ 6] = 3.25
>> CarpetRegrid2::radius_2[ 7] = 1.625
>> CarpetRegrid2::radius_2[ 8] = .8125
>> CarpetRegrid2::radius_2[ 9] = .40625
>>
>> admbase::metric_type = "Physical"
>>
>> #--- numerics
>>
>> # Output
>> #--------------------------------------------------------------------
>> ----------
>> #IO::new_filename_scheme = "no"
>> IO::out_fileinfo = "none"
>>
>>
>>
>>
>> CarpetIOASCII::out1D_every = 320
>> #CarpetIOASCII::out1D_every = 1
>> CarpetIOASCII::out1D_vars = "admbase::alp
>> admbase::metric admbase::curv"
>> ADMMacros::spatial_order=4
>> CarpetIOScalar::outScalar_vars = "admbase::alp"
>> CarpetIOScalar::outScalar_every = 320
>> CarpetIOScalar::outScalar_reductions = "minimum maximum"
>>
>> #====================================================================
>> ==========
>> NaNChecker::check_every = 16
>> NaNChecker::check_vars = "admbase::metric admbase::curv"
>> NaNChecker::action_if_found = "terminate"
>> NaNChecker::check_after=2
>>
>> cactus::terminate="runtime"
>> cactus::max_runtime=28800
>>
>>
>> SphericalSurface::nsurfaces = 3
>> SphericalSurface::maxntheta = 39
>> SphericalSurface::maxnphi = 76
>>
>> SphericalSurface::ntheta [0] = 39
>> SphericalSurface::nphi [0] = 76
>> SphericalSurface::nghoststheta[0] = 2
>> SphericalSurface::nghostsphi [0] = 2
>>
>> SphericalSurface::ntheta [1] = 39
>> SphericalSurface::nphi [1] = 76
>> SphericalSurface::nghoststheta[1] = 2
>> SphericalSurface::nghostsphi [1] = 2
>>
>> SphericalSurface::ntheta [2] = 39
>> SphericalSurface::nphi [2] = 76
>> SphericalSurface::nghoststheta[2] = 2
>> SphericalSurface::nghostsphi [2] = 2
>>
>>
>> AHFinderDirect::find_every = 64
>> AHFinderDirect::move_origins = yes
>>
>> AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial
>> interpolation"
>> AHFinderDirect::geometry_interpolator_pars = "order=3"
>> AHFinderDirect::surface_interpolator_name = "Lagrange polynomial
>> interpolation"
>> AHFinderDirect::surface_interpolator_pars = "order=3"
>>
>> AHFinderDirect::N_horizons = 3
>>
>> AHFinderDirect::origin_x [1] =
>> 1.7604572
>> AHFinderDirect::origin_y [1] = 0.0
>> AHFinderDirect::origin_z [1] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__x_center [1] =
>> 1.7604572
>> AHFinderDirect::initial_guess__coord_sphere__y_center [1] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__z_center [1] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__radius [1] = 0.3
>> AHFinderDirect::which_surface_to_store_info [1] = 0
>> AHFinderDirect::reset_horizon_after_not_finding [1] = no
>>
>> AHFinderDirect::origin_x [2] =
>> -4.7455652
>> AHFinderDirect::origin_y [2] = 0.0
>> AHFinderDirect::origin_z [2] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__x_center [2] =
>> -4.7455652
>> AHFinderDirect::initial_guess__coord_sphere__y_center [2] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__z_center [2] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__radius [2] = 0.3
>> AHFinderDirect::which_surface_to_store_info [2] = 1
>> AHFinderDirect::reset_horizon_after_not_finding [2] = no
>>
>> AHFinderDirect::origin_x [3] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__x_center [3] = 0.0
>> AHFinderDirect::initial_guess__coord_sphere__radius [3] = 2.5
>> AHFinderDirect::which_surface_to_store_info [3] = 2
>> AHFinderDirect::reset_horizon_after_not_finding [3] = no
>>
>>
>> CarpetTracker::surface[0] = 0
>> CarpetTracker::surface[1] = 1
>>
>> CarpetIOHDF5::checkpoint = "yes"
>> CarpetIOHDF5::use_reflevels_from_checkpoint = "yes"
>> CarpetIOHDF5::use_grid_structure_from_checkpoint = "yes"
>>
>> IO::checkpoint_ID = "no"
>> IO::checkpoint_every = 3250
>> IO::checkpoint_keep=2
>> IO::checkpoint_on_terminate = yes
>>
>>
>> IO::recover = "auto"
>>
>> ---------------------------------------------------------------------
>> ---
>>
>> _______________________________________________
>> developers mailing list
>> developers at lists.carpetcode.org
>> http://lists.carpetcode.org/listinfo/developers
>>
>
> _______________________________________________
> developers mailing list
> developers at lists.carpetcode.org
> http://lists.carpetcode.org/listinfo/developers
>
--
Erik Schnetter <schnetter at cct.lsu.edu>
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from www.keyserver.net.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/developers/attachments/20070710/9e5c20d3/attachment-0001.pgp
More information about the developers
mailing list