[Carpet] Bugzilla Bug 118

[Luca Baiotti]

Luca Baiotti wrote:
> Hallo Erik, hallo Thomas, hallo everyone.
> 
> Is there any news about Carpet Bugzilla Bug 118 filed by Bela long ago?
> 
> I might have encountered it again, in a simulation that ran for a few 
> weeks and had performed > 10^5 iterations. The run had recently 
> recovered, ~300 iterations before this fatal error:
> 
> INFO (CarpetRegrid2): Centre 0 is at position [0,0,0]
> INFO (CarpetRegrid2): Centre 1 is at position [0,0,0]
> INFO (CarpetRegrid2): Centre 2 is at position [0,0,0]
> INFO (CarpetRegrid2): Regridding
> INFO (Carpet): Grid structure statistics:
> INFO (Carpet): GF: rhs: 371k active, 485k owned (+31%), 843k total 
> (+74%), 46.9 steps/time
> INFO (Carpet): GF: vars: 85, pts: 263M active, 330M owned (+25%), 568M 
> total (+72%), 1.0 comp/proc
> INFO (Carpet): GA: vars: 581, pts: 3M active, 3M total (+0%)
> INFO (Whisky): Setting up the atmosphere mask: all points are 
> not_atmosphere
> INFO (Whisky): Setting up the atmosphere mask: all points are 
> not_atmosphere
> INFO (Whisky): Setting up the atmosphere mask: all points are 
> not_atmosphere
> WARNING level 0 in thorn CarpetLib processor 0 host node0052.admin
>   (line 257 of 
> /data1/baiotti/Cactus/configs/belladonna_NoDebug/build/CarpetLib/gdata.cc):
>   -> Internal error: extrapolation in time.  time=3153 
> times=[3153,3152.94,3152.87]
> 
> 
> Actually this might be a different problem. Of course I am producing 
> output with verbose=yes, but debug information will require more time, 
> since the machines I am using have problems running with DEBUG = yes.
> So I first wanted to know whether there is any new info about this.
> 
> 
> While I was writing this email, the run with increased verbosity 
> finished. You can find the 49Mb output file on belladonna in 
> /home/baiotti/tmp

Hallo,

about the above problem, I produce readable corefiles by compiling with 
DEBUG=yes and OPTIMISE=no. The readable data are on peyote in

/data5/baiotti/Time-error

However, this is what gdb has top say:

 > gdb cactus_carpet_experimental_DEBUG_NoOpt 
core-ic0092-cactus_carpet_e-15087
GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host 
libthread_db library "/lib/libthread_db.so.1".


warning: core file may not match specified executable file.
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0xffffe000
Core was generated by `cactus_carpet_experimental_DEBUG_NoOpt 
-parameter-level=strict Meudon-moving-6l'.
Program terminated with signal 6, Aborted.

warning: svr4_current_sos: Can't read pathname for load map: 
Input/output error

Reading symbols from /opt/intel/9.0-c027-f028/lib/libimf.so...done.
Loaded symbols for /opt/intel/9.0-c027-f028/lib/libimf.so
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /opt/intel/mkl70/lib/32/libmkl_def.so...done.
Loaded symbols for /opt/intel/mkl70/lib/32/libmkl_def.so
Reading symbols from /opt/intel/9.0-c027-f028/lib/libguide.so...done.
Loaded symbols for /opt/intel/9.0-c027-f028/lib/libguide.so
Reading symbols from /usr/lib/libgsl.so.0...done.
Loaded symbols for /usr/lib/libgsl.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libutil.so.1...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /lib/libssl.so.5...done.
Loaded symbols for /lib/libssl.so.5
Reading symbols from /lib/libcrypto.so.5...done.
Loaded symbols for /lib/libcrypto.so.5
Reading symbols from /usr/lib/libgslcblas.so.0...done.
Loaded symbols for /usr/lib/libgslcblas.so.0
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /opt/intel/9.0-c027-f028/lib/libifcore.so.5...done.
Loaded symbols for /opt/intel/9.0-c027-f028/lib/libifcore.so.5
Reading symbols from /opt/intel/9.0-c027-f028/lib/libsvml.so...done.
Loaded symbols for /opt/intel/9.0-c027-f028/lib/libsvml.so
Reading symbols from /opt/gcc/3.4.4/lib/libstdc++.so.6...done.
Loaded symbols for /opt/gcc/3.4.4/lib/libstdc++.so.6
Reading symbols from /opt/gcc/3.4.4/lib/libgcc_s.so.1...done.
Loaded symbols for /opt/gcc/3.4.4/lib/libgcc_s.so.1
Reading symbols from /opt/intel/9.0-c027-f028/lib/libcxaguard.so.5...done.
Loaded symbols for /opt/intel/9.0-c027-f028/lib/libcxaguard.so.5
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
#0  0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb73c7f90 in raise () from /lib/libc.so.6
#2  0xb73c9678 in abort () from /lib/libc.so.6
#3  0xb73c1269 in __assert_fail () from /lib/libc.so.6
#4  0x092511e9 in Carpet::Abort (cgh=0x0, retval=0)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/helpers.cc:150
#5  0x0805a538 in CCTK_VWarn (level=0, line=257,
     file=0x9c7f760 
"/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/gdata.cc", 
thorn=0x9c7fab0 "CarpetLib",
     format=0x9b283f0 "%s") at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/WarnLevel.c:646
#6  0x0805a00a in CCTK_Warn (level=0, line=257,
     file=0x9c7f760 
"/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/gdata.cc", 
thorn=0x9c7fab0 "CarpetLib",
     message=0xd18841c "Internal error: extrapolation in time. 
time=3153  times=[3153,3152.94,3152.87]")
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/WarnLevel.c:437
#7  0x092be5f2 in destr_detour24 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/gdata.cc:257
#8  0x092bd3c7 in gdata::transfer_from (this=0x303a0c70, 
state=@0xbfd60b18, srcs=@0xbfd60908, times=@0xbfd609b4, dstbox=@0xc46b0a8,
     srcbox=@0xc46b080, time=3153.0000000000009, order_space=3, 
order_time=2)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/gdata.cc:127
#9  0x092c5f8b in destr_detour164 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/ggf.cc:517
#10 0x092c4760 in destr_detour154 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CarpetLib/ggf.cc:340
#11 0x091f1218 in destr_detour15 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Comm.cc:198
#12 0x091f0acb in destr_detour14 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Comm.cc:161
#13 0x091f0001 in destr_detour12 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Comm.cc:53
#14 0x08071a63 in CactusDefaultSyncGroup (GH=0xd3bf658, 
groupname=0xc52b948 "PsiKadelia::WeylComponents")
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/comm/CactusDefaultComm.c:505
#15 0x0805dad6 in cctk_syncgroup_ (ierror=0xbfd61098, GH=0xbfd620fc, 
cctk_str1=0x9bdf6d4 "PsiKadelia::WeylComponents", cctk_strlen1=26)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/comm/OverloadComm.c:199
#16 0x088bde3b in psikadelia_boundaries (cctk_dim=3, cctk_gsh=0xd3bf6c8, 
cctk_lsh=0xd3bf6d8, cctk_lbnd=0xd3bf6e8, cctk_ubnd=0xd3bf6f8,
     cctk_lssh=0xcc82a58, cctk_from=0xd3bf718, cctk_to=0xd3bf708, 
cctk_bbox=0xd3bf728, cctk_delta_time=1.3439999999999999,
     cctk_time=4237.6320000000005, cctk_delta_space=0xcc1a500, 
cctk_origin_space=0xcc1a520, cctk_levfac=0xcc206e8, cctk_levoff=0xcc26898,
     cctk_levoffdenom=0xd3bf748, cctk_timefac=32, cctk_convlevel=0, 
cctk_convfac=2, cctk_nghostzones=0xcc206c8, cctk_iteration=100897,
     cctkgh=222033496, x0ijinvariants=0, x0riccicomponents=0, 
x0ricciscalar=0, x0weylcomponents=0, x1ijinvariants=0, x1riccicomponents=0,
     x1ricciscalar=0, x1weylcomponents=0, x2ijinvariants=0, 
x2riccicomponents=0, x2ricciscalar=0, x2weylcomponents=0, icurvim=0x0,
     icurvim_p=0x0, icurvim_p_p=0x0, icurvre=0x0, icurvre_p=0x0, 
icurvre_p_p=0x0, jcurvim=0x0, jcurvim_p=0x0, jcurvim_p_p=0x0, jcurvre=0x0,
     jcurvre_p=0x0, jcurvre_p_p=0x0, psi0im=0x0, psi0im_p=0x0, 
psi0im_p_p=0x0, psi0re=0x0, psi0re_p=0x0, psi0re_p_p=0x0, psi1im=0x0,
     psi1im_p=0x0, psi1im_p_p=0x0, psi1re=0x0, psi1re_p=0x0, 
psi1re_p_p=0x0, psi2im=0x0, psi2im_p=0x0, psi2im_p_p=0x0, psi2re=0x0,
     psi2re_p=0x0, psi2re_p_p=0x0, psi3im=0x0, psi3im_p=0x0, 
psi3im_p_p=0x0, psi3re=0x0, psi3re_p=0x0, psi3re_p_p=0x0, psi4im=0x0,
     psi4im_p=0x0, psi4im_p_p=0x0, psi4re=0x0, psi4re_p=0x0, 
psi4re_p_p=0x0, ref_levels_used=0, ricscal=0x0, ricscal_p=0x0,
     ricscal_p_p=0x0, ricxx=0x0, ricxx_p=0x0, ricxx_p_p=0x0, ricxy=0x0, 
ricxy_p=0x0, ricxy_p_p=0x0, ricxz=0x0, ricxz_p=0x0, ricxz_p_p=0x0,
---Type <return> to continue, or q <return> to quit---
     ricyy=0x0, ricyy_p=0x0, ricyy_p_p=0x0, ricyz=0x0, ricyz_p=0x0, 
ricyz_p_p=0x0, riczz=0x0, riczz_p=0x0, riczz_p_p=0x0, x0confac=0,
     x0confac_1derivs=0, x0confac_2derivs=0, x0coordinates=0, x0curv=0, 
x0lapse=0, x0metric=0, x0shift=0, x1confac=0, x1confac_1derivs=0,
     x1confac_2derivs=0, x1coordinates=0, x1curv=0, x1lapse=0, 
x1metric=0, x1shift=0, x2confac=0, x2confac_1derivs=0, x2confac_2derivs=0,
     x2coordinates=0, x2curv=0, x2lapse=0, x2metric=0, x2shift=0, 
alp=0x0, alp_p=0x0, alp_p_p=0x0, betax=0x0, betax_p=0x0, betax_p_p=0x0,
     betay=0x0, betay_p=0x0, betay_p_p=0x0, betaz=0x0, betaz_p=0x0, 
betaz_p_p=0x0, coarse_dx=3.8399999999999999,
     coarse_dy=3.8399999999999999, coarse_dz=3.8399999999999999, 
conformal_state=0, gxx=0x0, gxx_p=0x0, gxx_p_p=0x0, gxy=0x0, gxy_p=0x0,
     gxy_p_p=0x0, gxz=0x0, gxz_p=0x0, gxz_p_p=0x0, gyy=0x0, gyy_p=0x0, 
gyy_p_p=0x0, gyz=0x0, gyz_p=0x0, gyz_p_p=0x0, gzz=0x0, gzz_p=0x0,
     gzz_p_p=0x0, kxx=0x0, kxx_p=0x0, kxx_p_p=0x0, kxy=0x0, kxy_p=0x0, 
kxy_p_p=0x0, kxz=0x0, kxz_p=0x0, kxz_p_p=0x0, kyy=0x0, kyy_p=0x0,
     kyy_p_p=0x0, kyz=0x0, kyz_p=0x0, kyz_p_p=0x0, kzz=0x0, kzz_p=0x0, 
kzz_p_p=0x0, psi=0x0, psix=0x0, psixx=0x0, psixy=0x0, psixz=0x0,
     psiy=0x0, psiyy=0x0, psiyz=0x0, psiz=0x0, psizz=0x0, r=0x0, 
shift_state=1, x=0x0, y=0x0, z=0x0)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/PsiKadelia/PsiKadelia.f:1764
#17 0x08103a0a in CCTKi_BindingsFortranWrapperPsiKadelia (_GH=0xd3bf658, 
fpointer=0x88bb3e2)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/CactusBindings/Variables/PsiKadelia.c:40
#18 0x080560c5 in CCTK_CallFunction (function=0x88bb3e2, 
fdata=0xd3d7884, data=0xd3bf658)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/ScheduleInterface.c:272
#19 0x09262996 in destr_detour42 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/CallFunction.cc:261
#20 0x080589a6 in CCTKi_ScheduleCallFunction (function=0x88bb3e2, 
attribute=0xd3d7878, data=0xbfd62dd8)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/ScheduleInterface.c:2480
#21 0x0805becb in ScheduleTraverseFunction (function=0x88bb3e2, 
attributes=0xd3d7878, n_whiles=0, whiles=0x0,
     item_entry=0x8058530 <CCTKi_ScheduleCallEntry>, item_exit=0x8058716 
<CCTKi_ScheduleCallExit>,
     while_check=0x80588da <CCTKi_ScheduleCallWhile>, 
function_process=0x8058964 <CCTKi_ScheduleCallFunction>, data=0xbfd62dd8)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/schedule/ScheduleTraverse.c:488
#22 0x0805bc74 in ScheduleTraverseGroup (schedule_groups=0xd395ef0, 
group=0xd4056b8, attributes=0x0, n_whiles=0, whiles=0x0,
     item_entry=0x8058530 <CCTKi_ScheduleCallEntry>, item_exit=0x8058716 
<CCTKi_ScheduleCallExit>,
     while_check=0x80588da <CCTKi_ScheduleCallWhile>, 
function_process=0x8058964 <CCTKi_ScheduleCallFunction>, data=0xbfd62dd8)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/schedule/ScheduleTraverse.c:309
#23 0x0805baaf in CCTKi_DoScheduleTraverse (group_name=0x9c69c88 
"CCTK_POSTREGRID", item_entry=0x8058530 <CCTKi_ScheduleCallEntry>,
     item_exit=0x8058716 <CCTKi_ScheduleCallExit>, while_check=0x80588da 
<CCTKi_ScheduleCallWhile>,
     function_process=0x8058964 <CCTKi_ScheduleCallFunction>, 
data=0xbfd62dd8)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/schedule/ScheduleTraverse.c:144
#24 0x08057502 in ScheduleTraverse (where=0x9c69c88 "CCTK_POSTREGRID", 
GH=0xd3bf658,
     CallFunction=0x925fb84 <Carpet::CallFunction(void*, cFunctionData*, 
void*)>)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/ScheduleInterface.c:1206
#25 0x0805674f in CCTK_ScheduleTraverse (where=0x9c69c88 
"CCTK_POSTREGRID", GH=0xd3bf658,
     CallFunction=0x925fb84 <Carpet::CallFunction(void*, cFunctionData*, 
void*)>)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/ScheduleInterface.c:776
#26 0x091f57a9 in destr_detour33 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Evolve.cc:554
#27 0x091f3b20 in destr_detour23 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Evolve.cc:307
---Type <return> to continue, or q <return> to quit---
#28 0x091f2017 in destr_detour20 () at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Carpet/Evolve.cc:70
#29 0x08051c4a in main (argc=4, argv=0xbfd63504)
     at 
/data1/baiotti/Cactus/configs/carpet_experimental_DEBUG_NoOpt/build/Cactus/main/flesh.cc:84



Let me know,

Luca