all test and production sites crashed
- This topic has 11 replies, 7 voices, and was last updated 11 years, 8 months ago by
.
-
Topic
-
We had an unusual occurance yesterday. All engine processes and the monitor daemon PANIC’ed yesterday afternoon. We have 2 sites on each of our test and production machines. All the engine processes on the test machine went down at 14:31 and all engine processes on the production machine went down at 14:46. The test machine is on AIX 5.2 and Cloverleaf 5.4.1 and the production machine is on AIX 5.1 and Cloverleaf 5.2 . Here’s an example of what showed on the process log: [pti :sign:WARN/0:test2_ps03_cmd:02/13/2007 14:31:58] Thread 0 received signal 11
[pti :sign:WARN/0:test2_ps03_cmd:02/13/2007 14:31:58] PC = 0x103c5174
PANIC: “0”
PANIC: Calling “pti” for thread test2_ps03_cmd
and at the end of the log:
PANIC: Calling “dbi shutdown” for thread test2_ps03_cmd
PANIC: Calling “dbi shutdown” for thread test2_ps03_xlate
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
PANIC: Calling “dbi shutdown” for thread algcer_oi_3
PANIC: Calling “dbi shutdown” for thread algcer_oo_3
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
[dbi :dbi :WARN/0: ceralg_oi_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
[dbi :dbi :WARN/0: ceralg_oo_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
[dbi :dbi :WARN/0: ceralg_oi_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
[dbi :dbi :WARN/0: ceralg_oo_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
[dbi :dbi :WARN/0: ceralg_oi_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
[dbi :dbi :WARN/0: ceralg_oo_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
[dbi :dbi :WARN/0: ceralg_oi_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
[dbi :dbi :WARN/0: ceralg_oo_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oi_3
[dbi :dbi :WARN/0: ceralg_oi_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Calling “dbi shutdown” for thread ceralg_oo_3
[dbi :dbi :WARN/0: ceralg_oo_3:02/13/2007 14:31:58] NULL DTD when closing DBI
PANIC: Process panic—engine going down
PANIC: assertion ‘0’ failed at PthreadInterface.cpp/699
We did bring our production machine back up after 15 minutes and we crossed our fingers. It stayed up and has ever since.
The only thing that was going on was our system admin was attempting to load a new version of the OS (5.2) onto a split off mirror image of the OS disk. But this should not have had any effect and definitely have no effect on our test machine. I will attach the full log of the test process that I quoted from above.
- The forum ‘Cloverleaf’ is closed to new topics and replies.
