RDM Semaphore/Mutex error – DBVista Error -924

[Lori Owen]

Our engine has crashed twice in 4 days. We are getting a few errors per below. We do a cleanup and the engine comes back up fine. Another post dealing with similar errors mentioned this could be caused by backup jobs. We have our sysadmins looking into this as well. Any thoughts? Thanks in advance!

pdl :PDL :ERR /0:int_mtadt_out:01/14/2014 02:30:57] read failed: Unknown error

[pdl :PDL :ERR /0:int_mtadt_out:01/14/2014 02:30:57] read returned error 0 (No error)

[pdl :PDL :ERR /0:int_mtadt_out:01/14/2014 02:30:57] PDL signaled exception: code 1, msg device error (remote side probably shut down)

[dbi :dbi :ERR /0:int_mtadt_out:01/14/2014 02:31:02] Db_Vista database error -924: ‘SYSTEM/OS error: -924

RDM Semaphore / Mutex error

C errno = 9: Bad file descriptor’

[pti :sign:WARN/0:int_mtadt_out:01/14/2014 02:31:02] Thread 35 received signal EXCEPTION_ACCESS_VIOLATION:

 The thread attempted to read from or write to a virtual address for which it does not have the appropriate access.

[pti :sign:WARN/0:int_mtadt_out:01/14/2014 02:31:03] PC = 0xffffffff

PANIC: “0”

PANIC: Calling “pti” for thread MEDITECH_cmd

---

Scheduler State

---

Thread Events     State      Priority Runnable  PT Msgs

  0      0   SCHED_IDLE         0       0       0,0,0

  1      0   SCHED_IDLE         0       0       0,0,0

  2      0   SCHED_IDLE         0       0       0,0,0

  3      0   SCHED_IDLE         0       0       0,0,0

  4      0   SCHED_IDLE         0       0       0,0,0

  5      0   SCHED_IDLE         0       0       0,0,0

  6      0   SCHED_IDLE         0       0       0,0,0

  7      0   SCHED_IDLE         0       0       0,0,0

  8      0   SCHED_IDLE         0       0       0,0,0

  9      0   SCHED_IDLE         0       0       0,0,0

 10      0   SCHED_IDLE         0       0       0,0,0

 11      0   SCHED_IDLE         0       0       0,0,0

 12      0   SCHED_IDLE         0       0       0,0,0

 13      0   SCHED_IDLE         0       0       0,0,0

 14      0   SCHED_IDLE         0       0       0,0,0

 15      0   SCHED_IDLE         0       0       0,0,0

 16      0   SCHED_IDLE         0       0       0,0,0

 17      0   SCHED_IDLE         0       0       0,0,0

 18      0   SCHED_IDLE         0       0       0,0,0

 19      0   SCHED_IDLE         0       0       0,0,0

 20      0   SCHED_IDLE         0       0       0,0,0

 21      0   SCHED_IDLE         0       0       0,0,0

 22      0   SCHED_IDLE         0       0       0,0,0

 23      0   SCHED_IDLE         0       0       0,0,0

 24      0   SCHED_IDLE         0       0       0,0,0

 25      0   SCHED_IDLE         0       0       0,0,0

 26      0   SCHED_IDLE         0       0       0,0,0

 27      0   SCHED_IDLE         0       0       0,0,0

 28      0   SCHED_IDLE         0       0       0,0,0

 29      0   SCHED_IDLE         0       0       0,0,0

 30      0   SCHED_IDLE         0       0       0,0,0

 31      0   SCHED_IDLE         0       0       0,0,0

 32      0   SCHED_IDLE         0       0       0,0,0

 33      0   SCHED_IDLE         0       0       0,0,0

 34      0   SCHED_IDLE         0       0       0,0,0

 35      0   SCHED_RUNNING  0       0       1,0,0

 36      0   SCHED_IDLE         0       0       0,0,0

 37      0   SCHED_IDLE         0       0       0,0,0

 38      0   SCHED_IDLE         0       0       0,0,0

 39      0   SCHED_IDLE         0       0       0,0,0

 40      0   SCHED_IDLE         0       0       0,0,0

 41      0   SCHED_IDLE         0       0       0,0,0

 42      0   SCHED_IDLE         0       0       0,0,0

Topic

[Terry Kellum]

Is this AIX or Windows?  Have you done an hcidbinit -AC ? (Warning…  This command can lose messages.)

[Lori Owen]

Hi,

We are on Windows 2003 – soon to be upgrading to 6.0 on Windows 2008. Can’t happen soon enough. We crashed again yesterday and I did an hcidbinit -AC during clean-up.

The only thing we have changed is adding two alerts to do a Stop/Start on two specific threads if we have had no activity for 30 minutes. These are VPN connections and we believe the vendor is not releasing our ports after sending messages. When they try to send the next message they get a socket error and the connection is not ‘opened’ until we bounce our threads. Sometimes bouncing their side of the VPN works as well. The vendor does not accept acks and does not log any of their messaging.

This is the only thing we can point to and we have Infor taking a look as well. We are thinking about setting the inbound threads up as multi-server so we can eliminate the alerts.

Thanks!

Lori

[Terry Kellum]

This may not be related, but it’s my understanding that when working on threads out of alerts, you have to “call” a tcl proc up that performs the work outside the alert itself.  My “Engine internals” memory is a bit fuzzy, but I think that a common mistake is to perform operations on the engine/thread from “inside” an existing TCL process in the engine.

i.e.  In the alert, you wouldn’t call “pstop, pstart” (pseudocode).  You would call /opt/mysite/myprograms/RestartMyThread.tcl  which would create it’s own interpreter and then pass in commands like “pstart, pstop”.

[Lori Owen]

Our alerts call a batch proc that contains the following commands…

hcicmd -p UCADT -c “docutap_adt_in pstop”

call wait 20

hcicmd -p UCADT -c “docutap_adt_in pstart”

We also had another site on our production server that had some duplicate thread names to our main site. We have removed the other site. We are kind of grasping at straws but hopefully will get this resolved soon  :D.

Thanks!

[Author]

Replies