Process won’t start

[Mark Brown]

This morning after I took down our test interface engine to initialize the database, I’m getting a couple of processes that won’t start back up. We have Cloverleaf 5.3.1 running on a Windows 2003 system.  This is part of the log file.  The directory it’s writing to has the exact same permissions set as all the working processes.  I’ve deleted the directory and let netconfig create a new directory.   I don’t know what to try next.  Anyone have any ideas?

[prod:prod:INFO/0:  STARTUP_TID] Copyright 1993-2003, Quovadx Inc.

[prod:prod:INFO/0:  STARTUP_TID] QDX(TM) Integration Services 5.3P Rev1

[prod:prod:INFO/0:  STARTUP_TID] Linked by michael_han on ROLEX at Wed Sep  8 08:58:49 2004 in C:/cloverrel/cloverleaf/engine/main (build 10)

[prod:prod:INFO/0:  STARTUP_TID] Started at Wed Sep 07 11:11:04 2005

[prod:prod:INFO/0:  STARTUP_TID] Engine process is 4812 on host srvie2

[prod:prod:INFO/1:  obhboc4_cmd] Msg space limit is 0

[prod:prod:INFO/0:  obhboc4_cmd] DiskQue Minimum # of Messages/que: 50

[prod:prod:INFO/0:  obhboc4_cmd] DiskQue Virtual Memory percent:75.000000

[prod:prod:INFO/0:  obhboc4_cmd] Applying EO config: ”

[prod:prod:INFO/0:obhboc4_xlate] Applying EO config: ”

09/07/2005 11:11:05

[dbi :dbi :ERR /0:       obhboc] Duplicate DB_VISTA user id: ‘testobhboc’

09/07/2005 11:11:05

[dbi :dbi :ERR /0:       obhboc] Duplicate DB_VISTA user id: ‘testobhboc’

09/07/2005 11:11:05

[dbi :dbi :ERR /0:       obhboc] Db_Vista database error -921: ‘SYSTEM/OS error: -921

DBUSERID is already being used

C errno = 0: No error’

09/07/2005 11:11:05

[pti :sign:WARN/0:       obhboc] Thread 2 received signal EXCEPTION_ACCESS_VIOLATION:

 The thread attempted to read from or write to a virtual address for which it does not have the appropriate access.

09/07/2005 11:11:05

[pti :sign:WARN/0:       obhboc] PC = 0xffffffff

PANIC: “0”

PANIC: Calling “pti” for thread obhboc4_cmd

Topic

[David Caragay]

I noticed the following in your process log….

[dbi :dbi :ERR /0:

[Mark Brown]

I tried that again and I get the same error message.    I  have two processess doing this.  The others come up just fine.

Here’s the last few lines of the log file if that helps anyone…

PANIC: Calling “dbi shutdown” for thread obhboc4_cmd

PANIC: Calling “dbi shutdown” for thread obhboc4_xlate

PANIC: Process panic—engine going down

PANIC: assertion ‘0’ failed at PthreadInterface.cpp/695

[Anonymous]

A little late here but you could try ‘hcilmclear’.

When I run it I get:

c:temp>hcilmclear -p obhboc -v

Clearing db user “obhboc_cmd”

Clearing db user “obhboc_xlate”

[Jonathan Christian]

I hate to ressurect old threads like this, but this is the exact same thing happening to us right now. Messages had been processing fine for weeks without issue and then suddenly stopped. Below is a copy of the output. Almost exactly the same as the one above.

Any ideas?

[prod:prod:INFO/0:

[Tom Rioux]

Did you try Greg Day’s suggestion?

[Gary Atkinson]

The information I have is for windows to shutdown the process, daemons and restart QDX service.

[Jonathan Christian]

Hello All,

I’ve run through all of the suggested solutions, with no success yet. We’ve even bounced the server to make sure we’ve covered all of our bases. It only seems to be occurring with this one process.

Additionally, I’ve dug a bit deeper and have found the following that might be of use as well…

Reported at: Fri Oct 12 09:26:32 CDT 2007

Status

[Michael Hertel]

Are all of those failed messages really in your error database?

Do you ever purge your error database?

I’d shut down, do an hcidbinit -AC, then hcimsiutil -Z and start again.

[Jonathan Christian]

Eureka! That did the trick. However, I feel that I’ve found the root culprit in the following error.

Proto Err: Fri Oct 12 12:07:35 CDT 2007

Error Msg: cannot open ……datainbound/OBGYN_CAD_db_20071004.BAK for reading: No

This error (and others similar to it) have cropped up before during testing and development of this interface in particular, but they haven’t blocked the flow of data into Xlates. Now though, the process is back up and running, but the inbound thread is still showing the preceding error.

We’re still pretty new at trying to troubleshoot these issues, so we haven’t really gotten our “sea legs” yet.

Thank you all, again, for your input and suggestions. They are greatly appreciated.

[David Burks]

Posting to this older thread because someone saw this and thought they were having same problem.  I wanted to clarify some things.  The relevant error for the original poster is this one.

[dbi :dbi :ERR /0:       obhboc] Db_Vista database error -921: ‘SYSTEM/OS error: -921

Now the following error message “EXCEPTION_ACCESS_VIOLATION” was in the log but was simply a consequence of the 921.  There are many things that can cause an EXCEPTION_ACCESS_VIOLATION that have nothing to do with a 921.  Usually that is either bad data or permissions caused by some other software or program accessing the cloverleaf files while cloverleaf is writing to them.  

For the 921 what you do depends on your OS.  First we will assume this was on a unix box.  We look and see what the duplicate user was.

[dbi :dbi :ERR /0:       obhboc] Duplicate DB_VISTA user id: ‘testobhboc’

The following command could have been used.

lmclear -u testobhboc -mp

Alternatively if this does not work the following command could be used

hcilmclear -p processname

where processname is the name of the process that is having a problem.  

Now if you are on windows the lmclear command will not work because there is no lock manager to process the command.  So when you get the dbvista -921 on windows you have to stop all engines on all sites and all daemons on all sites and then bounce the cloverleaf service.  I usually hit task manager and go to the processes tab to verify there is nothing hci still left running.  If possible the service should not be stopped while any engines or daemons are running.  The exact name of the service varies slightly on each cloverleaf version so if in doubt you may need to call support to be certain.  

So in summary the EXCEPTION_ACCESS_VIOLATION may be very important when not preceded by the 921 but when the two are found together deal with the 921 and the EXCEPTION_ACCESS_VIOLATION should disappear since it is 99% probably caused by the real 921 problem.

Hopefully this clears this up a little and does not confuse anyone more.

[Bala Pisupati]

Try stopping the lock monitor and monitor deamon, and rebooted the server. Also if this doesn’t work you can do clover leaf cleanup .

1.shutdown the site – hcisitectl -K -f

2. Remove the CMD and PID file in

[Author]

Replies