[Ed Mastascusa]

you could try the “prestart” in place of the pstop then no need for the pstart – I think prestart is just a pstop on the front end anyway –  just make sure your time param is fairly  large, if you think 5 go with 10.

[Ed Mastascusa]

since it sounds pretty repeatable you can always run hcitcptest and send a saved message. hcitcptest  displays everything received in hex.

$ hcitcptest -t mlp -h qted -p 5309

Data recieved from the host will be printed to standard

output in a hex dump format. (Including any encoding chars,

data is dumped exactly as read.)

If you wish to *send* data, just type it in and press return

If you wish to *send* a file use rd_filefilename and press return

Type C to quit

0x00000000 (00000)   6a656e6e 79204920 676f7420 796f7572   jenny.I.got.your

0x00000010 (00016)   206e756d 6265720a 00000000 00000000   .number.

Client closed connection

[Ed Mastascusa]

Hi mitesh –

for the stats it sounds like the hcimsitutil command would have the info you want. hcimsiutil -dd threadName will give you a bunch of information about thread activity.  there are also tcl commands (msiAttach, msiTocEntry, msiGetStatSample) that you can use to get a keyed list of the thread statistics that appear in the hcimsiutil command.

[Ed Mastascusa]

Hi Kumar –

I think you can infer from the line feed in the “unknown segment” messages (the 3 chars between the ‘ ‘s ) that in the bad message there is a line feed char after each segment delimiter. That out of order segment message takes the first 3 chars after the segment delimiter as the next “segment header” (as it should)

Turning up the engine output as was previously suggested can definitely prove or disprove that.

If the vendor can’t fix the issue you can do something pre-xlate in a tps proc.

[Ed Mastascusa]

Hello Danny,

If I read your second post right, you want to exec a SUPPRESS when PV1:10 is NOT one of {OBS,NRY,OOB,OBV}  so you want to “AND” (&&) and not OR (||) your individual “ne” conditions in the original “IF” statement like this:

IF   0(0).PV1(0).#10(0)  ne =OBS &&  0(0).PV1(0).#10(0)  ne =NRY &&  0(0).PV1(0).#10(0)  ne =OOB &&  0(0).PV1(0).#10(0) ne =OBV

  SUPPRESS

(no else)

re the echo statement, just make a COPY operation like Thomas suggested and put it above the suppress inside the if.

source: 0(0).PV1(0).#10(0)

dest: @scratchVar

in the “pre-proc” make sure TCL is chosen and put in this code snippet as a starting point. once you get comfortable with TCL and the xlt parameters you can get more complicated than this.

set temp [lindex $xlateInVals 0]

echo “suppressing message: PV1:10 is “$temp””

When you run the xlate tester then when the copy statement containg this pre-proc execs you’ll see the output and when you run for real it’ll  show up in the process log file.

[Ed Mastascusa]

Hello Danny – That syntax seems OK but the boolean logic in that statement is going to be true 100% of the time.  if you want to execute the “if” when PV1:10 is one of {OBS,NRY,OOB, OBV}, then the “ne”s should be “eq”s  or if you want the opposite then you need to change your “||”s to “&&”s

[Ed Mastascusa]

I recollect we had a similar problem (3.8-5.5 on AIX) where the cause was the host server. As a result we always kill our host server on our prod box unless we need it running in order to use the GUI. maybe stop/start that (hciss -k h) and see what happens.

if you search for “inrecoverable socket error” there are different ways that the monitor daemons can get slowed down (i.e. 10s of thousands of messages in the recovery DB) and there are a reasonable number of posts on this topic.

[Ed Mastascusa]

Hi Tim,

I wasn’t able to open the doc but the proc is not yet available because

[Ed Mastascusa]

Hello Peter,

Our IDN recently went live with Soarian on 9/15. We did not purchase openlink, so I can attest to some of the issues encountered receiving HL7 2.X ADT with about 30 downstream systems.  

1) Dynamic Delimiters. This one was my pet peeve. Soarian HL7 has an unusual approach to the implementation of characters that would otherwise be escaped.  In their messages they default to the standard delimiters, but if some of the message data happens to contain a delimiter, they do not encode the data with an escape sequence. Soarian has a list of “dynamic” delimiters and they try to pick delimiters from the list such that the delimiters do not conflict with the data.  So the delimiters will change from message to message depending on the message content.  Cloverleaf can handle this but many of our downstream ADTs weren’t dynamic enough.

To resolve this we wrote a tps proc to identify messages using the dynamic delimiter “feature”, then parse every field, locate the fields with the original standard delimiters, do the appropriate escape sequences, then reassemble the messages using standard delimiters. (I’ll post the proc at some point if anyone wants it)  

The list of delimiters is 16 long, and if you happen to deliberately test to see what happens when all the delimiters are used up, the outbound ADT is unable to send.  I should also add that given the large number of possible delimiter combinations you can’t test your routine with a huge deal of confidence.

2) In many of the “code set” type fields HL7 width conventions were totally ignored. Again, cloverleaf could handle the issue with variant changes but many of the downstream systems had truncation issues. In many cases we ended up mapping the Soarian codes to shorter mnemonics in cloverleaf.

3) This one is more of a business process issue, but our old system was all upper case and soarian allows mixed case. We ended up forcing a number of fields to be upper case in the interface several weeks after go live.

4) There are obviously going to be a lot of business process related issues. we had filter out a number of messages based on “non current” enounter indicators in the PV1:40. The Soarian concept of an “observation” patient being different from the business caused some issues for us.

5) in many of the ID list fields you may find you need to sort or toss out much of the data as lists are sent that include expired matter – we generally did this via tps procs.

feel free to contact me with specific questions

[Ed Mastascusa]

Hi Vince,

I’m presuming you’re on some unix variant based on your path – we’re on AIX.

We once hit the process status limit on the “hci” user and had problems logging in although that was not the exact error message we saw – we got some fork failure most of the time.  

root cause for us was that some OS setting that limited the number of PIDs per user had to be bumped up.  It almost looks like you hit the limit on total PIDs.

If you can telnet or have an open telnet session you can Try and turn off  some less important sites/ processes and see if the error goes away then comes back when you turn things back on.

you can also check if ” ps -ef | grep “hci ” | wc ”   indicates close to the process PID limit (default for us was 128 if I recall correctly)

[Ed Mastascusa]

Hi Vince,

check and see if your OS tcp “keepalive” setting is longer than the  firewall’s “session timeout”. (Or the shortest session timeout if there are firewalls on both ends) If the firewall times out the session that could collapse the tunnel in such a way that one side still thinks they are connectd.  

If the other side sees an “up” connection that is a symptom. If the problem only happens when things aren’t particularly busy that’s another.

If you have someone cooperating on the other end you might be able to test with hcitcptest on another port. start hcitcptest and let it sit with no data, then see if the connection breaks after a consistent interval. Then compare it to your keepalive setting and if its less than that the firewall is probably killing the connection before the keep alive happens.

[Ed Mastascusa]

Hi Bev,

check and see if your OS tcp keepalive setting is longer than the firewall’s “session timeout”.  

We had a situation where the firewall (Fortigate or Fortinet) changed their “system session time” from 60 minutes to 5 – our networks group did this due to some concerns about a memory or handle resource issue. Our CL box had the tcp keepalive set to 15 minutes.  The day of the change we had numerous external VPNs  (including some where corepoint was on the other end) with hanging isues similar to what you describe. When the fortigate box changed from 5 to 20 minutes the problems stopped.

My understanding of the issue is that the firewall essentially triggered an uncontrolled shutdown of the VPN such that the OS keepalives never got through after the firewall timed out.

One stop-gap way to work around this is to pstop/pstart cycle your thread before any of the components (your side / the VPN / and the Corepoint side) time out.

[Ed Mastascusa]

duh – found the issue with this –

we had an XLT file with an incomplete condition line in an IF.

   { COND { 0(0).MSH.#9.[1] eq =A02 &&  1(0).1(0).PV1.#10  eq  } }

should have had a value tot he right of the a second eq.

hcixlttest on just the xlt file didn’t indicate any syntax error/failure to load, although once a message occurred (or was run through the tester) we immedaitely got the same assertion failure.

[Ed Mastascusa]

Anyone ever see this one (same file, different line)?  TIA

PANIC: assertion ‘vs != ((struct ValueStack *) 0) && val != ((struct Value *) 0) && (val)->val_next == ((struct Value *) 0)’ failed at Value.cpp/958

[Ed Mastascusa]

Hi Gerand,

search on the assertion that failed (“assertion ‘0’ failed at PthreadInterface.cpp/699”). “PthreadInterface and 699” gets about 5 hits. I can’t tell you exactly what causes the initial panic but stopping the site and restarting cleared it up for us once.

[Author]

Replies