Connecting to Corepoint

[Bevan Richards]

I have an orders thread that goes to a Corepoint engine.

I am having difficulty keeping the thread running. It stays ‘UP’ but the orders backup up in my engine until I stop/start the thread. I only have this problem with this thread. The interface person on the Corepoint side reports the results thread act the same coming this way.

Does anybody have issues with keeping a thread running to a Corepoint?

Topic

[Jim Kosloskey]

Bevan,

Are the both of you connecting over a VPN with or without firewalls added?

If so this sounds like either or both could be the culprits.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

Through a VPN with firewalls. We just had a failure and I went on my engine box and did a tracert to their side and it completed just fine.

[Jim Kosloskey]

Bevan,

I am not sure that will indicate anything of value in this case.

The tracert utility uses port 7 I believe and I am fairly certain the connections you are concerned about use some other port.

The VPN and/or firewall could have the port you connected with dead but port 7 is unafected so tracert would work even if the port in question had an issue.

If the VPN/Firewall owners are unwilling or unable to assist you then you may need to investigate getting a sniffer to trace exactly what is occurring (to be certain both sides should snifgf at the same time).

There are some command line commands which can be used (I forget what they are off the top of my head other clovertechers will know those and hopefully will pipe up) which can tell you the state your port is in when the problem finally arises (I suspect that would show a FIN-WAIT-2 state or something like that) but that also does not actually point to the issue.

Really whoever is administering the VPN/Firewalls should step ujp and assist.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

I am certain the issue is on their side. They just reported that they are having a similar issue with another hospital that uses cloverleaf. They are having problems with hospitals that use the cloverleaf engine.

[Jim Kosloskey]

Hmmm if these are TCP/IP MLP connections it should not matter what the system is and especially not Cloverleaf since it is following the standard.

You are probably correct that the issue is with Corepoint but I suspect it is not specific to Cloverleaf – that is just smoke I would bet.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

As far as testing when this happens again, what if I telnet to the ip/port? Would that be a valid test (unlike the tracert).

[Jim Kosloskey]

Bevan,

Try hcitcptest.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

OK, I will look that up. Thanks.

[Bevan Richards]

I just tried hcitcptest from my test engine. I must be doing something wrong because I am getting a socket error. Here is what I am entering

hcitcptest -h {the target ip} -t raw -p {port #}

where {the target ip} is the ip of the server I am trying to connect to

and {port #} is the port on that server.

[Jim Kosloskey]

Bevan,

Is your host experiencing the issue you described? Is your Cloverleaf thread still on?

If so, you probably could get a connect error.

Try stopping your Cloverleaf connetction (thread), then retry the hcitcptest. If that does not work then the server to which you are attempting to connect is refusing the connection.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

Thanks, I stopped the entire process then reran the test on my test box. It connected so now, when I get a backup, I will stop the thread and run that test.

[Bevan Richards]

Had 2 failures last night. stopped the thread, ran the hcitpctest routine both times. Showed a connection. Restarted thread, messages passed through.

[Jim Kosloskey]

Bevan,

Are you waiting for replies? If so are you waiting forever or a specific time period?

If a specific time period, what do you do when the timeout occurs?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

Await replies, waiting forever

[Jim Kosloskey]

Bevan,

OK well I suspect you are waiting for a reply when the issue occurs.

So did you stop the Clovelreaf thread and then use hcitcptest or use hcitcptest without stopping the Cloverleaf thread?

Did you do this immedialtely after the issue was recognized or after some delay – if after some delay how long do you think?

This is just sounding more and more like a firewall/VPN issue and those folks need to investigate.

There are some workarounds which have limited success but they are band-aids – they do not fix the issue which should be repairable if the proper analysis is completed.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Bevan Richards]

I stopped the thread and did the hcitcptest. If I try it with the thread on, I get a socket error.

I have an alert that fires when there are more than 9 messages waiting to go out. I did the test as soon as I was notified of the alert.

I have started (this morning) saving the ACK’s they send to see if there is a correlation between waiting for an ACK and the messages stopping.

I really appreciate all the time and help you have given me on this.

[jigar mehta]

Bevan,

I think Jim is pointing in right direction earlier, we have issue with firewall/ VPN few years back (it was not cloverleaf related at all)

Let me ask you question.

Does data flow  stopped  (for not having any data to transfer or no more transaction from sending system) for period of time between cloverleaf and core point at any point preceding to issue?

We upgrade (& migrated to) our HIS system to HPUX and O/s stop sending Keep alive/ hearbeat signal when there is no traffic, so firewall was shutting down tunnel after 30 minutes and both client/ sender and receiver/llstener were up and no aware of firewall closing, so when next time data comes in, it will fail on sending end as firewall is closed and can not reach destination.

you might have totally different issue, but i just want to share our experiences.

Thank you,

Jigar

Bevan Richards wrote:

I stopped the thread and did the hcitcptest. If I try it with the thread on, I get a socket error.

I have an alert that fires when there are more than 9 messages waiting to go out. I did the test as soon as I was notified of the alert.

I have started (this morning) saving the ACK’s they send to see if there is a correlation between waiting for an ACK and the messages stopping.

I really appreciate all the time and help you have given me on this.

[Bevan Richards]

Thanks Jigar. I do not think it is the same issue. I have taken the ‘Wait for replay’ off this thread (not the best option, but it has stopped the calls at 2 in the morning). We have lost connection during a time when it is very busy and when it is quiet.

[Ed Mastascusa]

Hi Bev,

check and see if your OS tcp keepalive setting is longer than the firewall’s “session timeout”.  

We had a situation where the firewall (Fortigate or Fortinet) changed their “system session time” from 60 minutes to 5 – our networks group did this due to some concerns about a memory or handle resource issue. Our CL box had the tcp keepalive set to 15 minutes.  The day of the change we had numerous external VPNs  (including some where corepoint was on the other end) with hanging isues similar to what you describe. When the fortigate box changed from 5 to 20 minutes the problems stopped.

My understanding of the issue is that the firewall essentially triggered an uncontrolled shutdown of the VPN such that the OS keepalives never got through after the firewall timed out.

One stop-gap way to work around this is to pstop/pstart cycle your thread before any of the components (your side / the VPN / and the Corepoint side) time out.

[Bevan Richards]

OK Ed, I will check with the network people on both sides to see what the time out is.

Thanks

[Author]

Replies