Outbound thread using TCP/IP help

[Gary Atkinson]

I have an outbound thread for hl7 ORM’s using TCP/IP through a VPN line.  I see the message leave the engine, the write time is updated, but I do not see a Read time or a message coming in.  The vendor says they are not getting the message.  What I am saying is I am sending the message, but I am not getting an acknowledgment.  The outbound thread is in a UP status.  Both the recovery and error databases are empty.  I don’t have any recovery procs set-up (I working on this  😉 ).  I checked the log and the engine just says its idle.  What else should I look at?

Topic

[Max Drown (Infor)]

Try telneting to the ip:port of the receiving app and see what happens.

-- Max Drown (Infor)

[Mary Kobis]

Gary,

It maybe something in your VPN connection. You’ll have to have your Network Grus look into it. You can start by trying to PING the address you are trying to connect to… Firewalls on both sides also cause lots of problems if not configured properly….

Hope it helps, Mary.

[Gary Atkinson]

When I try to telnet, I get a unkown host.  I am able to ping the IP address.  

If I turn on enable-all in the thread, what should I look for in the log?

[Michael Hertel]

Before spending a lot of time on this, I’d configure the recover procs.

It doesn’t take that long and could save some headache.

[John Hamilton]

If you say you see the engine send it then it went out the socket.

I would confirm the message encapulation. Many times in my past this has happend where the vendor is expecting something other the the standard MLP.

[Gary Atkinson]

This morning I checked the outbound thread and see 10 messages came in, but 14 were sent out.  I did a netstat and could see an established tcp4 connection.  The thread was in a UP status, but messages were pending in state 11 in the recovery database.  When I restarted the thread, the pending messages eventually left the engine.  I don’t have the recovery procs added in yet, but I found a few versions in this forum I would like to try.  Also, I am using the standard mlp.

[Gary Atkinson]

I added in the recovery procs.

[Gary Atkinson]

Now that I have recovery procs set-up, this is what I am seeing in the log:

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 14:29:35] read failed: Connection timed out

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 14:29:35] read returned error 78 (Connection timed out)

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 14:29:35] PDL signaled exception: code 1, msg device error (remote side probably shut down)

Engine idle — 03/26/2008 14:29:50

Engine idle — 03/26/2008 15:40:22

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 1 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:42:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 2 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:42:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 3 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:43:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 4 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:43:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 5 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:44:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 6 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:44:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 7 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:45:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 8 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:45:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 9 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:46:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 10 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:46:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 11 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:47:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 12 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:47:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 13 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:48:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 14 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:48:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 15 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:49:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 16 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:49:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 17 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:50:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 18 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:50:45 EDT 2008

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 15:50:53] read failed: Connection timed out

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 15:50:53] read returned error 78 (Connection timed out)

[pdl :PDL :ERR /0:    Nighthawk:03/26/2008 15:50:53] PDL signaled exception: code 1, msg device error (remote side probably shut down)

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 19 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:51:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 20 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:51:45 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 21 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:52:15 EDT 2008

RESEND_OB_MSG_38/Nighthawk/reply_gen:     Resend 22 of saved msg with MID ‘0.0.16989925’ at Wed Mar 26 15:52:45 EDT 2008

I have await replies set to 30 seconds

[Gary Atkinson]

Hi-

I found a pdl that had the following settings:

hci_pd_msg_style basic  phrase:basic-msg

                       field:data

                       resync:\xb

                       timeout:60000

I used this pdl and turned eo config to PDL enable-all.  In the recovery database it had a message in state 14.  I started up the thread and viola it was sent, a reply was received and state 14 was removed.  I need to try more messages, as I am not convinced the pdl is the issue, but this is at least promising.

[John Mercogliano]

Gary,

 We had the same issue with a number of our VPN’s as well.  The cause was that the VPN tunnel was going down but not notifying the engine.  We solved 99% of our problems by decreasing our servers tcp keep alive setting to 15 minutes.  Here is a previous post that give a good discussion of this:

https://usspvlclovertch2.infor.com/viewtopic.php?t=734&highlight=keepalive” class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?t=734&highlight=keepalive

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Gary Atkinson]

John-

I have noticed in the process log where this thread resides every 2hr and 13minutes I see the time out pdl error.  The engine will be idle, then 2hrs and 13 minutes, the pdl time error appears.  Then if I try to send out some messages and I never get an ack back.  I don’t see this error in the process logs that have idle time.  I’ve talked to our network guy and he states if I can ping the receiver server everthing is ok.  How can I tell if the VPN tunnel is going down?

Gary

[John Mercogliano]

Gary,

  They only way is I’ve made the assumption that the tunnel is down if that the connection is up but messages are queueing.  They only thing we can do at that point is bounce the connection and see if it comes backup.  Also, pinging is not a real solution because just the act of pinging causing the vpn software to reattach, so just because the ping worked does not mean the tunnel did not go down.

John

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Gary Atkinson]

Thanks John!  The network folks worked something out (some settings; don’t know the specs though), I do not see the time-out errors in the process log anymore.  The vendor also, put something on their side to keep the connection alive.  Since then I have had only one time when it took about 3 minutes to receive an ack.  Now, usually I receive a ack almost immediately.  It is not uncommon where I need to prove the engine is not causing the problem  🙄

Thanks to all for tips and suggestions!

Gary

[Author]

Replies