Unable to Open port error from vendor.

[Robert Denny]

We have Ipeople sending hl7 messages to our Cloverleaf Interface Engine. We are continually having problems where the Ipeople Interfaces are unable to open the port to the cloverleaf server, and so they just stay in that state until the ipeople interfaces are stopped and restarted.

Anyone have this experience? How was it corrected?

Also, I am interested if there is a log I check on the Cloverleaf server for these processes that the ipeople interfaces are sending the data too. I would like to see what kind of data Cloverleaf is recording about the data not coming accross?

We have an alarm setup so that if there is no data for 60 minutes a few support folks would get an email, and text message, but if the threads are reset at all during the hour that the data is not sent, then clock resets.

Topic

[Anonymous]

There are a lot of unknowns in what you are asking.

What is the state of the thread they are connecting when this happens ?

Is the connection up all the time or do the connect and disconnect after sending ?

Is this over a VPN ?  Or is there a firewall between you and the sending system.

What pdl are you using ?

I have seen this problem several times, but the circumstance vary based on the connectivity and settings.

The log file really can’t tell you about data your not getting but if you turn the eo alias up enough you can see the details about connecting or disconnecting which might help.

[Robert Denny]

Thanks for the reply.

this is an ipeople server that we have within our network. I am not part of the system engineering group, but I would assume that there is not a firewall, or vpn connection. It is just a meditech session running on this server, and ipeople configures the data from our meditech system and sends the data to our Cloverleaf interface engine, and all the servers reside within the same server room. So they are all behind the same firewall, so I do not think it plays any role between these servers.

The connections are always on, and as soon as a user adds data to the meditech system, and files it the data is queued up to send from our meditech system to the ipeople session for data modification, then the data is sent directly to our cloverleaf interface engine, and from there it is sent to our EMR system which is a logician system. If we toggle an thread at the time that the Ipeople session is sending data to a thread, it can get stuck in an unable to open port error and will stay in that state until the ipeople interface is stopped and restarted, at that time the data normally will start sending once again.

[Russ Ross]

I would try the following things at the moment your problem occurs and still exists if possible.

Make sure you aren’t using a port number in the ephemeral range.

If that doesn’t make sense then just make sure the prot number is below 32K.

Try to use hcitptest when the problme occurs to see if you can connect to confirm there is a connection problem.

Try to see if something else is using the assigned port at the time the problem occurs.

The way I check if more than one thing is using the same port is to use

lsof | grep “my_port_number_of_interst”

The second column of the lsof output is the PID of the processes using the port.

then I use the command

“ps -ef | grep “PID_of_interest”

to find out what processes are using that port at that moment and can get valuable insight into what is fight each other for the same port number.

If you don’t have lsof loaded and working on your server a quick approach to indirectly guess if this is the problem it to pick another port that hopefully isn’t used already either.

We try to keep a list of our assigned prots but I still have the issues with non-cloverleaf applications using ports and I didn’t discover the ephemeral range concern until after it hit me in the face and I have considerable remdiation to eventually do with regards to those ports.

I also suspect ports assigned over 65K get adjusted down adding to the problem.

It is likely I’ve said something that isn’t completely accurate because I’m still running into issues myself but this is my current way of very breifly describing the port world challenges I thought about when I read your post.

Russ Ross
RussRoss318@gmail.com

[John Mercogliano]

Robert,

One thing of note that you said and I want to make sure that I’m correct, is that this only happens when you toggle the thread when ipeople is in the process of sending data.

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Chris Williams]

To add a side note, Russ is correct about the port numbers, but there can be a fair amount of misunderstanding when it comes to assigning them.

Port numbers are held as a 2-byte integer (0-65535). The numbers will roll over if you specify one greater than 65535, e.g., 65536–>0, 65537–>1, etc.

The upper half of the range (32768-65535) is referred to as “ephemeral” which is a fancy way of saying they are reserved for use by the operating system itself.

When you connect as a client to another system, the port you specify is the port on the other system. On your side the connection is made using one of your ephemeral ports, selected automatically by your operating system.

When you act as the server, the port you specify in CL is the port on your system (note the host address box is grayed out), and the remote system will connect to you using one of their ephemeral ports.

Also, when using netstat to find a port number that’s available to use, be aware that the numbers you see are ones that are active at that instant. A number that you have assigned won’t appear on the list if the connection is not up at the time you run netstat.

John mentioned using grep. You can get grep for Windows from http://www.oakroadsystems.com

[John Mercogliano]

One more thing about ephemeral port assignment.  Not all OS’s use the same range.

Here is a good link that give some more details

http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html” class=”bbcode_url”>http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

And here is a clovertech discussion that might be of interest.

http://clovertech.infor.com/viewtopic.php?t=2681&highlight=ephemeral” class=”bbcode_url”>http://clovertech.infor.com/viewtopic.php?t=2681&highlight=ephemeral

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Robert Denny]

Thanks for all the replies. Also I am curious about sending a command to clear a port that is locked?

I used to do a send break to a port whenever it was stuck on terminal servers. I connected via telnet, and then logged on with admin rights, then issuing the command send break . I would clear the lock and data would start transmitting again.

We are curious if this might still work if we telnet into the box that has a locked port, or if there is a command in tcp/ip for this task? Do any of you have procedures for clearing an “UNABLE TO OPEN PORT” error on a system that is sending data to your cloverleaf interface engine.

Does turning the thread on and off do the same thing? Does it Kill the port or break the port?

Does this site have documentation specific to Vendors like Ipeople? I am going to write to them next to see what steps they take when we call the to clear the “UNABLE TO OPEN PORT” error message.

[David Barr]

Robert Denny wrote:

I used to do a send break to a port whenever it was stuck on terminal servers. I connected via telnet, and then logged on with admin rights, then issuing the command send break . I would clear the lock and data would start transmitting again.

TCP/IP ports and serial ports are two different things.  Terminal servers use serial ports.  A “break” is a special operation that you can perform on a serial port that is irrelevant to TCP/IP.

Robert Denny wrote:

Does turning the thread on and off do the same thing? Does it Kill the port or break the port?

When a TCP/IP port is in use, the simplest way to free it up is to terminate (or reconfigure) the process that has it opened.  The steps that Russ described will work.

[Nick Schneider]

I have recently run into the problem where the firewall TCP/IP Idle timeout was occurring milliseconds before the vendor keep-alive was able to come across.  Like others have mentioned and I have seen elsewhere my inbound thread thinks the vendor is still connected and refuses all connection attempts until the thread is bounced.  I have been unable to get the Network groups on either end to address this so I went ahead and setup multi-server and have been able to get around this issue with one exception.  I am not seeing the OS drop the abandoned connections.  We are running CL 5.6 on Windows Server 2003.  Does anyone know if there is a setting I need to change in order to have Windows recognize and release the abandoned connections before I write a proc to stop and start the process every so often?

[John Mercogliano]

When you do a netstat, do all the connections show established or does it say timeout, fin_wait or some other state?  If established then they won’t go away until you bounce, but any other state should eventually go away as the OS cleans up stale connections.

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Bob Richardson]

Greetings,

Responding to duplicate port assigns issue:

I just wanted to offer a solution to avoid assigning duplicate ports to your threads in the NetConfig files of your sites.  We ran into a problem where we did double assign a port and then during a failover/restart two of our client threads wound up with the same server port.  The “wrong guy” grabbed the port first. [A change was made but not applied before the failover]. The data streams were crossed resulting in a long debugging session and “trial” before the end users the next week.

My colleague, Ian Smith, developed a TCL script that reads the server.ini file and builds a list of all the available sites.  It traverses all the NetConfig files for each site and builds a list of all assigned ports for TCP/IP (PDL or otherwise) threads.  By checking the final sorted list, duplicates are located and we are warned.  It runs every hour on both our Test and Live servers and works very well to keep us in line.  

I have included this TCL procedure for your review and possible usage.

[Author]

Replies