CSC Remote Connection dropped Error

[Arslan Khan]

Hi,

We are running CSC 4.3 Server with many clients. There is one CSC alert/error that we haven’t find any way to control/suppress. This is where when CSC client (on the down stream side) drops its connection with the receiving interface (in most cases, a physician EMR). When this happens, CSC server gets bombabrded with below error:

*************************

Error – CSC

Error – Connection to Remote connection dropped on connection 123.1.1.123 22222

*************************

This occurs every 30 seconds, thus 2 alerts per minute. We keep getting these alerts till the time connection is restored with the downstream interface (or the client bring down the said courier). The only workaround we have found so far, and only when data flow is outbound only, is to re-route the monitoring data to another folder than default.

Is there a tcl out there to suppress these alerts coming inbound to CSC server? Does anyone else experience the same errors?

Any comments/suggestions is highly appreciable.

Thanks.

Arslan Khan

RelayHealth/Mckesson

Emeryville, California

Arslan.Khan@Mckesson.com

Topic

[Scott Folley]

We have coded around this issue.  The important thing to consider is that the code that alerts about this is in the csc_monitor.pkg.  The Alert proc to be exact.  This is what we have in there:

Code:

   proc Alert { client type msg } {
   
    variable SS_INBOUND_DIR
    variable error_event
   
    set alert_file [file join $SS_INBOUND_DIR $client LAST_ALERT.dat]
   
   
set err_num  [format %07d [CtrNextValue $error_event]]

# For log    
    set outmsg “[clock format [clock seconds]] : $err_num : $client : $type : $msg”

# For Email Message
set email_msg “CSC Alert: $err_num   [clock format [clock seconds]]n”
append email_msg “————————————————————-n”
append email_msg “Error – $typenn”
append email_msg “$msg”

# For Error File
    catch { set fd [open $alert_file a+] }
    catch { puts $fd “$outmsg” }
    catch { close $fd }
   
    # For process log
    puts “ALERT: $outmsg”
# No alerting if main alert.off file exists
     set csc_base_install_dir C:/quovadx/csc4.3/server
     set SS_DIR [file join $csc_base_install_dir lws] ;# CSC Server working dir
     set SS_REGISTERED_DIR [file join $SS_DIR registered] ;# CSC Server ‘registered’ dir
if { [ file exists $SS_REGISTERED_DIR/alert.off ] } {
   puts “All alerts are turned off because $SS_REGISTERED_DIR/alert.off exists”
} else {
       # Send an email Alert
   switch -glob — $msg {
 “*Could not create*” –
 “*Remote connection drop*” –
 “*Unable to connect*” –
 “*Error while sending*” {
 }
 default {
catch {email_alert $client “$email_msg”} err
puts “After email_alert: $err”
 }
   }
}
   }

This will stop the alerts from coming, but the _error_ directory for that client will fill up with a bunch of error files.  To prevent this you need to look in the csc_msg_route.tcl proc and do a similar thing.  What I did is put a flag variable called info into that proc and, within that branch, I set info to 1 when that type of message is encountered.

Code:

   _error_  {
set ext “err”
# Write a message to the Last_Alert.dat file
switch -glob — $msg {
   “*Remote connection drop*” {
set info 1
   }
   default {
csc_monitor::Alert $client_id CSC “$msg”
   }
}
   }

when it comes time later in the script to write the file out I check the info variable to see if it is set and I don’t write out those informational messages.

Code:

# Write message
if { !$info } {
   append csc_msg_path “.$ext”
   set fID [open $csc_msg_path w]
   puts -nonewline $fID $msg
   close $fID
}

You might ask, well doesn’t that keep me from knowing if the client disconnected?  The answer is that YES IT DOES.  I am comfortable with that because this message is too verbose and we have a lot of clients that connect, send us a message, and then disconnect.  However, with a bit more logic added in, you could stem the flow of those messages by checking to see if there are already recent messages in the error folder for that client and set info to 0 if it has been “a while” (whatever that means to you) since it last occurred.

Of course these messages are basically running through the version of cloverleaf that is on the CSC server so once these changes have been made you would need to bounce the engines associated with them in order for the change to take effect.

Of course, the usual caveats should be taken into consideration as I do use this in my production environment this is a “use at your own risk” solution.  And, of course, when you upgrade you WILL lose these changes and have to re-implement them.

If you understand how and why this works then that will be more than half the battle.

Hope that helps.

[Arslan Khan]

Thanks for the detailed info Scott, much appreciated. I’ll look into that alert proc and see what we can do based on our environment/needs.

Just a quick comment though, how is this different than changing the inbound file path on the CSC settings? Alerts do stop once we do that. Once everything is back to normal, we usually go into that inbound path and delete all error alerts and change the path back to the original one.

[Scott Folley]

What you have to keep in mind is that those configuration settings in the CSC GUI are used to map the CSC paths to the Cloverleaf paths.  A cursory look at the setup of the Cloverleaf instance will show you that, as far as the Cloverleaf instance is concerned, there is only one inbound directory.  That directory is %HCISITEDIR%/csc/inbound on windows and $HCISITEDIR/csc/inbound on linux.  HCISITEDIR is going to be the applicable directory specification “C:quovadxqdx5.6integratorcsc_server” or “/quovadx/qdx5.6/integrator/csc_server”.  The thread that reads the inbound directory is a standard Fileset-Local Protocol.

If you are just sending outbound and are not routing the acknowledgements back to your server then changing that inbound path could be a workable solution.  If you are trying to receive orders or adts or any other message type then that would not be a viable solution because you would not receive the errors but you would also not receive the data messages or the heartbeat messages.

What you could do that would be more robust and flexible would be to implement a similar thing as I talked about originally but set up a flag file that would allow you to turn this functionality on or off based on the existence of that file.  We have done this for other purposes and it works great.  Basically in that instance you would get the client id from the message and check the client’s directory to see if the flag file exists.  If it exists then you would set info to 1 and the messages would be suppressed.  If it does not exist then info would stay 0 and you would get the alerts and you would accumulate the _error_ files.

It is good to hear that there are other folks out there using CSC, we use it quite extensively and it is a great system for bridging gaps where a VPN tunnel does not make sense.

[Arslan Khan]

Thanks for the clarification Scott.

BTW, are you using 4.3 or 4.4? How many clients do you support on CSC?

I haven’t seen any specs for 4.4 yet and not sure how they (CL folks) address the client upgrade process. I am hoping that they’ll come up something very transparent in terms of upgrading a client from 4.3 to 4.4.

[Scott Folley]

We are using 4.3 and 4.4:

4.3 on a windows 2003 server

4.4 on redhat linux

We are in the process of the upgrade from 4.3 to 4.4 but it is challenging because there is not a simple process to upgrade.  4.4 is definitely where you are going to want to go because it sends much faster and is better in every way that we have seen up to this point.

We have a little under 200 clients at this point.

[Arslan Khan]

So, basically, in order to do an upgrade, you have to reach out to each of your 200 clients and do the re-regoistration??

This will be a killer, specially if you have to upgrade, say, 500 clients.

[Scott Folley]

Yes, that day is not here yet but we will likely be at that point in the next couple of years or so.  The way that we handle it is by using a remote access program.  This is also how we handle support of the CSC clients, that makes it a bit more challenging but we have made that a standard so it is not as bad as it sounds.  At the user group 2 years ago I gave a presentation on this and I still stand by what I said then, I would not install CSC on a client that I do not have remote access to in some way.  I have had to back off of that on two occasions and both of those occasions have come back to bite me from a support standpoint.

Even though it IS a killer to upgrade all of those clients, I strongly believe that the whole thing could be fully automated.  That would (will) take some time and effort but that is what I am shooting for next.  My biggest issue with that (believe it or not) is with the constant modification of the ports that CSC uses for its connections.  We have several clients where their network people block ports going outbound so we are stuck with either changing the ports that tomcat is listening to or having the newest set of ports opened.  We have elected to modify the ports that Tomcat listens to but this means that we cannot run two different versions of CSC at the same time.  The effect of that is that we have to flip the clients back and forth between our two servers during an upgrade.  That is bad but the benefit is that we maintain full control over when the upgrade will occur and we are not stuck waiting for a port to be opened.

This automated upgrade process is becoming more of an option in 4.4 because 4.3 and below are unable to transfer the large amount of data that the CSC install executable represents.  As far as I can tell, 4.4 handles it just fine.  I am sure that this limitation on the size of a transfer has kicked you in the tail a time or two (just try having the server retrieve the client’s log files and you will know what I mean…on second thought don’t actually try it just trust me that you don’t want to).  In 4.4 it looks like, within reason, those options really are valid.  This product is improving exponentially with every new version and the changes from 4.3 to 4.4 are no exception.

We already have a process that automatically does everything on the client side up to sending the registration to the server so most of the scripting required to auto-upgrade is complete.  Our challenges to that process are the ports and the transfer of the client install software.  Using a task in 4.4 we could transfer the software so if we can solve the port problem then we will be there.  I have a couple of ideas there but nothing solid yet.

[Arslan Khan]

Interestingly, we haven’t face this much problem with ports as such. All of our clients do follow the standard ports, so it is a non-issue with us. As far as controlling the client via remote session, based on the number of clients we have, it is non-workable for us.

So far, other than upgrading, we haven’t had many issues with this product. There are regular disconnects, but most we manage via restarting via remote (recycle_all to dummy test courier on prod machine). We push the client install via secure web link. Clients can download and install the sw with no issues. Trouble comes when we have to schedule a time with them to register and transition them from old version to new (say from 4.0 to 4.3). Having such large number of clients, it is a constant pain to run after a client to schedule a time for upgrade. I am thinking about a way to script this piece so that, once client send us a registration, we can send a script via CSC server which can do the transition for them (from an old CSC version to new CSC version) without the need for any intervention from the client. Not sure whether it is doable or not, but indeed it is worth trying.

[Scott Folley]

How many clients do you have using CSC?  I am assuming it is fairly extensive.

[Arslan Khan]

So far, about 250 but adding more on daily basis.

[Author]

Replies