Engine slowness due to outside thread connections

[Yamil Velez]

Hi All

   It seems that we are experiences slowness on the engine, when threads that go to outside systems via VPN or VLAN slow down.   We are wondering if anybody else ever experience this and if anybody knows of a solution.

Thanks for your help

Yamil

Topic

[Steve Carter]

Yamil,

This is something that we experienced several years ago.  By default, the threads are setup to auto-reconnect every 5 seconds.  If you have a significant number of outbound VPN connections that are in an opening state, you’ll use quite a bit of systems resources trying to reconnect every 5 seconds.  We changed our outbound reconnect time to 300 seconds and it has significantly reduced our resource consumption (we have just over 3,000 outbound connections).

Hope this helps.

Thanks.

Steve

[Yamil Velez]

In our instance none of the connection were in opening state or did they ever disconnect and we still had message latacy up to 40 minutes.

[Jim Kosloskey]

Yamil,

Are only the VPN threads affected or are all threads affected?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Yamil Velez]

All threads are being affected, and our network team reported that they had reach 100 percent utilitization on the internet.

[Yamil Velez]

Hi Jim

   Just for more infomation we are using Cloverleaf 5.6 and it is hosted on Linux Red Hat 5.2.

[Jim Kosloskey]

Yamil,

And if you stop the VPN threads all returns to normal?

Are these HTTP or TCP/IP protocols or ???

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Yamil Velez]

These are TCP/IP connection and when we brought down one of the threads that we suspected as the offender.  There was a slight improvement.

[Yamil Velez]

Good Morning Jim

   Any idea what is causing this, it has became a norm and any solution will be welcome.

Thanks again for your help

Yamil

[Jim Kosloskey]

Yamil,

You indicate this only happens when the VPN threads slow down. I am assuming by that you mean they take longer to acknowledge a message.

During that period do you get some serious queue depths?

Do you have any idea why they slow down – do they all slow down at the same time (might indicate some sort of network or internet issue)? Does one slow down first then that starts the others slowing down?

Do all of these connections exist in the same process, in the same site?

Are there any sleeps in Tcl procs realted to these integrations?

I really do not have any ideas specifically related to VPN connections slowing down the engine. We do not have many (maybe one) VPN connectikons here and do not experience that.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Yamil Velez]

Hi Jim

 My answers are below your question.  Hopefully they help

You indicate this only happens when the VPN threads slow down. I am assuming by that you mean they take longer to acknowledge a message.

(Yes they take longer to process messages)

During that period do you get some serious queue depths?

(Yes, some queues get as high as 600)

Do you have any idea why they slow down – do they all slow down at the same time (might indicate some sort of network or internet issue)? Does one slow down first then that starts the others slowing down?

(These threads slow down when the hospital Internet gets max out, running 100 utilization,  They start to collect and thus causing the other connections to slow down.   When we do a top command at the Cloverleaf server System load is greater 2.17)

Do all of these connections exist in the same process, in the same site?

(We have three sites production, prod_orders_results and cis_pacs.  In production we have 2 vpn connection.  In prod_orders-results we have 3 vpn connection.   In cis_pacs , we have know known.   The sites most effect are production and prod_orders_result)

Are there any sleeps in Tcl procs realted to these integrations?

(These threads are constantly getting ADT messages throughout the day,  I do not believe they go to sleep)

I really do not have any ideas specifically related to VPN connections slowing down the engine. We do not have many (maybe one) VPN connectikons here and do not experience that.

[Author]

Replies