Opening thread causes throughput issues – CIS 5.8.3.0 P

[Calvin Palmer]

[CLIS 5.8.3.0P on AIX 6.1] I have been able to easily reproduce a generally unacceptable lag time in processing when one or more threads in a process are in an opening state, message throughput suffers greatly for those threads that are up and connected. I’ve documented lag times of up to twenty seconds on processes that contain only five to eight threads. It appears to only happen when threads are configured to use MLLP encapsulation. I do not have the issue when threads are configured to use the TCP/IP PDL. We are forced to use the MLLP encapsualtion due to other issues that are introduced with the PDL configuration concerning establishing a connection. It appears that the process is being bogged down with connection attempts as I see lots of “operation in progress” in the log. We are planning to go live with version 5.8 shortly and this issue may be a show stopper for us. I’ve attached the process log. Any advice would be appreciated.

Topic

[James Cobane]

Calvin,

Have you opened this as an issue with Support?  Also, what issues were you encountering that forced you to configure for the MLLP vs. the mlp_tcp.pdl?

Thanks,

Jim Cobane

Henry Ford Health

[Calvin Palmer]

Jim, Yes I’ve opened a case and am awaiting a reply. The main issue that we see if we use PDLs has been mentioned on CLTech earlier. The TCP/IP host has to be up before any CLoverleaf outbound clients attempt to connect or the connection never happens.

[Mike Ellert]

Hi  Calvin.  I’ve been live with 5.8 since Jan 18th.  I initially had all my threads set up as TCPIP with MLLP encapsulation but had to switch all the threads back to the mlp_tcp pdl a couple days later.

With the mlp_tcp pdl, I only have the connection issue you speak of when the host and server are on the same server – I use client/server pairs for inter-process communication.  For these, I have to make sure the server is up first or the client will never connect.

When the server is another host/application, the mlp_tcp pdl client seems to be able to connect under all conditions.

[Calvin Palmer]

I saw your post concerning the connectivity issues, Mike. I notice that you’re on Red Hat. We’re on AIX 6.1 and run into that connection issue using the PDL on outbound connections to several different remote servers and not just inter-site connections. I don’t know if the difference in what we’re seeing lies in the fact that we are using a different O/S than you.

I hope to hear from my favorite support guru at Lawson today on what he sees. I’ll keep you all informed.

[Mike Ellert]

I have an open case with support as well and R&D is looking through log files.

I also have another open case regarding the smat file cycling not releasing file handles.  Part of the problem still exists even after Rev 3 was applied.  If a thread or process is stopped and started when the log file is empty, the file handle will remain used.

[Calvin Palmer]

I’ve received word that R and D is looking into the issue. I’m also told to expect a fix to be issued in a rev level release. No time frame was given. Meanwhile we plan to go live on 5.8.3 P this weekend with all of our IP connections re-configured to use the PDL instead of encapsulation.

[Mike Ellert]

I am experiencing another difficulty with 5.8.3.0P that you might run into.  Currently about once a week, one of our processes panics after it starts receiving these types of errors:

[msi :msi :ERR /0:   softlab_in:02/11/2011 08:03:18] msiSectionLock: Can’t lock semaphore for thread softlab_in: Too many open files

[msi :msi :ERR /0:   softlab_in:02/11/2011 08:03:18] msiExportStats: Can’t lock data section for thread softlab_in

Lawson support is working on it and thought it was maybe the semaphore settings.  Increasing it did not help.  We have our open file limits set to 20,000 and as far as I can tell, at the time this error occurs, all of the hci processes have a total of about 1,500 files open.

If anyone else has experienced this and has a solution, I’d be happy to hear what you’ve done.

[Scott Folley]

I don’t know specifically about 5.8.3 so this may not be your issue but I know that we had issues with opening threads in a previous life due to the auto-connect/reopen time.

[Michael Hertel]

Mike,

You didn’t say what platform you are on but two things I remember from days gone by was that HP-UX had problems with semaphores.

The other thing you could look at is:

Use lsof to watch the files grow on that process.

It could be that you have an tcl proc that opens a file for writing, but never closes it.

[Alice Kazin]

We had the Semaphore errors when we tried running 5.4.1 and 5.7 on the same box.  I think the issue was that we didn’t have enought free Semaphores.

[Mike Ellert]

I’m running on Redhat.  This is the exact configuration that was running on CL5.5 (also on Redhat).  This problem has only started since upgradeing to 5.8.

[Author]

Replies