On this site, we use a tcl proc that reads in messages and either continues them or kills them depending on contents. We also use hcitpsmsgkill and kill_ob_save on the outbound threads on that site.
So, here is the question (finally )
Does executing all these kills somehow “use up” all available semaphores on a site level?
To address this part of your post: hcitpsmsgkill acts upon a message, not a file and thus would have no impact on open file handles or semaphores.
kill_ob_save only handles housekeeping of getting rid of a saved state14 message and clearing a memory variable. Again, not an action that should be related to the errors you mention. These items should not spawn processes that need killing and therefore should not be related to the kill processes noted by your unix admin.
Your error referenced here mentions thread sendafile. Is it always this thread? If so look closer at that thread and any tcl code associated with it for possible leaks.
You might also turn the engine output for the associated thread/process up to enable_all and see if you get more information about what is happening leading up to the panic. Be sure you have auto log cycling turned on as log will grow substantially.
