We use EMC with both AIX and Sun. We had the wrong configuration in the Sun box and experienced something similar. We stopped writing to the disk but the processes continued working with the RAM.
In some interfaces, we noticed that since the box was unable to write to the disk, the message was accepted but no ACK’ed. The sending application continued sending the same message again and again. Once the path to the disk was restored, all the messages in memory were saved and ACK’ed. We noticed this because we realized that we had duplicate charges.
We also found that when the EMC team updates drivers or do some maintenance, our Sun box is disconnected from the SAN for several seconds (up to 15 seconds some times). This is not a problem unless you have an interface that expects an ACK back in less than 15 seconds.
We monitor the disconnects with a script that “touches” a file in the SAN every 10 seconds. The touch command is timed and if it is taking too long (more than 2 seconds) a page is issued. We were able to catch some instances were the paths to the disk were failing and helped the support group to diagnose the problem. Now we have the right configuration and the EMC admin team knows that we are notified… The number of short outages was reduced drastically too
