PDL question

[Michael Hertel]

I’ve caught one of our inbound interfaces taking 16 seconds to ACK.

The log shows the PDL was calling read.done

I don’t know where to find the PDL tcl library.

Does anyone out there know what read.done does? Or where I can find the code?

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  43 6c 69 6e  69 63 20 4f  |Clinic O|

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  6e 6c 79 29  0d 5a 53 32  |nly).ZS2|

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  7c 30 30 30  31 7c 7c 7c  ||0001||||

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  7c 4e 4f 20  52 45 46 45  ||NO REFE|

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  52 52 41 4c  20 43 52 45  |RRAL CRE|

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43]  41 54 45 44  0d 1c 0d     |ATED…|

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] IDLE and 687 bytes but no error: starting READ

[pdl :PDL :DBUG/2:icast_sched_in:05/21/2007 12:23:43] PDL changed states: old 0, new 1

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] Calling Tcl procedure: hci_pd.read

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] with args: {}

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] Tcl procedure hci_pd.read returns ‘RECEIVE’

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] trying to match phrase: basic-msg

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] multi_phrase_2: status = ok

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] Calling Tcl procedure: read.done

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:43] with args: {{status ok} {end {687 0}} {data {1 684}}}

[pdl :PDL :DBUG/0:icast_sched_in:05/21/2007 12:23:59] Tcl procedure read.done returns ”

[pdl :PDL :DBUG/2:icast_sched_in:05/21/2007 12:23:59] PDL changed states: old 1, new 0

[pdl :read:DBUG/1:icast_sched_in:05/21/2007 12:23:59] PDL did read msg: code = 0

[pd  :pdtd:INFO/0:icast_sched_in:05/21/2007 12:23:59] [0.0.282840052] Received msg from driver

[pd  :pdtd:INFO/2:icast_sched_in:05/21/2007 12:23:59] [0.0.282840052] Writing msg to inbound save file /archive/cnew/icast_sched_in.save

[pd  :thrd:INFO/0:icast_sched_in:05/21/2007 12:23:59] [0.0.282840052] Placing msg on IB pre-SMS queue.

[pd  :pdtd:INFO/3:icast_sched_in:05/21/2007 12:23:59] [0.0.282840052] IB pre-SMS message details

msg: 0x3001403c

Topic

[Charlie Bursell]

read.done is one of the subroutinesin the mlp.pdl script.  It looks like:

proc read.done {info} {

    keylset accept text

]

    keylset accept end [keylget info end]

    hci_pd_accept $accept

}

Note that all it does is set a keyed list called accept with data and amount of data from the buffer and returns that data to the engine via the hci_pd_accept command.

(From User Guide, Volume 2 / version 5.5 / page 265)

hci_pd_accept info:

This arranges to deliver the specified portions of the input as a message to Cloverleaf

[Michael Hertel]

So the reading is not really done when read.done is called?

We’ve changed our SAN and I wanted to make sure read.done wasn’t accessing disk. That’s now ruled out.

[Russ Ross]

You indicated that you changed your SAN and I would like to say that our SAN was upgraded and it became unreliable in random ways until we remediated.

Our SAN logical volumes are cloverprod and archive and the AIX commands we used to remediate were

Quote:

dpovgfix cloverprod

dpovgfix archive

fsck -y /dev/fslv01

fsck -y /dev/fslv02

fsck -y /dev/fslv03

fsck -y /dev/fslv04

fsck -y /dev/fslv05

fsck -y /dev/fslv06

Here is the checklist I created and used to help me get things in the right order:

Quote:

su perfman; whoami; killall (perfman was not running so we left it alone)

get current snapshot of Cloverleaf (/quovadx/qdx5.2/integrator/sbin/mda_cloverleaf_IPL_make_scripts.ksh)

su hci -c ‘crontab -r’

gracefully shutdown Cloverleaf by running

/quovadx/qdx5.2/integrator/sbin/mda_cloverleaf_IPL.ksh shutdown

psbatch to make sure real-time to batches are not running

kill_hs

manually unmount SAN filesystems

    cloverprod volume group has the following SAN filesystems

         /ftp

         /hcitest

         /data

         /sites

         /upgrade

    archive volume group has the following SAN fiesystems

         /oldmsgs

varyoffvg archive

varyoffvg cloverprod

lspv >lspv_1.txt

lsvg -o

dpovgfix cloverprod

dpovgfix archive

lspv >lspv_2.txt

varyonvg archive

varyonvg cloverprod

lsvg -o

fsck each of the SAN filesystems listed above

rebuild JFS logs

manually mount the SAN filesystems listed above

lspv to check vpaths again

spot check some file permission of the SAN filesystems to be sure they seem reasonable

manually start Cloverleaf by running

/quovadx/qdx5.2/integrator/sbin/mda_cloverleaf_IPL.ksh startup

start_hs

crontab hci.cron

psbatch to make sure real-time to batches are running

check SNA , TIF, NIC mappings

alpha page to let know we are done and call computer operations

Russ Ross
RussRoss318@gmail.com

[Author]

Replies