Another mysterious Tcl proc: hcitpstblconvert

[Paul Glezen]

Last month I asked about a Tcl proc, hcitbllookup, that I found referenced in code I inherited from a retiree but could not locate in the documentation.  I was informed that this was the same as the tbllookup function, which I did indeed find documented.

I seem to have a similar issue.  One of my outbound threads yielded the following error message during a test.

Code:

[pd  :pdtd:ERR /0:       ims_ob:04/13/2016 13:25:54] Tcl error:
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]    msgId   = message1
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]    proc    = ‘hcitpstblconvert’
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]    args    = ‘{TABLENAME ibmtext_a2e}’
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]    result  = ”
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]    errorInfo: ‘
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–] Error msg sequence error (max?)
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]     while executing
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–] “msgmapdata $mh $tblname”
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]     invoked from within
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–] “if [keylget args MSGID mh] {
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]                    set uarg    [keylget args ARGS]                         ;# user args
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]                    set tblname [keylget uarg TABLENAME]            ;# conversion table
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]                    
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]                    ms…”
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]     (procedure “hcitpstblconvert” line 10)
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–]     invoked from within
[pd  :pdtd:ERR /0:       ims_ob:–/–/—- –:–:–] “hcitpstblconvert {MSGID message1} {CONTEXT prewrite} {ARGS {{TABLENAME ibmtext_a2e}}} {MODE run} {VERSION 3.0}”‘

The string “Error msg sequence error (max?)” seems to imply an index which exceeds the size of the table was passed to this function.

When I inspect the thread in Network Configurator, I do see that the Outbound tab’s Prewrite Procs field contains the name hcitpstblconvert.  The supplied args are {TABLENAME ibmtext_a2e}.  Ostensibly, this is an ASCII to EBCDIC conversion utility since the target is a mainframe CICS server listening on TCP.

Remembering my lesson from my previous inquiry, I searched the API docs for tsptblconvert.  But I could not find that either.  The closest I could find was tspconvert; but that didn’t seem to be the same thing.

I also couldn’t find a table named ibmtext_a2e in my list of tables.  I grep’ed all over the place and found the table created with the adddatamap function in

/tcl/lib/cloverleaf/hciStart.tlib

But I’m still at a loss to where the hcitpstblconvert command comes from.  Is it a deprecated Cloverleaf Tcl extension?

Topic

[Rob Abbott]

Hi Paul, yes the hcitpstblconvert is no longer suggested for use but it should still exist.  It would be in $HCIROOT/tclprocs/tps.tcl.  You may have a problem with the tclIndex in that directory.

However newer versions of Cloverleaf use Unicode internally which causes problems with this old method of converting ASCIIEBCDIC.

To convert ASCII to EBCDIC in newer versions of Cloverleaf (5.8 and above I believe) you will want to do the following:

– remove hcitpstblconvert from the outbound TPS

– in the thread properties, change the Encoding to “CP037”

Similar configuration will work for inbound threads that are receiving EBCDIC (remove any calls to hcitpstblconvert and set encoding to CP037)

If CP037 is not appearing in the dropdown for encodings, you can add it via the Server Admin GUI.

Rob Abbott
Cloverleaf Emeritus

[Paul Glezen]

Thanks, Rob.  I’m hesitant (but not unwilling) to flip the CP037 switch since my predecessor related to me that he spent a good deal of effort getting the a2e and e2a working after the upgrade to 6.1.  (Maybe all he really had to do was use CP037).  But I also know he wrote a great deal of custom code doing things like packed decimal conversions, etc.

One angle I’m investigating is an inbound 2-byte binary field from the mainframe.  It’s a binary length and thus has no EBCDIC/ASCII representation.  Of course, that doesn’t stop the converter from converting it.  In theory, it should be harmless since that particular length isn’t used for downstream processing.

But I’ve noticed certain values seem to cause problems.  I’m trying to investigate more deeply by dumping error messages for failures and SMAT messages for successes in order to compare them.  I view the contents in a hex viewer and I’m trying to understand what I’m looking at.

I’ve read that Cloverleaf stores the messages internally as unicode.  When SMAT or DB messages are written to files, they are utf-8 encoded.  I’ve been assuming that.  But in one case, the error DB exported the following sequence (from outbound to mainframe just before ASCII to EBCDIC conversion):

Code:

32 31 20 9b 31 35

In ASCII, 3x maps to the single digit x.  20 is a space.  But 9b is suspicious.  It’s not a valid utf-8 mapping.  If it was really 9b in unicode, it would map to c2 9b in utf-8.

Earlier in the message the 2-character length field I mentioned earlier shows as

Code:

7f c3 9b

If I assume these are utf-8 encoded, I expect them to represent unicode characters  0x7f and 0xdd.

I doubt it’s a coincidence that the 9b on the end of my 3-byte example is the same value that causes trouble in the first sequence I listed.  The translation rule that handles this inserts an 8-character field right after the two-character (2-byte) length.  The result seems to be that the 0x9b is in both places: (the original place, which is 2-bytes in length, but expands to 3-bytes in utf-8; and 8 characters later, it appears again.  The FRL subfield definition lists the 2-byte length type as a character field of length 2.

I’m contemplating a band-aid by which I simply convert those two field positions to 0x40 (EBCDIC spaces) in the inbound TPS stage before any translation rules run.  But I want to be sure I understand what I’m seeing when I view SMAT and DB output.

[Charlie Bursell]

As you have discovered when the method of storing data internally was changed to Unicode it sort of screwed over the conversion tables.  These tables date back to the mid 1990s.

Rob is correct when he says by far the best method is to use CP037 or similar.  There are other encodings to be used for European, etc.  You can Google those.

Another problem arises with vendors like SMS that use some specialized codes for their EBCDIC data.  That is why we had a special table (sms_e2a) for that

I have enclosed a specialized proc to show what can be done with your own mapping.

[Paul Glezen]

Thanks, guys, for your replies.  That helped me zero in on the problem.

As I related above, there was a suspicious 2-byte length field that seemed dangerous to attempt an EBCDIC to ASCII to Unicode to ASCII to EBCDIC conversion.  I’m new here; and I was told that this has all been running fine in PROD for a long time (the grief I related above is only occurring in TEST and I refuse to promote any changes until I get them working in TEST).  Well, you can probably see where this is going.  After checking PROD, I found an extra TPS inbound data script that plugs those two bytes to zeros.  This happens before any of the conversion procs.  It had never been back-ported to TEST.

Once I added the extra TPS inbound data invocation, TEST has not experienced the conversion issue.  This will at least allow me to promote recent changes from TEST to PROD while I contemplate a longer term CP037 solution.

[Author]

Replies