CL 6.0: hcismat issue ‘bad vfariable name’

[Jim Kosloskey]

After converting old SMAT format to new – using hcismatconvert – trying to extract to a nl file get an error:

–> hcismat -i converted.idx -nl -orsf converted.nl -sall

hcismat: Invalid argument bad variable name “::SMAT::argArray(TARGET)”: upvar won’t create a scalar variable that looks like an array element

Anyone else experienced this or have suggestions?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

Topic

[Chris Williams]

Jim,

We’re not up on 6.0 yet, but I did get some sample SMAT files from the vendor so we can be prepared. The OFFSET and LENGTH values from the idx file will still give you the messages you want without the new stuff that is tacked onto the end of each message.

Cheers.

[Jim Kosloskey]

Chris,

Thanks,

Yes we have a Tcl proc originally written by Jim Cobane we can use. That has been tested and gives us what we want.

We do want the Cloverleaf distributed command hcismat to function though. I think this is broken in 6.0. The thought is it is related to a new release of Tcl and the way something is written in the hcismat tool.

We would like to confirm though that others on 6.0 are experiencing the same issue. It would be noce if someone else that has 6.0 could give hcismat a run. It appears the issue has nothing to do with the data but rather the handling of the arguments.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

Jim:

Any chance I can get a copy of the SMAT file.  I can’t duplicate the error

This was written with 6.0 in mind and I have never had a problem with it

[Jim Kosloskey]

Charlie,

I will get a file to your email address tomorrow if that is OK.

This is a 5.6 SMAT file which was run through the hcismatconvert utility to convert it to 6.0 format.

Thanks for looking into this.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

I think I found it Jim.  It seems that getopt acts differently in Tcl 8.5 and Tcl 8.4.  Try this one and let me know.

I still need to work on making it work with old style SMAT files

If on Unix just remove the .htc

[Jim Kosloskey]

Charlie,

Will do and will report back – probably tomorrow.

Thanks again my friend.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Keith McLeod]

Was hoping to normalize my date/time range for a more accurate compare between smat files.  Looking for the correct syntax for hcismat.

Using something like this:

hcismat -i to_somewhere_adt.obsv.141109.1.idx -nl -sds 201411091200 -sde 201411091700 -oruf /hci/cis6.1/integrator/somewhereprod/data/61_to_somewhere_adt.nl

Not having much luck with the date/time range.  Any suggestions?

It does write out all the messages and not the specified range.  I have tried changing oruf to orsf which returns nothing.

[Charlie Bursell]

Kieth:

I rewrote hcismat as the old version depended on using Tk which does not work any more.  I tried to keep it the same as before only adding the -nl flag.

I tested dates with a SMAT file of my own and it seems to work OK for me.  Could it be that you used the -oruf flag that says you want unselected records? If you want the selected records, use -orsf.

I tested it using 5.8 since that is where I wrote it.  I seem to be having a problem with running using my Cloverleaf 6.0.  I get the error message:

hcismat: Invalid argument bad variable name

“::SMAT::argArray(TARGET)”: upvar won’t create a scalar variable that looks like an array element

It looks like the optargs routine I use to parse arguments may have changed with Tcl  8.5

Will someone that has both the hcismat script and 6.0+ give this  a try and see if you get the same error?  I just want to make sure I am not spoofing myself before I chase this too much.

[Keith McLeod]

I am using the version downloaded from above jun 17 2013, I believe.  I am on CLoverlaf 6.1.  I was able to run it on 5.7 and 6.1 SMATS, just not for a selected date and time.

I just ran against all my SMATS across sites using this command inside a script. Similar for 57.

hcismat -i ${smatfile} -nl -orsf ${in}/${curDir}/data/61_${filebase}_${dt}.nl -sall

The date still eludes me.  I have used both oruf and orsf.  I have even thrown in -sdn.  It looks like whether sdn is used or not it saves to the SELECTED array, before you use intersect3 for res.  I haven’t spent a lot of time on it yet…

I just figured if I could take a range of time where I knew both SMAT files were relatively equal, I could compare messages more easily…

You know I appreciate all the help you have provided.  I will be looking in the morning but need to keeep moving forward on my compares….deadlines loom.

[Charlie Bursell]

Strange!  I am able to select a range of dates using mine/  I will download he one I uploaded before to see if differences

[Charlie Bursell]

Kieth:

Did some more check on the one I downloaded from here.  First I no longer get error under 6.0 – very strange.

Next I found I had forgotten to change the time from SMAT from offset to YYYYMMDDHHMM before compare.  I have changed this one.  I will continue to test here.  I appreciate any further testing you do with this

Most people just use it as a quick and dirty to convert SMAT to NL but it should be more than that

Thanks

[Robert Kersemakers]

Hi Charlie/Keith,

I have been looking at hcismat as well, as we are getting 12 MB messages (=files) in SMAT that we can’t open in the GUI.

I tried this last version, but it still won’t work. I had a look and put in some echo’s:

Code:

orbis_0210 /prod/cloverleaf/sites60/orbis/exec/processes/transport> hcismat -i prhc_hr_dvb_in -orsf test_rk01.txt -nl -sds 20141111
start: ; end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:

hcismat: 3 written to test_rk01.txt

You have new mail in /var/spool/mail/hci
orbis_0210 /prod/cloverleaf/sites60/orbis/exec/processes/transport> hcismat -i prhc_hr_dvb_in -orsf test_rk01.txt -nl -sds 20141111 -sde 20141111
start: ; end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:

hcismat: 0 written to test_rk01.txt

orbis_0210 /prod/cloverleaf/sites60/orbis/exec/processes/transport> hcismat -i prhc_hr_dvb_in -orsf test_rk01.txt -nl -sds 20141111000000 -sde 20141111235900
start: ; end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:

hcismat: 0 written to test_rk01.txt

The SMAT contains 3 messages, 1 message each day.

So the first try should give me 1 message: the message of today. But because the startdate is only a date (YYYYMMDD), it will be always smaller than a complete timestamp (YYYYMMDDHHMMSS).

In tries 2 and 3 the enddate is given and it should work, but the ‘clock format $time’ is missing the %S, so time is always smaller than startdate and no messages are returned. Adding %S will make it work if you

I think the best thing to do is ‘clock scan’ all input-dates into seconds and compare those, but you will need to take into account that only dates can be input instead of complete timestamps. I haven’t got the time to do this myself right now; will keep you updated if I find some time.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Charlie Bursell]

Did you try the new one  I just posted?

[Robert Kersemakers]

Hi Charlie,

Yes, this was tested with the new one.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Charlie Bursell]

When you give a start and no end date the end date will default to something like the year 2045.  It expects a timestamp not a date.  If you wanted everyting since 20141111 just input 201411110000.  If you want only that date then input 201411110000 – 201411112359

Won’t that work for you?

[Robert Kersemakers]

You will need to change

Code:

   # Select if match – convert time to YYYYMMDDHHMM
   if {[clock format $time -format %Y%m%d%H%M] >= $START && $time <= $END} {

into

Code:

   # Select if match – convert time to YYYYMMDDHHMMSS
   if {[clock format $time -format %Y%m%d%H%M%S] >= $START && $time <= $END} {

so with %S in the time format.

If you do this then yes, inputting a complete timestamp will work:

Code:

orbis_0210 /prod/cloverleaf/sites60/orbis/exec/processes/transport> hcismat -i prhc_hr_dvb_in -orsf test_rk01.txt -nl -sds 20141111000000 -sde 20141111235900
start: ; end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:
time: ; clock format time: ; start: ;end:

hcismat: 1 written to test_rk01.txt

However (and don’t shoot the messenger…) the ‘help’ mentions that it is possible to input a date, without a time.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Keith McLeod]

I made this change.  I think this works, but will run a few more tests.

regsub — {(d{8})(.*)$} $START {1 2} s_start

regsub — {(d{8})(.*)$} $END {1 2} s_end

           if {$time >= [clock scan $s_start] && $time <= [clock scan $s_end]} {

So far I got the correct results.

[Robert Kersemakers]

Hi Keith,

If you don’t specify either the start or end, then these will be defaulted to a ‘seconds’ value (0 resp. 2145852000). Then your regsubs will go wrong, right?

And Charlie: I see now that you mentioned the start/end dates in the form ‘YYYMMMDDHHMM’, without the seconds. Used like this,  the last version will work. But this also deviates from what is mentioned in the ‘usage’ of this script.

I may be nitpicking though…

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Charlie Bursell]

I thought the usage statement listed it as a time stamp.  I will take a look and change as required

Let me know of any needed enhancements and I will consider them

Just trying to make an old command usable in today’s environment  😀

[Keith McLeod]

I set START to default to 000000000000 so the criteria is met.  Not ssure how much effort to make old smat files from 5.7 work.  That would be helpful especial using the START and END Date/Times.  It looks like in some of the code lgets is used instead of gets in the while loop to accomodate the stanza structure of the .idx file.  I ran the command manual and it appeared to work, however only pulled ‘{‘ more often than not.  The code would exit where the one of the three keys is blank. This is in the selectDate proc.

# If no OFFSET or LENGTH or TIMEIN bad file

  if {$off eq “” || $len eq “” || $time eq “”} {

   puts stderr “n$myName: Invalid or corruted SMAT file

$argArray(SMATFILE).idx”

puts stderr “Exit ……………n”

exit -64

   }

[Charlie Bursell]

I stand corrected.  I forgot the time was optional.  I guess I never did test the dste stuff enough – sorry bout that.

While in there I found a couple of other “undocumented enhancements”  😀

Plus it seems the clock command will not scan a date past 20371231. I did not know that.  I will query the Tcl gurus about that.

Here is my latest.  Please feel free to put it through the wringer and make suggestions or comments, even snide comments.

As always for UNIX simply remove the .htc suffix make it executable and owned by hci.  If on pre-6.0 you will have to replace the one in the $HCIROOT/bin directory with this one.   On 6.0+ you could just ut in the contrib directory

I don’t have any pre-5.8 SMAT files to test with or I would see about making the script work with those.  I will look around and see if I can find some.   If you have any that do not contain PHI data send me some

[Charlie Bursell]

I should have checked.  I was testing using 5.8

The scan limitaton with the clock command only applies to pre-Tcl 8.5 – Cloverleaf 5.8 or less.  You can scan well into the next century with the new clock command

[Robert Kersemakers]

Hi Charlie,

Just tested with dates (full timestamps and only dates), mids, not, sending (un)selected, newline or len10. Everything works!

Only thing I haven’t tested is resending the messages to a thread.

Thanks!

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Author]

Replies