Replace A0 ascii character with A20 ascii character

[Kirby Knight]

I am using the following proc to try an replace the hex A0 space char with the hex 20 space char.

It seemed to work but The issue is that other characters changed as well. For ex. The

Topic

[Jim Kosloskey]

Kirby,

Just a guess here but I think you mean this

     msgset $mh $newmsg

     return  “{CONTINUE $mh}”   ;# run

Instead of

     msgget $mh $newmsg return

      “{CONTINUE $mh}”   ;# run

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

What Jim says is true.  Also the statement:

   regsub -all — “xA0” $msgdata “x20” newmsg

Since the fields are ib quotes instead of braces they will be interpreted by Tcl as well as the regsub command and can give weird results.  If using the regsub I would do it like:

    regsub -all — {xA0} $msgdata “x20” newmsg

However in this case I would not use a regsub at al I would use a string map command

      set newmsg [string map

$msgdata]

[Jo Ellen Laansma]

I have invented my own code, tried supplied code, and cannot get what you are suggesting to work.

In my message, there is an embedded xd7 in OBX-5.

I’ve tried:

[David Barr]

Jo Ellen Laansma wrote:

x7f to xff to see if there was something I wasn’t seeing.

But that doesn’t come close to covering the range of non-ascii characters. In TCL they go up to at least 0xffff.

Why don’t you try printing out the values of each character in your string? TCL may be doing some kind of strange conversion of the message before it gets to your script.

You could add something like this to your TCL proc.

Code:

foreach c [split $msg {}] {
 scan $c %c dec
 echo $c $dec
}

[Jo Ellen Laansma]

I’m a little confused by your statement.  The largest value in one signed byte is FF.  And a hex editor shows D7.

But I did what you suggested and the mystery character came up with 65533 which is FFFD .  

I guess that proves the theory that what it’s reading in is not what I thought and I agree with your statement that the tcl must be somehow interfering with the read.  But it still leaves me unable to change the value within the tcl preproc.  (I’ve tried xffxfd and xfffd).

I think I’m relegated to capturing the bad messages in a file and use a unix command to replace the original d7 problem (which does work), although cumbersome.

Thank you for your comments.  If you figure out (based on the message I uploaded) how to change the character, I would still be very interested in the result.

[Jo Ellen Laansma]

By gum, using a Unicode translation fixed it!

I regsub’d ufffd instead of x.

[Jo Ellen Laansma]

And then, after successfully testing, I try to implement it and it will not work.    Such frustration.

I thought maybe it was the “state” it was in when I resent it, so I made sure the inbound thread received it as it normally would by creating extra threads and sending the message down a pipeline.

Another post said that the tester needed \ or {} to indicate it was a control character and the implemenation did not.  I’ve tried both ways and neither work.  I gave up passing as a variable, afraid that was impacting the esacape sequences and it still does not work.

[Jo Ellen Laansma]

So I thought maybe now that it’s going through the engine, it interprets the characters differently.

…

Yes, it’s back to the original d7.

[David Barr]

Jo Ellen Laansma wrote:

I’m a little confused by your statement.

[Michael Hertel]

If you’re trying to convert only nulls why not use msgget?

set msgdata [msgget -cvtnull x20 $mh]

From help contents:

msgget

Retrieves message data from the region specified by offset and length.

msgget ?-cvtnull char? msgId ?offset? ?length?

Nulls in the retrieved data are converted to char if you use -cvtnull. If the message contains nulls and you do not use the -cvtnull option, you cannot access any message data past the first null in the message from Tcl.

In addition, if you store the string returned in this non-conversion case, only the Tcl-visible string portion is stored.

This command returns the retrieved, converted data

[Jo Ellen Laansma]

Lastly…  I found that when you “resend” messages it had the unicode character ufffd and when they came from a thread in the traditional way it used xd7.

In case anyone wasnted to know there is a difference between resending and thread throughput.

[David Barr]

I tried setting up a tcp-ip inbound thread, a file outbound thread to a /dev/null file and a raw static route between them. I put a proc on the route to print the message contents (and the hex code for non-printable characters). Here’s the code I used:

Code:

           keylget args MSGID mh
           set msg [msgget $mh]
           set msg2 “”
           foreach ch [split $msg {}] {
               scan $ch %c dec
               if { $dec >= 32 && $dec <= 126 } {
                   append msg2 $ch
               } else {
                   append msg2 [format "” $dec]
               }
           }
           echo msg = $msg2

           lappend dispList “CONTINUE $mh”

I sent your message to the file using the TCP tester like this:

Code:

hcitcptest -h localhost -p 42772 -t mlp < badlabNL.txt

I used a resend command to send the message like this:

Code:

hcicmd -p helloworld -c “win1252_in resend ib_post_tps data 5120 $HCIROOT/test_data/badlabNL.txt nl”

I used the testing tool to run my proc like this:

Code:

hcitpstest -r run -x windows-1252 -f nl -c sms_ib_data -e “hcitpstestshowbydisp ” /quovadx/cis5.8/integrator/test_data/badlabNL.txt “tps_win1252”

In all of these cases the multiplication character was in my string as D7.

Code:

msg = MSH|^~&|PATHNET|GHS|GEMS|GHS|20150624124246||ORU^R03|Q309464539T306492554|P|2.3PIDPV1ORCOBROBX|||||883056 883134 88302

The only way that I was able to replicate your problem was to use the testing tool and change the encoding to ASCII like this:

Code:

hcitpstest -r run -x ASCII -f nl -c sms_ib_data -e “hcitpstestshowbydisp ” /quovadx/cis5.8/integrator/test_data/badlabNL.txt “tps_win1252”

This is the output:

Code:

msg = MSH|^~&|PATHNET|GHS|GEMS|GHS|20150624124246||ORU^R03|Q309464539T306492554|P|2.3PIDPV1ORCOBROBX|||||883056 883134 88302

The Unicode FFFD (�) character is actually a special character called the “replacement character”. According to Wikipedia “It is used to indicate problems when a system is not able to render a stream of data to a correct symbol.” It makes sense that I would see this if I picked an encoding that doesn’t match the data.

I was not able to replicate the FFFD problem using the resend command.

[Russ Ross]

I too have pulled my hair out when dealing with what I will refer to as funky characters.

I have discovered the tester and a real interface will not produce the same outcome when dealing with funky characters.

In some cases even the real interface is stubborn when dealing with funky characters.

Sometimes this is complicated by advanced encoding schemes and UTF-8 that adds to the struggle, so I don’t envy you.

So what I’ve said so far, sort of summarizes what I see already talked about in this post, so congrats on still clinging to the mountain side.

Keep up the faith because seeing it work in hcitcl tells you it works but just have to get the agrs passed without all the extra unwanted intelligence corrupting it.

Now that you are aware of all these pitfalls but still stuck, I can share one more approach I resorted to in an extreme case to get a hard-coded funky character passed as an argument, when all else failed to work.

I created a cloverleaf lookup table and placed the funky character in the lookup table (not the x0… but the actual character using copy/paste then double check lookup table funky character on back end with hcihd), then accessing the lookup table read the funky character into an argument variable.

I know this is a high level overview of the trick I’m describing but is the best I can do at this time with my work load.

Russ Ross
RussRoss318@gmail.com

[Author]

Replies