Linux uudedode "No `begin’ line"

Can you post a sample HL7 message and the full TCL script that you’re using to do the conversion?

What does ZPD-3 in your message look like? Is it something like “begin 644 image.tifX0D\X0A#0V%T…”? Can you at least post the first few characters of that field?

Also, what is the output of this command in your log file (the first few lines)?

Can you manually run uudecode on the EKG0000000047.tmp after the process fails to decode it? Does it give you the same error? What’s in the .tmp file?

It looks like your script is doing the FTP from within the script. Is there a reason that you’re not calling [msgset] on the TIFF data and letting the FTP protocol driver send the file?

And you should probably put a delay in this loop (probably just say “after 1000”), because this will use lots of CPU while waiting for the file.

[code]

Looks like David is on to something here.  In your Tcl proc echo out the first 100 characters of the file just before the decode

Here’s my recommendation on how to handle this: Do everything in memory. Don’t write out temp files. Don’t call the FTP package. After you decode the message with the Tcllib library (::uuencode::decode), call msgset to  put the data in a message and route it to an outbound FTP thread. Just because you can do everything in one thread doesn’t mean that you should. By trying to do this in a script, you lose the retry features of the recovery database if your destination server is down.

In your code comment, you said that ::uuencode::decode “does not error, but does not create a readable image”. That’s probably because you aren’t dealing with binary data correctly. TCL is especially tricky because it tries to treat binary data as Unicode strings. My guess is that the decode is working correctly, but you’re writing to a file using end-of-line translation (because you haven’t used “wb” on the open flags or set the channel to binary with the fconfigure command), and it’s corrupting the data. But as I said above, I wouldn’t write to a file anyway. Just keep it in a message.

Another thing you have to watch out for is the encoding on your outbound FTP thread. The encoding on the properties tab should be “binary” and the Data Type on the FTP Options tab should be Image/Binary.

As an example, here’s some code that I use to handle an HL7 message with a Base64 encoded PDF.

“Protocol Properties / Fileset Options / Outbound / Style” should be “single”.

You could also try putting the command “encoding system identity” right before your ::decode command, but I don’t know if that will help. Also, make sure you’re restarting your process (or doing a reload on the proc) after all TCL changes.

I’ll have to think about what else we can try.

I think I might have figured it out. I had to read the man page for uudecode. What a pain.

Try this:

It looks like the ::decode command can’t deal with the “begin 644” header. You have to use the ::uuencode::uudecode command or strip off the header yourself. Also, it looks like ::uudecode returns the data in a weird list format. That’s why I put the “lindex” commands in there. There might be a cleaner way, but I don’t feel like reading more documentation.

I don’t see the attachment.

We are upgrading from v6.0 to v6.2 and migrating from AIX to Linux. I have a tcl proc that used to work but no longer works in the new environment. The error:

[sms :sms :ERR /0:  f_muse_rslt:04/14/2018 11:20:58] Tcl error:

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]    msgId   = message0

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]    proc    = ‘UICformatMusepdf’

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]    args    = ”

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]    result  = ‘uudecode: /tmp/muse/04003286920180414112058.uu: No `begin’ line’

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]    errorInfo: ‘

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–] uudecode: /tmp/muse/04003286920180414112058.uu: No `begin’ line

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]     while executing

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–] “exec uudecode -o ${path}${pdf_file} ${path}${encoded_file}”

[sms :sms :ERR /0:  f_muse_rslt:–/–/—- –:–:–]     (procedure “UICformatMusepdf” line 121)

When I ran the uudecode command from the tcl prompt, it works just fine and created the pdf file. But when it runs from within the tcl proc, the error comes up.

First let’s be clear that uuencode and uudecode is *NOT* Base64 so the base64 package will not work here/

uuencode has some very stringent rules.  Look at the Wikipedia https://en.wikipedia.org/wiki/Uuencoding

You have not posted your file yet so we cannot look at it.  One thing I note is mapping characters in the encoding message.  Have you tried to uudecode it and then do that.

It is time to turn this over to the Support guys as a bug!  

I created a simple tcl proc which receives a uuencoded ADT_A03 message and simply decodes it.

If I run the proc from the command line – hcitpstest -f eof adtencode cbb – it works fine and outputs an ADT_A03 message.  However if I run the exact same thing through the engine using Fileset/local set for eof, I get the Error “No begin line”

I don’t understand what the proc is doing in the engine that is different from hcitpstest.

In the engine I wrote the contents of the msgget to a file.  If I diff that file with the adtencode file, there is no difference.  It makes no sence unless the exec is working different in the engine.

I have attached a copy of my simple script plus the test message if someone else wants to try. Two files in test.tar.

I used CL 6.2 on Linux for testing

uuencoded messages are meant for email.  Can’t you get them to do a simple Base64?

From what I can see, its probably related to how Cloverleaf processes inbound/outbound data.

Have a look at the “identity” encoding: https://usspvlclovertch2.infor.com/viewtopic.php?t=6138

When a inbound message comes in and the inbound format is binary, Cloverleaf still uses encoding to convert the data.

The result is that any processing for binary data will be wrong.

It has to be converted to “identity” and then mapped back after processing.

When there is no manipulation of the binary data, cloverleaf will send it through ok, as it’ll map it back if both inbound/outbound are both binary.

Elisha:

As stated in my message above, I wrote the contents of the msgget to a file.  I then compared that file to the inbound file.  There was no difference.  Therefore I do not believe the encoding to be a problem

Sandy:

Try this, albeit a bit convoluted it seems to work.

I was going under the assumption that the uudecode command was doing something weird in the engine so I decided to move the command.

First I wrote a very simple, 2 line,  csh script and put it in $HCISITEDIR/scripts so there would be a path.  Make sure the script is executable.  I called mine decode.csh but name does not matter

#!/usr/bin/csh -f

uudecode -o /dev/stdout $1

My simple proc is like the below:

run {

           # Message handle

           keylget args MSGID mh

           # Assume we are passed the entire file

           set data [msgget $mh]

           set fd [open tmpxx wb]

           puts -nonewline $fd $data

           close $fd

           if {[catch {exec decode.csh tmpxx} msg]} {

               echo nERROR executing uudecode $msg

               echo To Error DBn

               msgset $mh USERDATA “Error executing uudecode: $msg”

               return “{ERROR $mh}”  

           }

           msgset $mh $msg

           file delete tmpxx

           return “{CONTINUE $mh}”

       }

I run this in the engine passing a uuencoded ADT_A03 message via Fileset/Local.  I get the A03 message out with no errors.  Like you, when I try to exec the uudecode command from the engine I get the error “No Begin Line”

Try it and let me know

I took Charlie’s code (and his C-shell script) and modified it slightly. The main difference is to read the data from ZPD-3.3, and I had to use “encoding system identity” before the system call to the script. This would not be necessary when using Tcllib uudecode (which is what I recommend).

The script works from the testing tool and from within the engine (version 6.2.1.0). I tested by routing to an outbound file thread and compared the file checksum with the testing tool version and the version produced by uudecode outside of the engine. The file looks fine with Windows Photo Viewer.

It still needs to be reported as a bug – undocumented enhancement 🙂

Why can’t you exec uudecode from within a running engine?

If they need a test case just give them the little script and test message I sent.

My $0.02

Cloverleaf 6.2 has Tcl interally support UTF-8 directly. Uudecode and uuencode should work with file only in ASCII.

uudecode: /usr/integrator/prod/muse/tmp/EKG0000000047.tmp: No `begin’ line

I guess

Tao, we tried binary and none.  What is the difference between ASCII and UTF-8?  I always thought on the system UTF-8 was ASCII.  From the Tcl manual:

“The default encoding for newly opened channels is the same platform- and locale-dependent system encoding used for interfacing with the operating system, as returned by encoding system.”

From the Tcl comand line:

tcl>encoding system

utf-8

I did not try your method since I finally pulled my head out of my butt and remember, or was reminded by another post, that there is a uudecode command in the Tcllib package. It is great since it will encode or decode a string or a file.  See:

https://core.tcl.tk/tcllib/doc/trunk/embedded/www/tcllib/files/modules/base64/uuencode.html

So, if I change my little proc to something like:

run {

           # Message handle

           keylget args MSGID mh

           # Assume we are passed an entire uuencoded file

           set data [msgget $mh]

           package require uuencode

           if {[catch {uuencode::uudecode $data} msg]} {

               echo nERROR executing uudecode $msgan

               msgset $mh USERADATA “ERROR executing uudecode $msg”

               return “{ERROR $mh}”

           }

           msgset $mh $msg

           return “{CONTINUE $mh}”

       }

Note no need to read or write files.  Works fine in the engine, no errors.

Sandy:

You should be able to change your code to fit this paradigm.  No need for scripts or to read/write files.  Should simplify things a lot.

Sorry it took so long but I am getting old    😀

I also tried the Tcllib uudecode, and you have to use uuencode::uudecode, not uuencode::decode (as I said in an earlier message). When I did it this way, I also had to do “msgset -binary” or the message would get corrupted. I can post this code if you’re interested. It’s my recommended solution.

I’m curious if making another change to Sandy’s code makes a difference. How about changing “regsub -all {\X0D\} $data “r” data” to “regsub -all {\X0D\} $data “” data”? Getting rid of the CR altogether may help because we’re using the Unix version of uudecode. You could also do this with fconfigure flags, but if you’re already doing a regsub then this is easier.

David:

I did not have to msgset -binary I guess because I was just inputting a uuencoded HL7 message.  Perhaps you decoded a picture or something else that included binary data then this would be necessary.

My code snippet did use uuencode::uudecode, and not uuencode::decode.

My thanks to you, I think it was your post that jarred my feeble memory to think about Tcllib.

About the regsub, if changing all, I prefer string map.  But to each his own.

David:

I did not have to msgset -binary I guess because I was just inputting a uuencoded HL7 message.

Sandy:

Did you get this to work? The whole thing irritated me so much I had to figure it out.  I just now had the time to do it.

First of all the uuencode package that is part of Tcllib works just fine so no need to exec an external command.  The kicker is when executing a command like:

              set decoded [uuencode::uudecode $waveform]

The comand returns a list of lists.  Since a file to be decoded can contain multiple records, the first list is a list of records decoded.  Then each list contains a 3 part sublist.  The first element is the filename, the second element is the mode and the third element contains the decoded value.  So the above command, to extract the decoded waveform would look like:

             set decoded [lindex [uuencode::uudecode $waveform] 0 2]

to return the decoded waveform.  That variable could then be written to a file like wave.tifl and you have the waveform.  When writing to the file it must be written as binary like:

            set fd [open wave.tif wb]   ;# Open file for write as binary

            puts $fd $decoded

             close $fd

You could also continue the decoded message on to another thread.  As David said you must set the threads for binary encoding.

Another thing when you are mapping values prior to the decode:

/X0D//X0A/ is CRLF from Windows and the pair should be mapped to a newline (n) not individually to CR and NL.

I set up a Fileset/local IB with binary encoding and raw routing to an OB thread.  I put the proc as an IB proc.  The OB simply writes to a file.  I ran it through the engine and was able to display the waveform from the OB file.  No errors.

Trying to figure out your Tcl was a chore.  I would suggest you learn to break it down to modules for ease of writing and maintaining.  I am not a big fan of the HL7 library as each call requires the message to be parsed again.

I have enclosed a Tcl script which you could modify to add your extra requirements.   I would think for recoverability you would be better served to send you file(s) to an OB thread and use the FTP protocol provided by Cloverleaf but that is up to you.

Take a look, let me know.  It x’ed a square for me.

Are you running 5.7?  Weird!

I notice you ran this in the ASCII mode.  Try binary.

If it works here it should work there.  If you sre indeed running 5.7 that means an older version of Tcl

I will try it here

Sandy I’ve attached the script that we’re using to convert the uuencode to base64. If you turn the debug flag on it’ll write out the pdf file which you can view with adobe. Hope this helps.

I have run this on both Linux and Windows using Cloverleaf 6.2.  If you use hcitpstest it will return a corrupted file on the output I suppose since it is trying to output UTF-8.

If you remove the comments in the script I sent you where it writes to a file called my.tif you will see a good waveform file.  Also run it through the engine, it works.

You don’t have to always tie yourself to the GUI.  Try it like:

       hcitpstest -f eof MUSE_EKG_20180413141455.txt sandy

It will return a len10 encoded file tpsout.CONTINUE but as I said it is corrupted.  Just look at the TIF written by the proc.

I can appreciate your lack of Tcl skills but if you really want to learn, as with any language, you should write all but simple code snippets  in modules  for ease of development. troubleshooting and maintenance.  If you really feel the FTP should be done in this proc, it would be easy enough to add a module (subroutine) to do that.

I ran into this issue going from 6.1 to 6.2.  Our 6.1 is on AIX and our new 6.2 is back to Linux.

We have been using this process on our previous Linux servers and CLV versions.  Only now have we run into it being an issue.  

It seems everyone is spot on regarding it being an issue with 6.2.  Hopefully it has been reported.  

We had PDF we were decoding and oddly enough deals with the same contents as Sandy’s.  

I used a good chunk of Charlie’s logic from the Sandy TCL and was able to get it to work.  I did not need a lot off what Charlie supplied.  However, I like some of his other code that I did not really need, but preferred after seeing what he did 🙂

For me all I had to do was change from using this:

exec uudecode [filename or data here]

to using the field with the data and:

package require uuencode

package require base64

set decodeWF [lindex [uuencode::uudecode $DeCryptWF] 0 2]

Thanks toa ll of you for reporting and looking into this!