PROTOCOL:fileset-local save file after processing

[Jeffkohl]

Have an inbound thread set to PROTOCOL:fileset-local , once it reads the file, the file disapears would like to save the file after is is processed. I did not see an option on the screen using 2022.09. Could not find anything in the help files how ever I typed my question in to Chat GPT and came back with this:

When using the Fileset-Local protocol in Cloverleaf, the handling of a file after processing depends on how the protocol is configured. Here’s a breakdown of what happens to an inbound file after it’s been read:

After Cloverleaf processes a file from the Directory to Read, it will move or delete it based on the Post-Processing configuration:

📁 Key Configuration Fields (NetConfig > Inbound Thread > Protocol: fileset-local)

| Field | Purpose | Default Behavior |
| ————————— | ——————————————————– | ————————- |
| **Directory to Read** | Source directory for incoming files. | Files are read from here. |
| **Post-Process** | What to do with the file after processing. | Default is delete. |
| **Directory for Bad Files** | Where to move unreadable or invalid files. | Only used on error. |
| **Directory for OK Files** | Optional directory to move successfully processed files. | Must be explicitly set. |

I take what I find with GP chat with a grain of salt, but it found that information out there so there must be some information referencing this.

Anybody know how to save the file and move to another location after it is processed?

Thank,

Topic

[David Ma]

You can add “tps_fileset_no_delete.tcl” in ” FTP Protocol Prosperities” -> “Fileset Options” -> TPS -> Deletion

INFOR provided following codes to me.

 

proc tps_fileset_no_delete { args } {
global HciConnName
set module “tps_fileset_no_delete/$HciConnName”
set debug 1
set dispList {}
keylget args MODE mode

switch -exact — $mode {
start {}

run {
keylget args MSGID mh
set fileList [msgget $mh]
if {$debug} {echo DEBUG: fileList: $fileList}
set outList {}
if {$debug} {echo DEBUG: outList: $outList}
msgset $mh $outList
if {$debug} {msgdump $mh}
lappend dispList “CONTINUE $mh”
}

time {}
shutdown {}
default {}
}
}

 

[Paul Bishop]

What David said is correct.  Here is a little more information.  In the Properties window for fileset-local in the TPS, there is a place to put a tcl proc for parsing the directory and for deleting the files.    For the parsing proc, it is a list of all the files that were found in the folder that is in the Directories field.  For the deleting proc, it is a list of files that were processed by the engine.  This is a proc that I use to move (rename) files to an archive folder after they have been processed.  I have a log file and echoes in the proc that you don’t necessarily need – I just had them there in case I had to do some research at a later point.

proc move2archive_argument { args } {
keylget args MODE mode ;# Fetch mode

switch -exact — $mode {
start {
return “”
}

run {
# Get message handle
keylget args MSGID mh

# get the user arguments
set uargs {} ; keylget args ARGS uargs ;# fetch user-supplied arguments
set arg_dest_dir “” ; keylget args ARGS.DESTDIR arg_dest_dir

# Check for correct context
keylget args CONTEXT ctx

if {$ctx ne “fileset_ibdel”} {
echo \nERROR proc used in wrong context
echo Context should be fileset_ibdel
echo Proc called in: $ctx\n
return “{CONTINUE $mh}”
}

# In this context the message handed to this proc
# by the driver in not a data message. Rather
# it is a list of file names from the directory
# the fileset driver is configured to read.
#
# The list of files to process is accessed
# in the same way data messages in other contexts
# are accessed.
#
# The list is manipulated and returned
#
# The fileset driver now processes the files in
# order they appear in the returned message list.

set msg_list [msgget $mh]
set file_time_stamp [clock format [file mtime $msg_list] -format %Y%m%d%H%M%S]
set current_date [clock format [clock seconds] -format %Y%m%d%H%M%S]
set ofileid [open file_process.log a]
puts $ofileid “$msg_list\t$file_time_stamp\t$current_date”
close $ofileid
echo \nmove2archive_argument: msg_list: $msg_list\n
set destination_file $arg_dest_dir
# echo \ndestination_file $destination_file\n

file rename -force $msg_list $destination_file

# Put modified list in the message handle
msgset $mh $msg_list
return {}
}
}
}

Paul Bishop
Carle Foundation Hospital
Urbana, IL

[Jason Russell]

I love it when I come in and someone asked the same question I was about to.

First off, thank you for the answers above. I’m probably going to grab Paul’s code and modify a bit.

However, is there a ‘default’ tps that is used, or is file deletion simply written into the engine itself?

edit: Also, I’m not seeing the return of a dispList in some form, are these not necessary? I see it in Paul’s code for the error (IE: Nothing happens), but not after everything gets moved.

• This reply was modified 4 weeks, 1 day ago by Jason Russell.
• This reply was modified 4 weeks, 1 day ago by Jason Russell.

[Jason Russell]

Dear Cloverleaf, please let me edit my message over and over to cover for my mistakes:

I think my edit is answered by the comments, but it is still unclear to me why you would have: return “{CONTINUE $mh}” in the error but a blank return at the end of the command. That return near the end would stop the command before the displist return is processed.

• This reply was modified 4 weeks, 1 day ago by Jason Russell.

[Jeffkohl]

Thanks everybody what is the syntax for  when you pass through the ARGS

I used: {DESTDIR{/hci/cis2022.09/integrator/hmh_none_clin/data/archive}}

It did not like it and it kept the source file in the folder where it reads from and got stuck in a loop processing the same file with the messages over and over again

[Jason Russell]

Jeff, I simplified Paul’s script a bit, and it does what you’re looking for. You still have to rename the file for it to work properly. The files I use simply get named with ‘<filename>.processed’ in the same directory they were stored. I’m going to do something more akin to what Paul did on the FTPs, but this I’m using for local files.

The main issue I found with his is an errant return that didn’t return the disposition list which I believe is why you’re getting repeated files. Make sure the “return {}” is removed so it can finish processing the file properly.

# Get message handle
keylget args MSGID mh

# Check for correct context
keylget args CONTEXT ctx

if {$ctx != “fileset_ibdel”} {
echo “\nERROR proc used in wrong context”
echo “Context should be fileset_ibdel”
echo “Proc called in: $ctx\n”
return {CONTINUE $mh}
}

# In this context the message handed to this proc
# by the driver in not a data message. Rather
# it is a list of file names from the directory
# the fileset driver is configured to read.
#
# The list of files to process is accessed
# in the same way data messages in other contexts
# are accessed.
#
# The list is manipulated and returned
#
# The fileset driver now processes the files in
# order they appear in the returned message list.

set msg_list [msgget $mh]
set ofileid [open file_process.log a]
echo “Inbound File: $msg_list”
#set destination_file $arg_dest_dir
set destination_file $msg_list.processed
#echo “ofile: $msg_list.processed”

file rename -force $msg_list $destination_file

# Put modified list in the message handle
msgset $mh $msg_list

lappend dispList “CONTINUE $mh”

• This reply was modified 3 weeks, 5 days ago by Jason Russell.

[Jeffkohl]

Jason,

I am new to TCL can do basic hl7 adjustments but this one is new to me

This is the full TCL

Where am I assigning the the location the file should go after it is processed?

Is there anything not needed

proc test_move { args } {
keylget args MODE mode ;# Fetch mode

switch -exact — $mode {
start {
return “”
}

run {
# Get message handle
keylget args MSGID mh

# Check for correct context
keylget args CONTEXT ctx

if {$ctx != “fileset_ibdel”} {
echo “\nERROR proc used in wrong context”
echo “Context should be fileset_ibdel”
echo “Proc called in: $ctx\n”
return {CONTINUE $mh}
}

# In this context the message handed to this proc
# by the driver in not a data message. Rather
# it is a list of file names from the directory
# the fileset driver is configured to read.
#
# The list of files to process is accessed
# in the same way data messages in other contexts
# are accessed.
#
# The list is manipulated and returned
#
# The fileset driver now processes the files in
# order they appear in the returned message list.

set msg_list [msgget $mh]
set ofileid [open file_process.log a]
echo “Inbound File: $msg_list”
#set destination_file $arg_dest_dir
set destination_file $msg_list.processed
#echo “ofile: $msg_list.processed”

file rename -force $msg_list $destination_file

# Put modified list in the message handle
msgset $mh $msg_list

lappend dispList “CONTINUE $mh”
}
}
}

[Jason Russell]

** This is given assuming you’re writing in Linux, not windows. If you’re in Windows you’ll have to adjust the slash direction (/ vs \) and file path.

My script renames the file in place. $msg_list contains the full path, and I’m simply appending ‘.processed’ so it doesn’t get picked up by the file-path glob/regex. So I pick up *.txt normally, so having it end in *.txt.processed means it won’t get picked up again.

file rename -force /opt/cloverleaf/cis2022.09/integrator/sd_lab1/indata/file_to_run.txt /opt/cloverleaf/cis2022.09/integrator/sd_lab1/indata/file_to_run.txt.processed

If you want to change the folder completely and put it in an archival folder, you would have to get the file name by stripping off everything but the last ‘/’ ( set fileName [lindex [split [$msg_list] /] end] ), and set the full path in the destination. Paul got his ‘archive’ path by passing it in. You can code it in if you want.

Also, I haven’t had much time to play with the script since I put it in place, but you may want to change file rename to file copy, as I think cloverleaf will automatically clean up the file regardless, and I get an error saying it can’t find the original file.

[Author]

Replies