Upoc tcl open and write multiple files based upon max file size limit

[Bill Pitts]

Hi

Since increase Covid reporting requirements – we’re having issues with our ELR results file to state DPH exceeding DPH max file size limit.

We process using upoc protocol and advanced scheduler on outbound thread.

HL7 msgs from lab stored in tmp file; upoc reads temp file and writes output on timer.

Header/trailer added to output file

I need to split the output into multiple files if file max (1.5Mb) exceeded.

I can identify number of output files needed by dividing the input file size by max file limit (and adding 1 for “overages”) but I’m having trouble opening the proper number of output channels.  I’d then write the transactions from the input file based on transaction count (I have a pretty good estimate of how many transactions I can send on output).

We’ve also looked into setting up alert on file size and calling tclproc to split file but worried that approach might truncate data msgs.

Each HL7 msg starts with MSH segment and ends with SPM segments

We’re running CLV 19.1.1 on AIX

Thanks

 

Bill

Topic

[Charlie Bursell]

If you are creating tmp files using UPoC why not simply create multiple files?

You could append to a file until max size is reached and then create a new one.  I would set the high level to about 10% less than max just to be sure.  Size the file prior to appending using the file size command and compare to a global max size.

You could create an OB thread that uses fileset/local  and then when timer goes off move the tmp files to the fileset/local directory and let the engine handle the load.

Be sure and move do not copy.  Move is an atomic operation so the engine will not try to read files before they are in the fileset/local directory

[Bill Pitts]

 

 

Current tclproc, that successfully processes one nightly file, below:

#    *********************************************************************
#    on RUN mode:    Appends msgs to temp file TUFTS_MDPH_ELR_tmp_in.txt
#
#                    Make sure to set Process name to “softlab” for TEST/PROD
#
#    on TIMER mode:  Writes header, reads/write msgs from temp file to
#                    TUFTS_MDPH_ELR_OUT.yyyymmddhhmm file, writes trailer
#
#                    Deletes .tmp file upon completion
#
#    ***********************************************************************
#
#———————————————————————————–
# Modifications:
#
#  date   init     mod desc
#
#
#———————————————————————————–

proc tpsSoftlab_MDPHbatchAuto { args } {

keylget args MODE mode              ;# Fetch mod

set dispList {}                ;# Nothing to return

#Make sure to set Process name to “softlab” for TEST/PROD
set Process “softlab”
#set Process “testing”
set Outpath “/mdph_to_send/”

switch -exact — $mode {
start {
# startup stuff
# N.B.: there may or may not be a MSGID key in args
}

run {
# ‘run’ mode always has a MSGID; fetch and process it
keylget args MSGID mh
set msg [msgget $mh]

# Get rid of all \x1c and \x0b
regsub -all — {[\x1c\x0b]} $msg {} msg

#Get the time in system seconds
set systemTime [clock seconds]
#Set the time using the hour, minute and second descriptors.
set CurDate [clock format $systemTime -format {%Y%m%d}]
set CurDateTime [clock format $systemTime -format {%Y%m%d%H%M}]

# set directory path
set root_path [exec env HCIROOT]
set site_path [exec env HCISITEDIR]
set subdir “exec/processes/$Process”
append site_path “/$subdir/”

# set the user data… for the message
set filename “TUFTS_MDPH_ELR_tmp”
set filenameextension “txt”
set fullname $site_path$filename.$filenameextension

# open tmp file, write record
set curFileID [open $fullname a]
#################  puts -nonewline $curFileID $msg\r
puts -nonewline $curFileID $msg

# close tmp File
close $curFileID

# return message
# Always KILL original
# KILL may not work in UPOC write in Cloverleaf v5.3
# Get “Unsupported Disposition” Error and msgs in Error DB
# Use CONTINUE for now, may check KILL when upgrade to Cloverleaf v5.6. complete
set dispList

}

time {
# Timer-based processing
# N.B.: there may or may not be a MSGID key in args

# Build envelope
set cloverclock [clock format [clock second] -format %Y%m%d%H%M%S]

# Original batch header formatting
# set fhs “FHS|^~\\&|Softlab|TUFTS|ELR|MDPH|$cloverclock||$cloverclock|||\r”
# set bhs “BHS|^~\\&|Softlab|TUFTS|ELR|MDPH|$cloverclock||$cloverclock|||\r”

# New batch header formatting
set fhs “FHS|^~\\&#|SCC|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”
set bhs “BHS|^~\\&|LAB|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”
## set fhs “FHS|^~\\&|SOFTLAB^2.16.840.1.113883.3.3013.77.1^ISO|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”
## set bhs “BHS|^~\\&|SOFTLAB^2.16.840.1.113883.3.3013.77.1^ISO|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”

# Sample from MDPH
# MSH|^~\&|Sample Lab System^2.16.840.1.113883.19.3.1^ISO|SampleHospital^22D0076229^CLIA|MA-MDPH^2.16.840.1.113883.19.3.2^ISO
#|MA-MDPH^2.16.840.1.113883.19.3.2^ISO|20130101123558-0400||ORU^R01^ORU_R01|SH20130101123558|P|2.5.1|||AL|AL
#|||||PHLabReport-Ack^^2.16.840.1.113883.9.11^ISO

# Sample within Softlab record — 11/5/2014
# MSH|^~\&|SOFTLAB^2.16.840.1.113883.3.3013.77.1^ISO|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|20141103095309-0500||ORU^R01^ORU_R01|570009460-2014110309|P|2.5.1|||AL|NE

# set file header
### set fileheader $fhs\r\n$bhs\r
set fileheader $fhs$bhs

# set directory path
set root_path [exec env HCIROOT]
set site_path [exec env HCISITEDIR]
set subdir “exec/processes/$Process”
append site_path “/$subdir/”

#Get the time in system seconds
set systemTime [clock seconds]
#Set the time using the hour, minute and second descriptors.
set CurDate [clock format $systemTime -format {%Y%m%d}]
set CurDateTime [clock format $systemTime -format {%Y%m%d%H%M}]

# input file parameters
# set the user data… for the message
set filein “TUFTS_MDPH_ELR_tmp”
set fileInExt “txt”
set fullnameIn $site_path$filein.$fileInExt

# set output path and filename
# NOTE: “mdph_to_send” defined as Outpath, from beginning of proc
set sub_path_out $Outpath
append site_path $sub_path_out
set fileout “TUFTS_MDPH_ELR_OUT”
set fileoutExt  $CurDateTime

set outFile $site_path$fileout.$fileoutExt

# get input
set tempfileFlagIn [file exists $fullnameIn]
if {$tempfileFlagIn == 1} {
set outFileID [open $outFile a+]
# write header
puts -nonewline $outFileID $fileheader
#echo fileheader being written = $fileheader \n
set inFileID [open $fullnameIn r]
set wholefile1 [read $inFileID]

#
# split the file into its transactions
# remove any blank transactions (extra newlines)

set BatchTrnxCount 0
foreach BatchRec [split $wholefile1 “\n”] {
if {[clength $BatchRec] == 0} {
continue
} else {
puts -nonewline $outFileID $BatchRec
#echo batchrec = $BatchRec \n
set BatchTrnxCount [regexp -all {MSH} $BatchRec]
}
}

# set file TRAILER
set bts “BTS|$BatchTrnxCount|\r”
###echo bts = $bts \n
set fts “FTS|1|\r”
###echo fts = $fts \n
set filetrailer $bts$fts
#echo filetrailer = $filetrailer \n

### Write TRAILER

set EOFcurr [eof $inFileID]
if {$EOFcurr == 1} {
puts -nonewline $outFileID $filetrailer
close $outFileID
}
### move code to inside if loop?
###close $outFileID

close $inFileID

# delete tmp Input file
file delete $fullnameIn

}

}

}

return $dispList
}

 

 

[Bill Pitts]

Update: We got this working; thanks to Tom Patton (Tufts integration) for his tcl expertise and fix.

We split the output files from the tmp input file based upon record max counts.

Kept the same UPoC and Advanced Scheduler setup

Cron calls ftp script to send output files to server that stores java tunneling utility to State DPH.

Powershell script created on that server to send all output files to State, await responses, and send email notification back to internal hospital team

Modified (new) tclproc to split source tmp batch into multiple output files below:

 

######################################################################
# Name:    tpsSoftlab_MDPHsplitBatch
# Purpose:     Batch Header & Trailer and Output File for Softlab Results to MDPH ELR
#              Need to split file into multiple files to avoid MDPH 1.5Mb max size limit
#
# Copied from tpsMDPHbatchAuto.tcl, which is used for Cerner ELR Results
#
#
# UPoC type:    tps
# Args:     tps keyedlist containing the following keys:
#           MODE    run mode (“start”, “run” or “time”)
#           MSGID   message handle
#           ARGS    user-supplied arguments:
#                   <describe user-supplied args here>
#
# Returns: tps disposition list:
#          <describe dispositions used here>
#
#
#
#    *********************************************************************
#    on RUN mode:    Appends msgs to temp file TUFTS_MDPH_ELR_tmp_in.txt
#
#                    Make sure to set Process name to “softlab” for TEST/PROD
#
#    on TIMER mode:  Writes header, reads/write msgs from temp file to
#                    TUFTS_MDPH_ELR_OUT.yyyymmddhhmm file, writes trailer
#
#                    Modified to create multiple files if suspected of approaching
#                    MDPH max file size limit of 1.5Mb
#
#                    Deletes .tmp file upon completion
#
#    ***********************************************************************
#
#———————————————————————————–
# Modifications:
#
#  date   init     mod desc
#  7/9/2020 whp   Need to split file into multiple files to avoid MDPH 1.5Mb max size limit
#                 File sizes increased since Covid-19 reporting, approx early April 2020
#  8/3/2020 tgp   adjusted this script to embedd a file# in the file name
#
#———————————————————————————–

proc tpsSoftlab_MDPHsplitBatch { args } {

keylget args MODE mode              ;# Fetch mod

set dispList {}                ;# Nothing to return

#Make sure to set Process name to “softlab” for TEST/PROD
set Process “softlab”
#set Process “testing”
set Outpath “/mdph_to_send/”

# variables needed to determine and split large files
set filelargeFlag “N”
set multiFileFlag “N”
set fileNum {1}
set filesizeMax {700000}
set recCountMax {199} ; # keep 1 less than the actual count wanted
set numFiles {0}
set OutFilesSize {0}

switch -exact — $mode {
start {
# startup stuff
# N.B.: there may or may not be a MSGID key in args
}

run {
# ‘run’ mode always has a MSGID; fetch and process it
keylget args MSGID mh
set msg [msgget $mh]

# Get rid of all \x1c and \x0b
regsub -all — {[\x1c\x0b]} $msg {} msg

#Get the time in system seconds
set systemTime [clock seconds]
#Set the time using the hour, minute and second descriptors.
set CurDate [clock format $systemTime -format {%Y%m%d}]
set CurDateTime [clock format $systemTime -format {%Y%m%d%H%M}]

# set directory path
set root_path [exec env HCIROOT]
set site_path [exec env HCISITEDIR]
set subdir “exec/processes/$Process”
append site_path “/$subdir/”

# set the user data… for the message
set filename “TUFTS_MDPH_ELR_tmp”
set filenameextension “txt”
set fullname $site_path$filename.$filenameextension

# open  the tmp file, write record
set curFileID [open $fullname a]
##fconfigure $curFileID -translation lf
## puts -nonewline $curFileID $msg\r

####puts -nonewline $curFileID $msg
puts $curFileID $msg

# close the tmp File
close $curFileID

# return message
# Always KILL original
# KILL may not work in UPOC write in Cloverleaf v5.3
# Get “Unsupported Disposition” Error and msgs in Error DB
# Use CONTINUE for now, may check KILL when upgrade to Cloverleaf v5.6. complete
set dispList

}

time {
# Timer-based processing
# N.B.: there may or may not be a MSGID key in args

# Build envelope
set cloverclock [clock format [clock second] -format %Y%m%d%H%M%S]

# New batch header formatting
set fhs “FHS|^~\\&#|SCC|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”
set bhs “BHS|^~\\&|LAB|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|$cloverclock||$cloverclock|||\r”

# Sample from MDPH
# MSH|^~\&|Sample Lab System^2.16.840.1.113883.19.3.1^ISO|SampleHospital^22D0076229^CLIA|MA-MDPH^2.16.840.1.113883.19.3.2^ISO
#|MA-MDPH^2.16.840.1.113883.19.3.2^ISO|20130101123558-0400||ORU^R01^ORU_R01|SH20130101123558|P|2.5.1|||AL|AL
#|||||PHLabReport-Ack^^2.16.840.1.113883.9.11^ISO

# Sample within Softlab record — 11/5/2014
# MSH|^~\&|SOFTLAB^2.16.840.1.113883.3.3013.77.1^ISO|TUFTS^22D0074723^CLIA|RR1MSH5|MLAB^22D0074723^CLIA|20141103095309-0500||ORU^R01^ORU_R01|570009460-2014110309|P|2.5.1|||AL|NE

# set file header
### set fileheader $fhs\r\n$bhs\r
set fileheader $fhs$bhs

# set directory path
set root_path [exec env HCIROOT]
set site_path [exec env HCISITEDIR]
set subdir “exec/processes/$Process”
append site_path “/$subdir/”

#Get the time in system seconds
set systemTime [clock seconds]
#Set the time using the hour, minute and second descriptors.
set CurDate [clock format $systemTime -format {%Y%m%d}]
set CurDateTime [clock format $systemTime -format {%Y%m%d%H%M}]

# input file parameters
# set the user data… for the message
set filein “TUFTS_MDPH_ELR_tmp”
set fileInExt “txt”
set fullnameIn $site_path$filein.$fileInExt

# set output path and filename
# NOTE: “mdph_to_send” defined as Outpath, from beginning of proc
set sub_path_out $Outpath
append site_path $sub_path_out
set fileout “TUFTS_MDPH_ELR_OUT”
set fileoutExt  $CurDateTime

# get input
set tempfileFlagIn [file exists $fullnameIn]

if {$tempfileFlagIn == 1} {

set outFile $site_path$fileout$fileNum.$fileoutExt
set outFileID [open $outFile a+]

# write header
puts -nonewline $outFileID $fileheader

# read date in from input file
set inFileID [open $fullnameIn r]
fconfigure $inFileID -buffering line

set wholefile1 [read $inFileID]
# split the file into its transactions
# remove any blank transactions (extra newlines)

set BatchTrnxCount 0
set RecCount 0

foreach BatchRec [split $wholefile1 “\n”] {
if {[clength $BatchRec] == 0} {
continue
} else {
# check file size before batchrec write
set OutFileSize [file size $outFile]
# create and write trailer, close outFileID –
# start new file if over size limit
if {$RecCount > $recCountMax} {
set bts “BTS|$RecCount|\r”
set fts “FTS|1|\r”
set filetrailer $bts$fts
puts -nonewline $outFileID $filetrailer
close $outFileID

# prep new file
incr fileNum
set outFile $site_path$fileout$fileNum.$fileoutExt
set outFileID [open $outFile a+]
set RecCount 0

# write header for new file
puts -nonewline $outFileID $fileheader
}
#  write out batch record – then check if over sized and wrap up file and start new
puts -nonewline $outFileID $BatchRec
incr RecCount
}
}

# set file TRAILER
set bts “BTS|$RecCount|\r”
set fts “FTS|1|\r”
set filetrailer $bts$fts

### Write TRAILER
set EOFcurr [eof $inFileID]
if {$EOFcurr == 1} {
puts -nonewline $outFileID $filetrailer
close $outFileID
}

close $inFileID

# delete tmp Input file
file delete $fullnameIn
} else {
echo “No Lab ELR input file exists in process directory”
}
}

}

return $dispList
}

[Author]

Replies