Flat to flat file – output content based on IF variables

[Lawrence Nelson]

Let me preface this by saying I

Lawrence Nelson
System Architect - MaineHealth IT

Topic

[Peter Heggie]

I have this web page in my favorites: <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/contents.htm&#8221; class=”bbcode_url”>http://www.tcl.tk/man/tcl8.5/TclCmd/contents.htm – I just used it to look up ‘gets’ – I think that will help you read in a line at a time instead of reading in the entire file, which you have done in the posted code.

Also, you are doing two things at once in your loop – looking at each line to determine if you want it or things in it, and also you are taking the first two lines. Because you want to avoid reading in the entire file, you can’t create a huge list variable where each element is a line. Thats too bad because that would be a lot easier. It is possible that you could code that way if your system resources can handle that much memory.

But if not, then you have to set a counter at zero and while you loop through your lines, increment your counter and if it is less than two, write those lines to your output (or put them in variables to save for later). So you will have an if statement inside your main loop to write out the first two lines or write the line if the first character of the fourth word is a ‘3’

Code:

if {($ctr < 2) || ([string index [lindex [split $line "|"] 3] 0])} {puts $outfile $line}

Your output lines end with ‘3’ – is that a problem? If you read in each line with ‘gets’, into a variable, then you can write out that variable to your output and it will not truncate after the ‘3’.

You still have to look at each line and see if the fourth word starts with ‘3’ and you are on the right track as far as splitting that line into words and using a lindex to get the fourth word and check the first character.

Also, you probably want to avoid building the entire output file in memory and then writing it all at once at the end (but if that is ok, then it will be easier to code). So the output file should be opened before the main loop and closed after the main loop, and writing one line of output should happen in the main loop.

Peter Heggie

[Jeff Dinsmore]

I’m not sure if you want to read just the first two lines of the file, or find the first two matches in the file.

So, you can prevent reading the whole file by limiting “read” to a certain number of characters like this:

Code:

set frl_data [read $frl 1000]

… or, to read one line at a time, replace your “foreach” loop with something like this:

Code:

while { [gets $newFile msg] >= 0 } {
 # do your work on msg here
 # gets returns -1 when it hits end of file
 # if you only want to read a certain number of lines, set a counter and break when you’ve read your quota
}

To do your selection from the large data file and write to the smaller file, you can do something like this:

Code:

set lineNum 0

set newFile [open vrl_adt.txt a]

foreach msg $msgList {

 if { $lineNum >= 200 } {

   # your description was a little ambiguous, but I understood that you only want to read the first two lines of the file.
#   >> if that’s not the case, you can remove this “if”

# we’ve read lines one and two – break out of the foreach loop
break
} else {

set splitMsg [split $msg |]

if { [llength $splitMsg] == 4 } {

# there are four elements in splitMsg list

switch -exact [string index [lindex $splitMsg 0] 0] {
P –
8 {
# the first character of the first element is either 8 or P
if { [string index [lindex $splitMsg 3] 0] eq “3” } {
# the first character of the value in the fourth column is 3
#  >> This is your match

# write the four elements of splitMsg to your output file – separated by “|”
                 # if you need to do any manipulation of the values, do it here prior to writing them to the file
puts $newFile [join $splitMsg |]
}
}
default {
# not 8 or P
}
}

}

}

incr lineNum

}

close $newFile

Hope that helps.

Jeff Dinsmore
Chesapeake Regional Healthcare

[Lawrence Nelson]

I’ve been away – so sorry for the late follow up and thanks.

A couple of things and I don’t know how they effect either of response that were provided to me.

1 – I only want column 1 and 2 in the output.

    The columns representing the variable are not wanted

    – I was just tucking them in there in an attempt to see

     what it was that was being ‘picked’ up.

2  – The file is 350 thousand rows. I’m looking to pick through the rows     as they are being read. The output file should be much smaller than the original file.

I will re-read both responses to see if I get farther down the road with what I’m looking for.  

Thanks!

Lawrence Nelson
System Architect - MaineHealth IT

[Lawrence Nelson]

Apologies

This

I only want the first two rows and I only want the ones that

start with 8 or P in the first position and ALSO start with 3 in what

would look like the 4th column below. (column = between the pipes)

should read

I only want the first two COLUMNS output

and I only want the ones that start with 8 or P in the

first column position 0

and

ALSO start with 3 (position 0) in the 4th column

Only Column 1 and 2 output.

Lawrence Nelson
System Architect - MaineHealth IT

[Mark Thompson]

Hi Lawrence,

You are familiar with Tcl fragments (used by Xlate code), so why not do this in a translate?  You can use the same VRL for both in and out.  Let’s say for your example the VRL has 4 fields: field1, field2, field3 & field4.

A translate can decide whether to SUPPRESS a message or not using an IF statement.

Here’s the key – you need to write a Tcl fragment that picks the first character of the first element of XlateInVals and stores it as a list in xlateOutVals.  You may want to name your proc something like xltCopyFirstChar.

Your translate code goes like this:

COPY field1 to @var1 applying xltCopyFirstChar

COPY field4 to @var4 applying xltCopyFirstChar

IF ( @var1 eq =8 || @var1 eq =P ) && ( @var4 eq =3)

THEN

 COPY field1 TO field1

 COPY field2 TO field2

ELSE

 SUPPRESS

This process is less effecient than killing messages in a TPS proc, but depending on your TCL comfort level, it may be much easier to implement.

- Mark Thompson
HealthPartners

[Author]

Replies