Convert large text file to smaller individual files
› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › Convert large text file to smaller individual files
- This topic has 10 replies, 3 voices, and was last updated 16 years, 2 months ago by
Michael Hertel.
-
CreatorTopic
-
October 30, 2008 at 2:55 pm #50425
I have several large text files I would like to parse into smaller individual text files for processing. The files have nl after each line and CR at the point I would like to break it down. Best way of accomplishing this? Thanks for any input. -
CreatorTopic
-
AuthorReplies
-
-
October 30, 2008 at 6:50 pm #66061
There are many ways to do this. I think it boils down to what you are most comfortable doing. You could create a script at say the unix level to break it out.
Or you could create a TCL inbound tps that could break it out.
The options are all over the place.
Me I would do it in an inbound TPS proc. But I’m not sure if you are doing any translation or if this is just a raw feed kind of thing.
-
October 30, 2008 at 6:56 pm #66062
This would be a raw feed. I would prefer tcl, just unsure of how to do it exactly. -
October 30, 2008 at 8:15 pm #66063
One way I would do it: 1) Read message.
2) msgset $mh “”
3) Split message on x0d
4) Foreach loop to create a new messages with each item in the list
5) Set the metadata in each new message for a new filename
6) lappend Continue each new message to the dispList
7) lappend Kill the original message to the dispList
Hope this helps,
-mh
-
November 10, 2008 at 5:16 pm #66064
I have the logic splitting the file apart. But when I try to continue the first message it won’t process the other files. I can see the code splitting the file apart but when I add the continue it stops. Any thoughts would be greatly appreciated.
foreach segment $segmentList {
set str [llength $segment]
if {$str > 2} {
set block [lappend block $segment]
}
if {$str < 1} {
msgset $fmh [join $block n]
set dvr_ctl_str “{FILESET {{OBFILE $file_name}}}”
msgmetaset $fmh DRIVERCTL $dvr_ctl_str
return “{CONTINUE $fmh} {KILL $mh}”
set fmh “”
set block “”
set ctr_name “${HciConnName}”
set ctr_val [format %02d [CtrNextValue $ctr_name]]
set name [fmtclock [getclock] %d%H%M]
set dot .
set file_name “IB$name$dot$ctr_val”
}
}
-
November 10, 2008 at 5:37 pm #66065
Could you post the entire proc? When you:
return “{CONTINUE $fmh} {KILL $mh}”
That stops the proc completely and gives control back to the engine at that point.
You probably want to lappend to dispList and return $dispList after all messages are created.
-
November 10, 2008 at 5:51 pm #66066
Here you go.
######################################################################
# Name: streamline_file
# Purpose:
# UPoC type: tps
# Args: tps keyedlist containing the following keys:
# MODE run mode (“start”, “run” or “time”)
# MSGID message handle
# ARGS user-supplied arguments:
#
#
# Returns: tps disposition list:
#
#
proc streamline_file { args } {
keylget args MODE mode ;# Fetch mode
global HciConnName cntFile outName fmtLen
set dispList {} ;# Nothing to return
switch -exact — $mode {
start {
# Perform special init functions
# N.B.: there may or may not be a MSGID key in args
# Initialiaze the counter with the name of the
# thread that called CtrInitCounter. The counter
# is intialized to 1, has a max value of 99 and
# takes the default rollover value of 1. See the
# Cloverleaf TCL reference on Counter Commands for
# more information.
if {[catch [CtrInitCounter “${HciConnName}” file 1 99] cerr]} {
echo “Could not initialize counter”
}
}
run {
# ‘run’ mode always has a MSGID; fetch and process it
keylget args MSGID mh
set msg [msgget $mh] ;# Get the message data
set fmh “”
set segmentList [split $msg n] ;# Split the message into a valid list
set ctr_name “${HciConnName}”
set ctr_val [format %02d [CtrNextValue $ctr_name]]
set name [fmtclock [getclock] %d%H%M]
set dot .
set file_name “IB$name$dot$ctr_val”
set block “”
set str “”
# The fileset dirver’s configuration in NetConfig
# may be overridden by setting the equivelant key
# to the desired value in the messaga meta data
# field DRIVERCTL.
#
# Here the name of the outbound field is changed
# from what is configured in NetConfig to the
# value supplied by the user by setting the user
# argument NAME.
foreach segment $segmentList {
set str [llength $segment]
if {$str > 2} {
set block [lappend block $segment]
}
if {$str < 1} {
msgset $fmh [join $block n]
set dvr_ctl_str “{FILESET {{OBFILE $file_name}}}”
msgmetaset $fmh DRIVERCTL $dvr_ctl_str
return “{CONTINUE $fmh} {KILL $mh}”
set fmh “”
set block “”
set ctr_name “${HciConnName}”
set ctr_val [format %02d [CtrNextValue $ctr_name]]
set name [fmtclock [getclock] %d%H%M]
set dot .
set file_name “IB$name$dot$ctr_val”
}
}
}
time {
# Timer-based processing
# N.B.: there may or may not be a MSGID key in args
}
shutdown {
# Doing some clean-up work
}
}
return $dispList
}
-
November 10, 2008 at 5:54 pm #66067
Plus get these error codes.
message0 message0 message0
[0:TEST] ‘message0’ (returned by ‘streamline_file ‘) does not match { }
[0:TEST] ‘message0’ (returned by ‘streamline_file ‘) does not match { }
[0:TEST] ‘message0’ (returned by ‘streamline_file ‘) does not match { }
do not know how to fix that.
-
November 10, 2008 at 6:05 pm #66068
You need to create additional messages with either msgcopy or msgcreate.
So do this:
keylget args MSGID mh
set msg [msgget $mh] ;# Get the message data
msgset $mh {}
….
set fmh [msgcopy $mh]
msgset $fmh [join $block n]
set dvr_ctl_str “{FILESET {{OBFILE $file_name}}}”
msgmetaset $fmh DRIVERCTL $dvr_ctl_str
lappend dispList “{CONTINUE $fmh}”
…
lappend dispList “{KILL $mh}”
return $dispList
}
-
November 10, 2008 at 6:28 pm #66069
Looks closer now have added the code you recommended and now it is throwing error of “bad msgId”.
-
November 10, 2008 at 6:30 pm #66070
Could you post the error or call me? 206-515-5987
-
-
AuthorReplies
- The forum ‘Cloverleaf’ is closed to new topics and replies.
