Files FTP’d to Cloverleaf

[Tom Rioux]

All,

We are working with Covisint to deliver various claim files.   We recieve files from various vendors and send them to Covisint.   The flat files that are pipe delimited with 27 columns.   Each row in the file is a message and Covisint expects each row to be terminated with a CR/LF.

The files we are getting in do not all have the correct terminator.  Some are CR/LF, some are CR and some are LF.   There are even those that will have all CR or all LF except for the last message.  The last message won’t have a terminator.

We have a shell script already in place that checks the file for a couple of things, one being that it contains 27 columns.   However, I need something that can look at the file to determine if all of the terminators are CR/LF.  If not, we are marking the file as bad, sending an email to the vendor to have them send the file according to specs.   We DO NOT want to attempt to manipulate the file to change the terminators to the correct ones.

I have tried awk, perl, grep, sed, plus any combination of them to try to get this to work.   I tried gawk and dos2unix but our unix server doesn’t recognize those commands.  

Any ideas????

Tom Rioux

Topic

[David Barr]

You could try the “file” command:

Code:

~/tmp/dbarr$ file mkdas.txt
mkdas.txt: ASCII text, with CRLF line terminators

Or you could try something like this:

Code:

< mkdas.txt tr '15' | | grep -v '|$'

The second command replaces all CRs with pipes, so a good file should have pipe-LF at the end of every line. Then the grep removes all lines that don’t end with pipe.

[David Barr]

You could also do

Code:

grep -v ‘^M$’ mkdas.txt

, but you need to put a literal CR in the command instead of ^M. I think you can type ^V^M on the command line or in vi to do this.

[Tom Rioux]

David,

I was trying to use grep before but I didn’t try the -v option.   I will try that.  Also, by using the ^M, will that catch CR/LF or just CR since this is on a UNIX box.

Also…this is being done inside a shell script.

Thanks…

Tom

[David Barr]

It sounds like you need the following conditions:

1. every CR is followed by an LF

   grep -q ‘^M.’ file.txt

2. every LF is preceded by a CR

   grep -vq ‘^M$’ file.txt

3. the file is either empty or ends with CRLF

   test ! -s test.txt || tail -c 2 test.txt | od | grep -q 005015

[Tom Rioux]

I tried your command:

if grep -q ‘^M.’ file.txt ; then

…..blah, blah, blah

but I recieve the error:   -q is not expected

I try it without the -q, then it tells me my pattern is not expected.

Seems like I’ve tried variations of everything and it doesnt work….or maybe I’m just terrible at shell scripting.   😀

[David Barr]

I assume you are using bash, ksh or sh. Try it without the brackets:

if grep -q ‘^M.’ file.txt; then

 …

Also, it has to be a literal CR character in the script. If you’re using vi, type ctrl-v ctrl-m to enter it. With emacs you can type ctrl-q ctrl-m.

[Tom Rioux]

Thanks David!  Removing the brackets did the trick.   I had tried grep before I posted my original question but couldn’t get past the errors.   If I would have known it was that simple I could have had this done yesterday.

I appreciate the help!

Tom Rioux

[Author]

Replies