Files FTP’d to Cloverleaf

This topic has 7 replies, 2 voices, and was last updated 11 years, 4 months ago by Tom Rioux.

Creator

Topic
March 12, 2014 at 2:38 pm #54093
Tom Rioux
Participant
All,

We are working with Covisint to deliver various claim files. We recieve files from various vendors and send them to Covisint. The flat files that are pipe delimited with 27 columns. Each row in the file is a message and Covisint expects each row to be terminated with a CR/LF.

The files we are getting in do not all have the correct terminator. Some are CR/LF, some are CR and some are LF. There are even those that will have all CR or all LF except for the last message. The last message won’t have a terminator.

We have a shell script already in place that checks the file for a couple of things, one being that it contains 27 columns. However, I need something that can look at the file to determine if all of the terminators are CR/LF. If not, we are marking the file as bad, sending an email to the vendor to have them send the file according to specs. We DO NOT want to attempt to manipulate the file to change the terminators to the correct ones.

I have tried awk, perl, grep, sed, plus any combination of them to try to get this to work. I tried gawk and dos2unix but our unix server doesn’t recognize those commands.

Any ideas????

Tom Rioux
Creator

Topic

Viewing 6 reply threads

Author

Replies
- March 12, 2014 at 4:58 pm #80128
  David Barr
  Participant
  You could try the “file” command:
  
  Code: ~/tmp/dbarr$ file mkdas.txt mkdas.txt: ASCII text, with CRLF line terminators
  
  Or you could try something like this:
  
  Code: < mkdas.txt tr '15' | | grep -v '|$'
  
  The second command replaces all CRs with pipes, so a good file should have pipe-LF at the end of every line. Then the grep removes all lines that don’t end with pipe.
- March 12, 2014 at 5:04 pm #80129
  David Barr
  Participant
  You could also do
  Code: grep -v ‘^M$’ mkdas.txt
  , but you need to put a literal CR in the command instead of ^M. I think you can type ^V^M on the command line or in vi to do this.
- March 12, 2014 at 5:22 pm #80130
  Tom Rioux
  Participant
  David,
  
  I was trying to use grep before but I didn’t try the -v option. I will try that. Also, by using the ^M, will that catch CR/LF or just CR since this is on a UNIX box.
  
  Also…this is being done inside a shell script.
  
  Thanks…
  
  Tom
- March 12, 2014 at 5:40 pm #80131
  David Barr
  Participant
  It sounds like you need the following conditions:
  
  1. every CR is followed by an LF
  
  grep -q ‘^M.’ file.txt
  
  2. every LF is preceded by a CR
  
  grep -vq ‘^M$’ file.txt
  
  3. the file is either empty or ends with CRLF
  
  test ! -s test.txt || tail -c 2 test.txt | od | grep -q 005015
- March 12, 2014 at 8:45 pm #80132
  Tom Rioux
  Participant
  I tried your command:
  
  if grep -q ‘^M.’ file.txt ; then
  
  …..blah, blah, blah
  
  but I recieve the error: -q is not expected
  
  I try it without the -q, then it tells me my pattern is not expected.
  
  Seems like I’ve tried variations of everything and it doesnt work….or maybe I’m just terrible at shell scripting. 😀
- March 12, 2014 at 8:51 pm #80133
  David Barr
  Participant
  I assume you are using bash, ksh or sh. Try it without the brackets:
  
  if grep -q ‘^M.’ file.txt; then
  
  …
  
  Also, it has to be a literal CR character in the script. If you’re using vi, type ctrl-v ctrl-m to enter it. With emacs you can type ctrl-q ctrl-m.
- March 12, 2014 at 9:49 pm #80134
  Tom Rioux
  Participant
  Thanks David! Removing the brackets did the trick. I had tried grep before I posted my original question but couldn’t get past the errors. If I would have known it was that simple I could have had this done yesterday.
  
  I appreciate the help!
  
  Tom Rioux
Author

Replies

Viewing 6 reply threads

The forum ‘Cloverleaf’ is closed to new topics and replies.