Using split command for Large txt file

Homepage Clovertech Forums Cloverleaf Using split command for Large txt file

  • Creator
    Topic
  • #110140
    William McDavid
    Participant

    So I’m trying to split a file that has over 20k HL7 messages but it’s just creating a file that is the same size. The command I’m using is split -5000 filename. I’m told this should create files with 5k messages in each file. So I should have approximately 4 files but I’m only getting the one that is the size as the original file.

Viewing 2 reply threads
  • Author
    Replies
    • #110142
      Robert Kersemakers
      Participant

      What OS are you on?

      On RHEL you need to use parameter -l if you deviate from the default 1000 lines. So

      split -l5000 <filename>

      should give you the result you need. Use

      man split

      for the manual.

      Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

      • #110144
        William McDavid
        Participant

        Unix AIX 6.1

        This gave me the same result file splitting to a new file of the same size.

    • #110158
      David Barr
      Participant

      Try putting a space in the command: split -l 5000. If that doesn’t work, then it’s probably a limitation of the AIX version of this command. I’ve run into issues with AIX and HPUX utilities not dealing well with files that contain really long lines. You may have to write a script to split the file apart.

    • #110284
      Robert Kersemakers
      Participant

      Just a thought: maybe the command doesn’t split on lines because it can find only one.

      If the file contains CL lenght-encoded HL7 messages with only <CR> separating the segments and no <LF> or <CR><LF> to indicate the end of a line, then the command is working as it should.

      Have a look at the file with hcihexdump or notepad++ to see if this is the case. Or try to split a normal text-file and see if that works.

       

      Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

Viewing 2 reply threads
  • You must be logged in to reply to this topic.

Forum Statistics

Registered Users
5,117
Forums
28
Topics
9,292
Replies
34,432
Topic Tags
286
Empty Topic Tags
10