Clovertech

Output json records in correct format

› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › Output json records in correct format

  • Creator
    Topic
  • January 31, 2018 at 10:30 pm #55602
    Joshua Myers
    Participant

      Hey all…

      I’m developing an interface that processes some input file (can be .csv, json, xml, or HL7) and creates a json object for each record. Inbound reads from local file directory, xlate runs, then outbound sends 1 record at a time via sFtp.  The problem I’m running into is sending them a correctly formatted json for multiple records.

      My output file looks like this:  

      {patient1 json}{patient2 json}{patient3 json}

      But I need it to look like this:  

      [{patient1 json},{patient2 json},{patient3 json}]

      Preference is to do this all in Cloverleaf.

      One thought I have is to add another thread that writes to local disk after the xlate, have the sFTP thread pick up that file, and then have tcl format it  before sending the file out…but I’m hoping to avoid doing this due to performance concerns down the road when we’re trying to process huge volumes of messages.

      Is this the best way to do this in Cloverleaf or is there a better way that I’m missing?  Hope I explained this well…thanks!

    • Creator
      Topic
    Viewing 4 reply threads
    • Author
      Replies
      • January 31, 2018 at 11:35 pm #85880
        Jim Kosloskey
        Participant

          So do you want all of the individual records received inbound to be one record outbound with commas separating the records?

          IB –

          File 1:

          Rec 1

          Rec 2

          Rec 3

          OB –

          File 1:

          Rec 1,Rec2,Rec3

          email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

        • February 1, 2018 at 2:16 pm #85881
          Joshua Myers
          Participant

            Yeah, post xlate I need the output file to start with an open square bracket, terminate with a closed square bracket, and have commas seperating each json record.

            [Rec1, Rec2, Rec3]

          • February 1, 2018 at 2:35 pm #85882
            Jim Kosloskey
            Participant

              I think you will need some tcl.

              If the records do not need transformation (an Xlate) then the Tcl could be at the IB TPS or the OB TPS.

              If an Xlate is needed then the tcl could be post-xlate or OB tps I think.

              The trick is knowing when you have the last record from the last file IB.

              Maybe another way (assuming no Xlate) is to have a dirparse routine on the IB Fileset Local to issue a O/S command to combine the files to a new file then return that file name to the dirparse so the Fileset protocol picks up the concatenated file.

              The challenge there is to cleanup all the files once the Fileset is finished with the concatenated file but I think that is doable.

              There are probably other ways to do this with Tcl inside Cloverleaf as well.

              With the clarification you provided, I think other clovertechers may be able to provide more specific advice. Those were just the thoughts off the top of my head.

              email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

            • February 1, 2018 at 4:05 pm #85883
              Joshua Myers
              Participant

                I figured out a way to make this work, but any suggestions on a way to do it better would be welcome!

                1. Disk_in reads in file (for current customer, this is a .csv) line by line

                2. VRL format -> xlate -> json format

                3. Disk_out saves each json record to a file on local disk

                4. A second disk_in reads in entire file using eof termination instead of newline

                5. inbound tps proc on second disk_in surrounds entire file with [] and uses regsub to add in the comma between each json record

                regsub -all {}n{}  $msgText “},n{” msgText

                6.  sftp_out sends entire file to remote server

                I haven’t tested it yet, but I think this may end up processing faster overall since the records are written locally and the sftp proc only needs to send one large file (vs. sending each record one at a time over sftp)

              • February 1, 2018 at 4:10 pm #85884
                Jim Kosloskey
                Participant

                  That shows promise.

                  Let us know how it all works out.

                  email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

              • Author
                Replies
              Viewing 4 reply threads
              • The forum ‘Cloverleaf’ is closed to new topics and replies.