Output json records in correct format

[Joshua Myers]

Hey all…

I’m developing an interface that processes some input file (can be .csv, json, xml, or HL7) and creates a json object for each record. Inbound reads from local file directory, xlate runs, then outbound sends 1 record at a time via sFtp.  The problem I’m running into is sending them a correctly formatted json for multiple records.

My output file looks like this:  

{patient1 json}{patient2 json}{patient3 json}

But I need it to look like this:  

[{patient1 json},{patient2 json},{patient3 json}]

Preference is to do this all in Cloverleaf.

One thought I have is to add another thread that writes to local disk after the xlate, have the sFTP thread pick up that file, and then have tcl format it  before sending the file out…but I’m hoping to avoid doing this due to performance concerns down the road when we’re trying to process huge volumes of messages.

Is this the best way to do this in Cloverleaf or is there a better way that I’m missing?  Hope I explained this well…thanks!

Topic

[Jim Kosloskey]

So do you want all of the individual records received inbound to be one record outbound with commas separating the records?

IB –

File 1:

Rec 1

Rec 2

Rec 3

OB –

File 1:

Rec 1,Rec2,Rec3

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Joshua Myers]

Yeah, post xlate I need the output file to start with an open square bracket, terminate with a closed square bracket, and have commas seperating each json record.

[Rec1, Rec2, Rec3]

[Jim Kosloskey]

I think you will need some tcl.

If the records do not need transformation (an Xlate) then the Tcl could be at the IB TPS or the OB TPS.

If an Xlate is needed then the tcl could be post-xlate or OB tps I think.

The trick is knowing when you have the last record from the last file IB.

Maybe another way (assuming no Xlate) is to have a dirparse routine on the IB Fileset Local to issue a O/S command to combine the files to a new file then return that file name to the dirparse so the Fileset protocol picks up the concatenated file.

The challenge there is to cleanup all the files once the Fileset is finished with the concatenated file but I think that is doable.

There are probably other ways to do this with Tcl inside Cloverleaf as well.

With the clarification you provided, I think other clovertechers may be able to provide more specific advice. Those were just the thoughts off the top of my head.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Joshua Myers]

I figured out a way to make this work, but any suggestions on a way to do it better would be welcome!

1. Disk_in reads in file (for current customer, this is a .csv) line by line

2. VRL format -> xlate -> json format

3. Disk_out saves each json record to a file on local disk

4. A second disk_in reads in entire file using eof termination instead of newline

5. inbound tps proc on second disk_in surrounds entire file with [] and uses regsub to add in the comma between each json record

regsub -all {}n{}  $msgText “},n{” msgText

6.  sftp_out sends entire file to remote server

I haven’t tested it yet, but I think this may end up processing faster overall since the records are written locally and the sftp proc only needs to send one large file (vs. sending each record one at a time over sftp)

[Jim Kosloskey]

That shows promise.

Let us know how it all works out.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies