Using CL as a file transfer service

Jared,

Here at Henry Ford, we use “MoveIt” as an enterprise file transfer tool.  This tool has alot of capability around notifications, error handling, etc. so we can leave that kind of stuff off of Cloverleaf.  We have many interfaces that interact with this tool (i.e. call a script to initiate a MoveIt transfer in the sendOK context, etc.) and also send and receive files to/from our engine via this tool.  I agree that using Cloverleaf as the FTP transfer tool is not the best use of the engine.

Hope this helps.

Jim Cobane

Henry Ford Health

Since we have plenty of connections, and our server sits pretty much idle most the time, I actually utilize it to move quite a few files around.  This way we don’t need a separate system and we get more use out of what we already have.  It really depends just on if you want to do it that way or not.  I do have items like this in their own site/process just to keep things clean.

We as well use Cloverleaf to copy files from one directory to another for several systems. These are mostly files that are sent once a  day or a few times a week, so no real load there. It’s also nice to have the files/messages in SMAT, so you can see what is sent without having to create a separate backup of the files.

As we copy these files from one directory to another, we can use the same protocol for inbound as for outbound and therefore can do this copy-action within one thread (route the message to itself).

It was probably five years ago that I heard a statistic at a conference that over 50% of all interfaces were batch file transfers. I don’t think that number has gone down much since then.

Looking at some other vendors in this space that I have worked with, like Websphere MQ File Mover, MetaStorm / CommerceQuest PM4D, SeeBeyond/JCAPS and Sterling File Transfer, they have varying levels of ease of use and extra features. SeeBeyond’s EDI tool was incredibly cumbersome; it was better to create your own FT tool with JCAPS, but still it had to be developed. MQ is primitve, command-line, easy to use but a very simple set of ‘extra functions’ like pre- and post-activities, and ability to call from the command line; no operational GUI and at least one MQ server $$. PM4D was the easiest to use but a bit difficult to setup. Had a great operational GUI with granular security; with a simple script it could be called from the command line. It required an MQ server at each end point $$. Sterling’s newer offering has a great, customizable operational and administrative GUI that helps onboard external customers, but still relies on the underlying model of a central mailbox that won’t send the file to the final destination by itself.

Cloverleaf is in the middle as far as FT functionality, flexibility and ease of use. I would have put it at the top if it had an operational GUI that displays individual file transfers.

There is a lot of overlap between Cloverleaf and pure FT products, so also is the overlap in knowledge and experience in the people that use them. A lot of the bells and whistles in pure FT products are already in Cloverleaf, and part of its core strengths is the protocol management and overall component architecture, and as mentioned above it has excellent data archiving and debugging capabilities, as well as alerting tools.

Is it a downside to create a protocol thread for each endpoint? Maybe, but it can be useful operationally to be able to look at the monitor and tell if an endpoint is accessible.

With a small investment in the creation of a consistent approach, a cookie cutter template and accompanying scripts, you could create a repeatable process that does not take a lot of implementation effort for each interface. Some things you could add are: writing a database record for each step of the transfer, storing suitable meta-data for each transfer, allowing someone to write a browser-based GUI to track transfers. If you had the IB add-on, that browser app could be running on the Cloverleaf server, and you could add all kinds of capabilities to such a browser app, like user invocation or restart of a transfer. Another thing to add would be a command-line script to invoke a specific transfer, assuming your architecture is generic and ‘invokable’.

I don’t want to speak for anyone else, but I like Cloverleaf more than any other integration product because of of its flexibility – it is a toolman’s tool. I see a lot of creativity and knowledge in Clovertech and I know all this stuff is doable. But also probably like a lot of you, my company’s resources are so loaded and constrained that we don’t have a lot of time to build something like this (and management prefers buy vs. build).

Its too bad we can’t get a bunch of users together to build a shareable set of code that could implement such a solution.

With tcl, using Cloverleaf as a FT engine is very powerful. You can do anything your organization can dream up. 🙂

In Cloverleaf 6.0 we added the BOX functionality. This allows a user to quickly and easily import and export Cloverleaf objects such as threads, xlates, tclprocs, etc. This makes moving code from test to prod much easier and provides a way to share code with the user community.

We also have a module called Cloverleaf Secure Courier (CSC) that makes sending files or messages safely outside the four walls of the hospital quickly and easily. The messages are fully encrypted and protected. It eliminates the need for a VPN tunnel.

Greetings,

Running 5.8.5.0 and 5.7R2 (soon to be all 5.8.5.0) on AIX 6.1 TL 7.

The Cloverleaf Integrator is great for real time transfers of messages between applications.  However, we have found that using the FTP protocol is limited at best.  The canned protocol cannot handle Windows’ style files with its embedded spaces.  You could write custom drivers in TclCurl (we have done a couple) or TCL in a UPOC style protocol driver

to facilitate file transfers however it is not convenient and cookie cutter.

For batch file transfers we purchased the GlobalScape product (TumbleWeed is another alternative) which serves the purposes of configuring and transferring batch files using the usual set of secure protocols.  Gui based configuration with its clients on Windows.

The product runs on a Windows server.

Separates the functionality of live versus batch into the products that have that functionality as its strength.

In all fairness we have not evaluated the 6.0 Integrator or Secure Courier.

But then you are looking for what shops have done in your research and thought to contribute here towards your quest.

You can use the tcl ftp package instead of the built-in ftp protocol. The FTp package is very good.

Max and company,

If you are referring to the FTP package that is available using declarative “package require ftp” in a TCL, per INFOR R & D it is officially unsupported.   We ran into problems when upgrading from 5.7R2 on our DEV server to 5.8.5.0.  Consistently failed on get/close operations to a Windows FTP server. When we approached INFOR support the response was unsupported but try this other unsupported Tcl FTP package.

We found another solution:  a Perl NET package used in an external script (ksh) to manage the files.

Unless there is a supported Tcl FTP package that we do not know about?

Thanks.

There’s a lot of great stuff that’s not supported. 🙂 But, yes, I am talking about “package require ftp”.

This package comes installed with Cloverleaf, so don’t know why it wouldn’t be supported. I’ve used it a ton in the past.

I agree that FTP itself can be difficult to make cookie-cutter. I worked on a team that wrote an FTP tool in Java in SeeBeyond and we encountered many ‘non-standard’ endpoints that required tinkering deep in the Java code. We did not solve the Windows embedded space in file name problem – just requested that the file names be changed to use underscores. Which could easily be a non-starter in some situations.

I’m referring to Cloverleaf providing the tooling to create a cookie-cutter implementation and invocation solution. Out of the box the FTP protocol may not support 100% of the scenarios out there. Perhaps a tcl curl tool box could be supported by the community.

Cloverleaf already provides these features relating to file transfers:

– GUI and cmd-line definition of source and targets

– pre- and post-processing exit points

– file name preprocessing

– optional processing arguments like ascii/binary

– scheduled execution and directory monitoring

– logging, alerting and debugging

Cloverleaf also provides upocs with tcl hooks.

I don’t know the difficulties of creating curl scripts for FTP or dealing with embedded spaces. My thought is, and I know this may be a big assumption, is that the protocol pdls / curl tcl could be written to accomodate ‘non-standard’ scenarios. Standard answer – write a pdl. I don’t know if that is realistic.

But the rest of it is realistic. The biggest hurdle from an efficiency standpoint is the requirement to have a different thread for each directory. That can be quite a lot of threads and quite a lot of icons.

I think everything else can be engineered:

– a command-line trigger mechanism that returns a condition code, to fit into a larger business process

– post-xfer file size comparison/verification

– recording the meta-data

– a command-line interpreter that converts a simple FTP interface definition request into the Cloverleaf configuration commands

– notifications

The first thing I’ll throw out there, besides the above, is that what gets invoked is a defined interface. You would not enter in the source or target names or options – all those are defined and stored somewhere. You only execute the interface. Just my three cents.

Using MOVEit here too.  (James – I’m very interested in how your IFs interact with it.)

In general, managed file transfer (MFT) can offer a lot of accessible functionality to your customers, remove barriers, create efficiencies, etc.    I continue to be amazed at the extent we use MOVEit.

At the time (2006), I also found Sterling Commerce attractive for its BPM features.  Apparently IBM shared my taste (they bought it).  Too pricey and outside our scope at the time.   I still think our org may well have found those features a great facilitator of business processes.

We use Automate.  Pretty easy to setup a new scheduled task to pickup a file from one server and SFTP it to a destination.  The tool has a pretty good GUI to create the scheduled tasks along with associate file handling commands, etc.  Not sure if Cloverleaf can do SFTP without having to purchase the Secure Courier add-on?

We use Moveit. There are a few processes that have a lot of interaction between Cloverleaf and Moveit.

One of our systems, Chartmaxx, receives PDF files through FTP, but they have to receive an HL7 message over a TCP/IP socket before they will process each file. We have Moveit FTP the files to Chartmaxx, then a script on Moveit creates the HL7 message, FTPs the HL7 message to Cloverleaf, a fileset-local interface reads the message into Cloverleaf and sends it to Chartmaxx.

Cloverleaf is improving as a tool for building file based interfaces but it still needs some work.

The addition of metadata to the saved message files was a big improvment; this makes manual resends on file interfaces much easier. It would be nice if Cloverleaf had a GUI for reviewing transactions over file interface that only showed the filenames without the distraction of the file contents.

It would be nice if Cloverleaf would separate the protocol configuration of the thread from the processing configuration so that you could define a host then build multiple interfaces to that host without adding a bunch of redundant configuration.

Having a wizard to build simple FTP interfaces would also help. We have a lot of analysts with minimal training who can build interfaces in Moveit, and it would take much more training for them to be able to build simple interfaces in Cloverleaf.

The addition of metadata to the saved message files was a big improvment; this makes manual resends on file interfaces much easier. It would be nice if Cloverleaf had a GUI for reviewing transactions over file interface that only showed the filenames without the distraction of the file contents.

I think this could be done now with existing functionality. The meta-data could be read from smat files and stored in sqlite. A simple browser app hosted on the same machine could read the sqlite data and display it nicely. Not sure about using the browser app to actually invoke something, but I’ll bet it could push a ‘command’ message into a local file and be picked up by a ‘command’ thread.

Having a wizard to build simple FTP interfaces would also help. We have a lot of analysts with minimal training who can build interfaces in Moveit, and it would take much more training for them to be able to build simple interfaces in Cloverleaf.

Again, another local browser app that can take in user directives for configuring an interface, and then write the appropriate CL command-line commands. I know it is not easy to translate a simple FTP command syntax into the complex CL definitions, but all the pieces are there.

It would be nice if Cloverleaf would separate the protocol configuration of the thread from the processing configuration so that you could define a host then build multiple interfaces to that host without adding a bunch of redundant configuration.

This is the hard one. Maybe there is a nifty way to avoid changing the large amount of code that supports the current architecture. Maybe build a shell target thread which allows multiple instances to be used without being locked into the same target directory?

Group contributions and peer-review could help to avoid stumbles and save time.

Is it worth it? We don’t use Cloverleaf for any file transfer unless there is an HL7 translation, so there are a lot of FTP jobs here outside of CL. Most of that goes in and out of the mainframe, so it is controlled by the mf job scheduler. Moving the transfers to CL control adds extra hops and complexity because the mf transfers are part of a larger business process. It seems like unnecessary complexity to move the xfer to CL, and force the mf job stream to add steps to invoke the transfer in CL and then somehow get a notification back that the CL xfer completed. Also, the mf is either the source or target of most transfers, and moving the transfer to the CL server could add an extra FTP hop (but as it turns out a majority of the transfers actually do go to the CL server, so maybe its not that bad). It would be better if a mainframe step could execute and in real time reach out and invoke the CL-managed file transfer and go into a wait until it is finished (and get a return code). That keeps the business process lean, but I’m not sure about holding a wait across platforms.

I think this could be done now with existing functionality. The meta-data could be read from smat files and stored in sqlite. A simple browser app hosted on the same machine could read the sqlite data and display it nicely. Not sure about using the browser app to actually invoke something, but I’ll bet it could push a ‘command’ message into a local file and be picked up by a ‘command’ thread.

Most Cloverleaf clients don’t have the time or expertise to build this type of functionality into the engine, and even if we do then it isn’t always as polished or as well integrated as software that includes these features natively.

We don’t have funding for new products; we may not have funding for some existing products.

Polish and ease of use is important, especially when different staff members are setting things up. I’d like to think this can be addressed with a nice, simple web app. If we really need bells and whistles, then we should find funding to buy the dedicated products.

I think the limitation regarding configuration can be addressed by an enhancement to Cloverleaf where in the FTP Fileset protocol thread, Fileset Properties, a new TPS on the Inbound Properties pane can be specified to set the directory name. It is the responsibility of the tps to determine if and what to change the directory name to. The directory would be changed before the DirParse and Delete TPS run. For the outbound side, existing UPOCs can be used to set the DRIVERCTL OBFILE and OBDIR.

I would create a thread for every server, with the one thread used for both sending and receiving. I would use the same new DirName TPS to go out and check a SQLite table for any pending requests and get information like the directory name. At the appropriate UPOCs I would update a status table, again using a generic tcl.

One thing I can’t see yet is how to tag each transfer with a unique ID that can follow this transfer through every stage, to be used when recording status. Perhaps the v6.0 features of messaging tracing could be used.