SMAT Flat files and Cloverleaf 6.2

I believe it will be acceptable by the time we go to 6.2, however I also think a webinar and/or some good documentation on the SMATDB use should be put into place before this happens.

I concur with Keith in that it would be nice to have a better understanding of the SMATDB functionality/implications.  I suspect that by the time we migrate to 6.2 it would be acceptable as well as we are planning to move to 6.1 later this year (as long as other things don’t get in the way).

Thanks,

Jim Cobane

Henry Ford Health

I think we need to wait for more users to switch over to Smat database.

I think there needs to be better search capabilities.

I am not very experienced with SQLite.

I agree with Alice it would be nice to hold off on this until more clients (read my site) have more time to convert.

At our site, we have a ton of jobs that process SMAT files for statistical reporting and these will need to be rewritten to keep us current with Cloverleaf technology.  However, given the nature of the integration business our team never has time to reflect back upon processes in place, but are always moving forward with new initiatives, which seem to never end.

I agree, we will need to “get with the times” and SQL will be much better that the current SMAT flat files, but simply put, we just need more time.

Thanks,

I think Infor needs to understand more from their clients how we are using the SMAT files.  Without understanding this, they can’t really know if forcing us to use a DB option is better for us, the client.  

I haven’t used the SMAT tools in years, because it’s pretty bad.  I use HL7Spy, which does an amazing job of helping me analyse the data.

If I’m forced to using their DB option, I’d have to write lots of specific SQL code to reproduce the same type of queries.  I don’t really care to do this, but if they can provide the same queries in a “pre-canned” option, I might lean more towards this change.

If they segregate SMAT code in the engine, and make the interface to the logging utility a superset of parameters for SMAT and SQL, they will both improve the code and segregate SMAT for Deprecated status.

It would force a Separation of Concerns, which is a good thing….  😉

Am I right that the DB version SMAT was only introduced in CL6.1? It is not the ‘Message Archiving’ option we can see in CL6.0.2 now, right? If so, then dropping the flat file option in CL6.2 is rather fast in my opinion.

We use SMAT a lot, but straightforward through the IDE. Besides our smat-cycle scripts we don’t have scripts that use SMAT files. So for our site it wouldn’t be that big a problem. But I read lots of questions about the DB option and I wonder if it works as well as it should.

My suggestion would be to leave the flat file option in place besides the DB option until the next major upgrade (CL7?). This way all DB errors will be handled/corrected and companies have the option to try stuff in their test environments.

Dear,

We have so many scripts to handle the smart flat file that going to db file will be for us a big trouble. We will have to convert every script to handle the new format.  If you provide tools to do the extract as the same format as smart flat file (.idx, .msg) it should be manageable but we prefer to keep the smart flat file format.

Regards,

Yves

I see this is proposed for version 6.2, but has Infor mapped out the roadmap for when this version will be released and other current supported versions will reach end of life? Some more information about the timelines would be helpful in making an assessment.  

I don’t think this will be a big deal for our organization and we are actually looking forward to the possibilities of the expanded database capabilities due to the limitations of the current SMAT setup.  We have some scripts that do run against the flat files to generate “reports” and statistics, but I think it would be easier to maintain these using SQL and utilize other reporting utilities that would make it easier for users.

Greetings,

I cast a No vote for much the same reasons as other shops here who are advancing forward but never given the time to just take a breather and evaluate where we are and what can be re-designed/improved to make our support lives easier (or more manageable!).

Ok…

We have the Global Monitor 6.1.0 application implemented and it has bells and whistles to work with SMAT as well.   We need to evaluate them – it is a recent implementation with our Integrator 6.1.0.

I would like to know what INFOR plans regarding Global Monitor and 6.2 around SMAT files.

Thanks.

I voted NO as well and echo the sentiments of many others who have responded here.  

I agree with Ronald Ortiz.  Cloverleaf needs to understand how it’s customers are using SMAT before they can make informed decisions about where to take them in the future.

We do a lot of work with the existing SMAT .idx/.msg files here.  We sift through them, index them, etc.  I’m definitely not opposed to storage in SQLite – it’s a great little DB – but, like the others, need to know that my current SMAT-based tools don’t get broken.

I agree that a SQL DB mechanism has potential benefits over flat files, but there may well be drawbacks as well.  I think we need a much better understanding – and some  practical live experience – with how the new SQLite-based method works.

Some examples:

How does SQLite perform under heavy load?  Is it faster, slower or comparable to file-based SMAT?

How is concurrent SQLite access handled?  Does Cloverleaf maintain exclusive locks so that no external access to the current DB is allowed while the CL process is running?  If so, that would be bad.

Is the SQLite SMAT DB schema extensible?  If so, this would be a Very Nice Feature.

Hi all, some great feedback here.

Rob – thanks for the clarifications.

Since Cloverleaf doesn’t currently have a pruning option, I am assuming many of us would want to implement that feature ourselves.

I would need to write the SMAT DB in order to do that.

So, when I lock the DB to do my pruning, how does Cloverleaf react to that?  Will it simply wait?  Will it throw error(s) and dump message(s) to the error DB?

The reason I ask all of these questions is that I’ve spent a fair amount of time handling contention in SQLite and want to understand how Cloverleaf handles it.

When you say:

The performance actually improves as the data set gets larger, as opposed to flat file where it degrades.

… do you mean absolute performance of SQLite SMAT improves as the data set gets larger?

…or that, as SMAT data size grows, SQLite SMAT performance improves relative to a Flat Smat of the same size?

Rob,

Thanks.  This is good information.

Jim Cobane

Henry Ford Health

I also voted No on this.

We recycle SMAT files daily and organize them in directories by year, then by month. Each file has a unique name with date/time stamp.

We store about two years worth of SMAT files.

Just a few days ago I had to run a search on two years of messages for particular pattern and I just used Notepad++ search with regular expressions – no special tools needed.

It’s also quite easy to create a Tcl or Perl script to scan all SMAT files. While I understand that we could do the same with SQLite we’ll have to program scripts every time we need to search for particular pattern.

Also, if we want to store a few years of data – will we have to create multiple SQLite databases? – otherwise they will get way too big.

Sometimes we also create a text file with messages that we need to resend and then use a script we wrote years ago to generate .idx file for that – and then use SMAT tool to resend the whole thing.

The people have spoken 🙂  Thank you for your feedback.  We will plan to continue support for flat file as-is through at least 6.2.  

Be aware that new functionality (such as advanced search) will be delivered on the SMAT DB platform, so I’d encourage users to take a look at that.

Hi,

I’m just a little curious about the consequences of using SMATDB over the flat files.

Currently we have jobs that run nightly to cycle the smat files, gzip them up and archive them. At present this goes back indefinitly, so we can still get information from 2000 if required.

This job also updates a sqlite database with some metadata about the messages (ie message time, message id, account number, patient id, etc).

We had to split this database into year files to keep it a reasonable size, and there are indexes on certain columns for quick searching.

With this method, it usually takes less than 15 seconds to retrieve all of a patients history.

My main concern with using a database file would be:

Size of the database. Is there any option to use compression, where the database is still accessible. If I were to put the data into a database without compression, the disk usage is about 10 times the current size.

How well the database handles an power failure. Is the data always recoverable, or can there be fatal corruptions that loose entries other than the last entry being written at the time. The flat file is fairly simple in that if there is a half write, it can be manually/automatically edited to remove the extra data.

If there is the option of cycling the database, so an archive database is created and a fresh one started (not just deleting old data).

Elisha,

SQLite is very durable and not prone to corruption.  There’s a discussion of that on SQLite’s web site here:

<a href="https://sqlite.org/faq.html#q21&#8243; class=”bbcode_url”>https://sqlite.org/faq.html#q21

I’ve used SQLite for 10-12 years in Cloverleaf and elsewhere and don’t recall any situations where I’ve lost data.  Have your SQLite databases been prone to data loss?

Has Infor decided when/if it’s going to remove support for SMAT files (idx/msg) from Cloverleaf?

It’s stated in this post that it SMAT files will be supported through at least 6.2, but I’m looking for a more specific target if one exists.

At this time I cannot support moving to SMATDB due to a number of the same concerns stated.  Their is an educational/support curve…..that I simply cannot support right now due to project load

BOb

I am beginning to look at 6.22 as it seems to have stabilized at this level…

but…..a lot of assessement planning before beginning our moves (probably late 2018)

After flat smat files since 1996, we took the jump in 6.1 and have not looked back. They are great. They have quirks like the flat smats do, but the searching is just phenomenal across however many days you need to keep and how many files you want to search in. We even linked our main site so we can look for an account from start to finish. It saves lots of time. We cycle save weekly. We only support one large and one small hospital, so weekly works well. Resends are great too, but we found re-sending too many at once can overload, so we just send in smaller batches of 600-800 at a time.

Has Infor decided when/if it’s going to remove support for SMAT files (idx/msg) from Cloverleaf?

It’s stated in this post that it SMAT files will be supported through at least 6.2, but I’m looking for a more specific target if one exists.

We don’t plan to remove support for flat files at this time.  However, we also don’t plan to enhance the flat file functionality (e.g. HL7-aware search, ability to search across multiple files, etc).  

SMAT DB is the go-forward platform for enhancements.