Cycling SMAT and Log Files

[Greg Tataryn]

We are looking into optimizing how our SMAT and log files are cycled and archived.

We are on version 6.1.1 and are using SMATDB. We have a cron job that runs each Saturday night. It performs a cycle save, and moves the old SMAT and log files off to an archive directory.

We are considering changing this to nightly. Keep the current week in the process directory but all prior weeks (we keep 4) will be saved off into the archive directory and compressed. Any advantage or disadvantage to doing this with SMATDB?

Additionally, since SMAT and log archiving can be configured in the GUI, is there any advantage or disadvantage to using that vs a cron job?

Topic

[Max Drown (Infor)]

Greg, do a search on the forum for cycling. There are some good, thorough discussions on the topic.

In short, yes, there are very good reasons for using SMAT DB, encryption, and auto-cycling.

-- Max Drown (Infor)

[Greg Tataryn]

I understand the benefits of the automatic cycling. I was more curious with SMATDB if there are any pros/cons to performing the cycle nightly vs weekly.

[Max Drown (Infor)]

It doesn’t cycle on a schedule, it cycles on file size. The search tool can search across live and archived files eliminating the need for scheduled backups.

-- Max Drown (Infor)

[Tom Rioux]

Greg,

We just upgraded to 6.1.2.   We use cron to cycle the smatdb files.  That option works best for us because we know each file will have messages from basically the same time period.    Cycling on size can produce varied results depending on the amount of data you push through at times.

As for weekly vs daily, with the implementation of the smatdb, we have moved most of our cycling to a weekly schedule.  So far, the only things still have on a daily schedule are those interfaces that handle device data (which pumps out thousands of messages per day) and those interfaces that include an embedded PDF (which are huge messages).

One question I do have for the INFOR staff…is there a size limitation to the smatdb files?  We encountered an issue with one of our weekly files that caused the process to crash.   The engine wasn’t able to write to the smatdb due to the file size.   At that point, cycling the smatdb file was impossible.  We had to move the .smtadb file manually.  Once that was moved out of the process directory,  the -shm and the -wal files returned to normal size.

Also,  when it crashed, I noticed that files were created with a -sec as part of the file extension.  How are those -sec file used?

Thanks…

Tom Rioux

[Max Drown (Infor)]

Thomas Rioux wrote:

… we know each file will have messages from basically the same time period.

This is no longer necessary with SMAT DB. The searching is much more powerful. It searches not only the live files, but also the historical, archived data. In the search parameters, you can specify HL7 fields, date ranges, and many other criteria, too. The messages are stored as database rows which means there are columns that can be used for searching. The tool is fast and easy to use, and is still being improved by implementing user feedback.

An common example is to search on an MRN. You specify “PID.03=” which will find all records in the entire site for this MRN, including all archived messages! You can narrow the search to specific threads or choose specific files or date ranges or whatever you need. You no loner need to know which file contains the data!

Moving away from cron and into the engine makes migrations and patching much easier.

You’ll have to check on the file size limitations by submitting a ticket on Infor Xtreme as I don’t recall the size limitations.

-- Max Drown (Infor)

[James Cobane]

We still cycle our SMAT DBs on a daily basis, even after converting from the file-based SMAT.  It keeps the file/DB sizes smaller and takes up less disk space in the file system that the engine is running; we move off the cycled files/DBs to a different file system.  I would think performance-wise having large SMAT DBs might have some impact as the engine has to write to a larger file/DB.  Just a theory.

Jim Cobane

Henry Ford Health System

[Tom Rioux]

I have to submit a ticket to just get file size limitations?  Shouldn’t that be part of the documentation?

Thanks….

[Jim Kosloskey]

Tom,

It may be the O/S which is providing the file size limit. Check with your Admin to see where that is set.

If SqlLite is being used for the DB then perhaps a check of the SqlLite support page will point out if there are limitations imposed by SqlLite.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

We have not moved to SMATDB (we are on 6.0.0)) but I would think since the default archiving is based on size, one could limit the live (engine using) consumption that way (might mean very frequent archiving I guess given a small environment).

However, if the archiving can only be to the engine’s File System then I could see a resource issue arising there.

Is it not possible to archive to a different File System?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Max Drown (Infor)]

James Cobane wrote:

We still cycle our SMAT DBs on a daily basis, even after converting from the file-based SMAT.

-- Max Drown (Infor)

[Jim Kosloskey]

From SqlLite:

Maximum Database Size

Every database consists of one or more “pages”. Within a single database, every page is the same size, but different database can have page sizes that are powers of two between 512 and 65536, inclusive. The maximum size of a database file is 2147483646 pages. At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes (140 terabytes, or 128 tebibytes, or 140,000 gigabytes or 128,000 gibibytes).

This particular upper bound is untested since the developers do not have access to hardware capable of reaching this limit. However, tests do verify that SQLite behaves correctly and sanely when a database reaches the maximum file size of the underlying filesystem (which is usually much less than the maximum theoretical database size) and when a database is unable to grow due to disk space exhaustion.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[James Cobane]

We have a few configured to cycle based on size, but we like having our daily snapshots of the saved messages.  We are “old school” that way 🙂

Jim Cobane

Henry Ford Health System

[Tom Rioux]

With the old file based SMAT, you only had the .idx and the .msg.  Now you have 3 files in the process directory.  A .smatdb, a .smatdb-shm, and a .smatdb-wal.   When you have a process that includes imbedded PDF’s, all 3 of these files can be huge.

I love the new smatbd, don’t get me wrong.   It just takes some adjusting.

[James Cobane]

Where we have automatic cycling based on size configured are specifically those threads that contain the embedded PDFs; because they do get so large.

Jim Cobane

Henry Ford Health System

[Max Drown (Infor)]

Not trying to change anyone’s mind, but I do want new users, future readers, etc. to understand the function and best practices.

-- Max Drown (Infor)

[Author]

Replies