SMAT DB corruption….known issue? How to fix?

[Mike Keys]

We had an issue yesterday where one of our main ADT inbound *.smatdb files became corrupt, or at least we suspect it did.

Note: We have the process set to automatically cycle the SMAT file when it reaches a 10000 Kb limit.

Now that the *.smatdb files for this process have been archived (nightly script), I can see that if I deselect the *.smatdb file we suspect is corrupt, then we can pull up the HL7 messages with the SMAT Database tool. Including the suspect corrupt *.smatdb results in “No messages were matched.”

I have a ticket open with Infor but have some questions maybe the community can answer:

1. What causes the corruption? We have 5 members in our interface team group and it is not uncommon for two of us to look at the same SMAT db files at the same time to pull out a message to save off or resend. We do not experience a lock issue unless two people are trying to look at the SAME exact smat record. Are we to believe that having multiple people looking at the same SMAT files will cause corruption?

2. Is there a way to repair or recover messages from a corrupt *.smatdb file? The SMAT db file in question hold several hours of ADT data from a peak time during the morning.

3. What is Infor doing to address this? We are on 6.1.2. Is this a known issue that is fixed or addressed in the 6.2 release?

4. How do other organizations address having multiple team members working with SMAT? Do you allow only one person at a time to look at a particular SMAT file? This would be very counter-productive in our environment.

5. Does having SMAT encryption turned on for a site increase the risk of having SMAT DB files become corrupt?

Topic

[Mike Keys]

What is Error code 26?

smat:RMI TCP Connection(74509)-10.8.121.130: ERR: 02/21/2017 11:49:36: Failed to attach /hci/cis6.1/integrator/adt_prod/exec/processes/REG/SmatHistory/EMR_IA_REG.20170221105042.smatdb with conn -1733907592

smat:RMI TCP Connection(74509)-10.8.121.130: ERR: 02/21/2017 11:49:36: Error code 26 returned.

smat:RMI TCP Connection(74509)-10.8.121.130: ERR: 02/21/2017 11:49:36: File being opened does not appear to be an SQLite database file. Enctypiton key might be incorrect

smat:RMI TCP Connection(74509)-10.8.121.130: ERR: 02/21/2017 11:49:36: Ignore this file and not attach it.

[Peter Heggie]

I think there is more than one kind of smatdb corruption or sudden availability issues happening with 6.1.2.

Usually we have it when either multiple people have the smatdb open or when at least one person has it open and a new message is received on that thread.

We have had different error messages – we have not kept track. All resulted in the smatdb messages being unavailable.

Sometimes we cycle-saved the thread, pushing the questionable smatdb into SmatHistory. After doing this, sometimes the saved smatdb was immediately available, and sometimes not. When not, I would run a very simple SQL query against the smatdb SQLite database file like this:

frsmatdbname=$1

smatdbdir=$2

echo ”     smatdb frsmatdbname: $frsmatdbname”

echo ”     smatdb smatdbdir: $smatdbdir”

cd $smatdbdir

TMPSQL=”/tmp/hci_sqlite_$frsmatdbname.sql”

SQL=”attach ‘$frsmatdbname’ as db1;”

SQL=”$SQL SELECT count(*) FROM db1.smat_msgs;”

echo “${SQL}” > $TMPSQL

echo “${SQL}”

echo “select all from db1”                  

cat $TMPSQL | sqlite $tosmatdbname

And this would result in the .wal and .shm files disappearing and the smatdb file becoming available to search.

We have seen several variations so it is hard to keep track unless you are diligent in noting all of the conditions when it happens (and I’m not sure what all the conditions are).

We do not have 6.2 up yet.

Peter Heggie

[Peter Heggie]

sorry the last line of that script should be:

cat $TMPSQL | sqlite $frsmatdbname

Peter Heggie

[Brenda Carpenter]

We have another smat database (I work with Peter Heggie) that is corrupt and the script is not working to reset it.

Errors being received on it, it’s an outbound thread:

prod:prod:INFO/0:xto_soacom_ordr:05/02/2017 00:30:12] Starting protocol thread xto_soacom_ordr as tid 2.

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Error returned from SMAT database: database corruption at line 1364 of [091570e46d]. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Error returned from SMAT database: database corruption at line 1403 of [091570e46d]. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Error returned from SMAT database: statement aborts at 5: [] . Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Error returned from SMAT database: database disk image is malformed. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Unable access db xto_soacom_ord_ob. Error 11:’database disk image is malformed’

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Internal Init failed for SMAT xto_soacom_ord_ob

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 00:30:12] Initializing secondary for xto_soacom_ord_ob when not present

Additional errors:

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: database corruption at line 1364 of [091570e46d]. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: database corruption at line 1403 of [091570e46d]. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: statement aborts at 5: [] . Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: database disk image is malformed. Error code is 11

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Unable access db xto_soacom_ord_ob. Error 11:’database disk image is malformed’

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Internal Init failed for SMAT xto_soacom_ord_ob

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Initializing secondary for xto_soacom_ord_ob when  present

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: API call with NULL database connection pointer. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: misuse at line 44 of [091570e46d]. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error 21 ‘(NULL)’ while executing ATTACH ‘xto_soacom_ord_ob.smatdb-sec’ AS sec; in xto_soacom_ord_ob

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: API call with NULL database connection pointer. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: misuse at line 44 of [091570e46d]. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error 21 ‘(NULL)’ while executing ATTACH ‘xto_soacom_ord_ob.smatdb-sec’ AS sec KEY ‘ancinprd’; in xto_soacom_ord_ob

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: API call with NULL database connection pointer. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error returned from SMAT database: misuse at line 44 of [091570e46d]. Error code is 21

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error 21 ‘(NULL)’ while executing DETACH sec; in xto_soacom_ord_ob

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Initializing secondary for xto_soacom_ord_ob when not present

[pd  :open:ERR /0:xto_soacom_ordr:05/02/2017 09:13:52] Error 21: Secondary collapse failed for xto_soacom_ord_ob

[Author]

Replies