Messages getting stuck in state 7

[Rehman Masood]

Hi Guys, I am a little stumped with this one. I need to load somewhere around 200 million results for conversion. The inbound data is in files and I am trying to channel it to 10 different connections (the receiving applications has created 10 different interfaces for us). I have set the inbound in Cloverleaf to 4 threads that will read 4 different files. They are set to read 800 messages/20 seconds. If I run just 1 thread, I can see that there are 800 messages that get read in as state 2, then I can see them getting translated to state 5, then state 7 and eventually written out. I see the thread also then grabs the next 800 messages and processes them. The problem I have is when I start the 2nd inbound thread, I can see that there are now 1600 messages being picked up but once they get to state 7, the threads start loading more messages and messages just stay in state 7. If I start all 4 inbound threads, same thing happens. I should also mention that all 4 inbound threads are in 4 different processes.

Below is some information on the status of recovery database at about 5 second intervals,

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

  1 11

  1 16

1526 2

352 5

  1 6

6169 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 21 10

  1 11

1742 2

147 5

6182 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

1876 2

 40 5

6199 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

  2 1

 17 10

  5 11

  5 16

544 2

1260 5

  4 6

5977 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 21 10

413 2

1299 5

  3 6

6131 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

488 2

979 5

 12 6

6447 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

555 2

730 5

 12 6

6696 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

606 2

562 5

 12 6

6854 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 22 10

676 2

352 5

  9 6

7064 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

  1 1

 22 10

  2 11

  6 16

1500 2

 64 5

7853 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

  3 1

 21 10

  6 11

  7 16

1387 2

239 5

7687 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

  1 1

 18 10

  8 11

  1 14

  5 16

1105 2

558 5

7536 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 17 10

  7 11

  5 16

755 2

942 5

7373 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

  1 1

 13 10

  8 11

  1 14

  4 16

375 2

1362 5

7222 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 23 10

1476 2

173 5

  7 6

8592 7

  1 Prio

$ hcidbdump -r | awk ‘{ print $7 }’ | sort | uniq -c

  6

  1

---

 23 10

1631 2

173 5

  7 6

8592 7

  1 Prio

Topic

[James Cobane]

You may want to change your read parameters to read less messages more frequently rather than big chunks at longer intervals.  You may also want to play with translation throttling to regulate things.  The engine will always consume messages as priority over sending them.  So, if you are processing a lot of inbound data, it will want to process inbound before pushing outbound.

Jim Cobane

Henry Ford Health

[Rob Lindsey]

I agree with James.  Smaller chunks of data at the times you have.  Also change your Translation throttling so that the engine will release the data instead of always reading data in.  Right now we have our values at Min 10, Max 15, 100% and that seems to pass out the correct number of data messages on the outbound side instead of allowing the data to build up in state 5.  Of course the number of msgs read inbound is going to depend on a number of factors of your hardware.  You might have to play around with those numbers.

Sorry I can not be more specific.

Rob

[Author]

Replies