Hibernate - C3P0 deadlock detection - thread run too frequent - c3p0, dead lock, detection, frequent, Hibernate, operation, thread, too

We are using Hibernate and the c3p0 connection pool library. So far, this combination works well, until recently we decided to increase maxPoolSize to 1000 and perform a lot of stress testing on our application. The peak load of our application caused the DB to respond very slowly, so the deadlock detector of c3p0 spit out APPARENT DEADLOCK warning again and again.

Based on the c3p0 documentation, we changed the maxAdministrativeTaskTime to 10 minutes , Assuming that deadlock detection occurs every 30 minutes (the code indicates that the frequency of deadlock detection is three times that of maxAdministrativeTaskTime).

However, when analyzing the c3p0 log, the deadlock detection thread runs more than 30 minutes. The relevant part of the log is attached. Surprisingly, the frequency is not uniform.

Line 573745: [Timer-2] 2013-06-26 04:47:52,492 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 573746: [Timer-2] 2013-06-26 04 :47:52,512 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 574292: [Timer-2] 2013-06-26 04: 49:12,493 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
 Line 574293: [Timer-2] 2013-06-26 04:49:12,513 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 575004: [Timer-2] 2013-06-26 04:50:32,494 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!< br />Line 575005: [Timer-2] 2013-06-26 04:50:32,511 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 576062: [Timer-2] 2013-06-26 04:51:52,495 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! pending Creating emergency threads for unassigned tasks!
Line 576063: [Timer-2] 2013-06-26 04:51:52,536 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status : 
Line 576720: [Timer-2] 2013-06-26 04:53:12,496 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 576721: [Timer-2] 2013-06-26 04:53:12,516 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
 Line 594087: [Timer-2] 2013-06-26 04:55:52,550 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
Line 594088: [Timer-2] 2013-06-26 04:55:52,569 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: < br />Line 594753: [Timer-2] 2013-06-26 04:57:12,550 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pendi ng tasks!
Line 594754: [Timer-2] 2013-06-26 04:57:12,572 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 595624: [Timer-2] 2013-06-26 04:58:32,552 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 595625: [Timer-2] 2013-06-26 04:58:32,570 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK! !! Complete Status: 
Line 596416: [Timer-2] 2013-06-26 04:59:52,552 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!! ! Creating emergency threads for unassigned pending tasks!
Line 596417: [Timer-2] 2013-06-26 04:59:52,572 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 611011: [Timer-2] 2013-06-26 05:02:22,556 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! pending Creating emergency threads for unassigned tasks!
Line 611012: [Timer-2] 2013-06-26 05:02:22,577 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status :

Can someone explain this exception?

maxAdministrativeTaskTime does not affect the thread pool’s deadlock detector; it just limits attempts to interrupt in the pool ( ) The length of a single task was previously allowed. One (hackish!, ugly!, not recommended!) The way to try to avoid APPARENT DEADLOCKs is to set it to the SHORT interval, so that slow tasks will be interrupted instead of hanging for too long. I’m sure it’s stuck. Please see some discussion here.

Thanks to the questioner (via email), I double-checked my own %$^ *&! The code and the deadlock detection interval are affected by maxAdministrativeTaskTime. As the question shows, it should be 3×maxAdministrativeTaskTime. So the mystery deepens.

It is worth understanding the role of the deadlock detector. c3p0 maintains a thread pool , numHelperThreads is very large. The deadlock detector will periodically record that all task threads in the pool are running, then sleep for a period of time, and then check again. If no task is completed, that is, the active task remains exactly the same as the last check, a deadlock is declared . c3p0 expects a certain task to complete the time interval of 10 seconds, it is not configurable now. It can be configured.

You can post some examples of APPARENT DEADLOCK you see. c3p0 prints a lot of information to help You understand what may hang.

Here are two suggestions to reduce APPARENT DEADLOCK under very heavy loads:

>The deadlocks you have observed related to Connection acqusistion What is it? If so, if you haven’t, please upgrade to c3p0-0.9.2.1 or the latest 0.9.5 pre-release version.
>Did you try to increase numHelperThreads when expanding the load? If some tasks are very slow, c3p0 will not declare a deadlock unless all threads are blocked by the slow task. If there are enough threads to keep the agile task still executing, you will not see a deadlock.

< p>I hope this helps!

We are using Hibernate and c3p0 connection pool library. So far, this combination works well, until recently we decided to increase maxPoolSize to 1000 and perform A lot of stress testing. The peak load of our application caused the DB to respond very slowly, so the deadlock detector of c3p0 spit out the APPARENT DEADLOCK warning again and again.

Based on the c3p0 documentation, we set maxAdministrativeTaskTime Change to 10 minutes, assuming that deadlock detection occurs every 30 minutes (the code indicates that the frequency of deadlock detection is three times that of maxAdministrativeTaskTime).

However, when analyzing the c3p0 log, the running frequency of the deadlock detection thread More than 30 minutes. The relevant part of the log is attached. Surprisingly, the frequency is not even.

Line 573745: [Timer-2] 2013-06-26 04 :47:52,492 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 573746: [Timer-2] 2013- 06-26 04:47:52,512 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 574292: [Timer-2] 2013-06 -26 04:49:12,493 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 574293: [ Timer-2] 2013-06-26 04:49:12,513 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 575004: [Timer -2] 2013-06-26 04:50:32,494 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 575005: [Timer-2] 2013-06-26 04:50:32,511 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 576062 : [Timer-2] 2013-06-26 04:51:52,495 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 576063: [Timer-2] 2013-06-26 04:51:52,536 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 576720: [Timer-2] 201 3-06-26 04:53:12,496 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 576721: [ Timer-2] 2013-06-26 04:53:12,516 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 594087: [Timer -2] 2013-06-26 04:55:52,550 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 594088: [Timer-2] 2013-06-26 04:55:52,569 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 594753 : [Timer-2] 2013-06-26 04:57:12,550 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 594754: [Timer-2] 2013-06-26 04:57:12,572 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
 Line 595624: [Timer-2] 2013-06-26 04:58:32,552 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
Line 595625: [Timer-2] 2013-06-26 04:58:32,570 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: < br />Line 596416: [Timer-2] 2013-06-26 04:59:52,552 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 596417: [Timer-2] 2013-06-26 04:59:52,572 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status: 
Line 611011 : [Timer-2] 2013-06-26 05:02:22,556 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Line 611012: [Timer-2] 2013-06-26 05:02:22,577 WARN [null] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@662cee3b - APPARENT DEADLOCK!!! Complete Status:

Can someone explain this anomaly?

maxAdministrativeTaskTime does not affect the thread pool’s deadlock detector; it just limits the length of a single task allowed before the pool attempts to interrupt(). A (hackish! , Ugly!, not recommended!) The way to try to avoid APPARENT DEADLOCKs is to set it to a SHORT interval, so that slow tasks will be interrupted instead of hanging for so long that the pool determines that it is stuck. See some discussion here.

You can post some examples of APPARENT DEADLOCK you see. c3p0 prints a lot of information to help You understand what may hang.

Here are two suggestions to reduce APPARENT DEADLOCK under very heavy loads:

< p>I hope this helps!

Hibernate – C3P0 deadlock detection – thread run too frequent

Leave a Comment Cancel reply