1*4882a593Smuzhiyun=============== 2*4882a593SmuzhiyunLock Statistics 3*4882a593Smuzhiyun=============== 4*4882a593Smuzhiyun 5*4882a593SmuzhiyunWhat 6*4882a593Smuzhiyun==== 7*4882a593Smuzhiyun 8*4882a593SmuzhiyunAs the name suggests, it provides statistics on locks. 9*4882a593Smuzhiyun 10*4882a593Smuzhiyun 11*4882a593SmuzhiyunWhy 12*4882a593Smuzhiyun=== 13*4882a593Smuzhiyun 14*4882a593SmuzhiyunBecause things like lock contention can severely impact performance. 15*4882a593Smuzhiyun 16*4882a593SmuzhiyunHow 17*4882a593Smuzhiyun=== 18*4882a593Smuzhiyun 19*4882a593SmuzhiyunLockdep already has hooks in the lock functions and maps lock instances to 20*4882a593Smuzhiyunlock classes. We build on that (see Documentation/locking/lockdep-design.rst). 21*4882a593SmuzhiyunThe graph below shows the relation between the lock functions and the various 22*4882a593Smuzhiyunhooks therein:: 23*4882a593Smuzhiyun 24*4882a593Smuzhiyun __acquire 25*4882a593Smuzhiyun | 26*4882a593Smuzhiyun lock _____ 27*4882a593Smuzhiyun | \ 28*4882a593Smuzhiyun | __contended 29*4882a593Smuzhiyun | | 30*4882a593Smuzhiyun | <wait> 31*4882a593Smuzhiyun | _______/ 32*4882a593Smuzhiyun |/ 33*4882a593Smuzhiyun | 34*4882a593Smuzhiyun __acquired 35*4882a593Smuzhiyun | 36*4882a593Smuzhiyun . 37*4882a593Smuzhiyun <hold> 38*4882a593Smuzhiyun . 39*4882a593Smuzhiyun | 40*4882a593Smuzhiyun __release 41*4882a593Smuzhiyun | 42*4882a593Smuzhiyun unlock 43*4882a593Smuzhiyun 44*4882a593Smuzhiyun lock, unlock - the regular lock functions 45*4882a593Smuzhiyun __* - the hooks 46*4882a593Smuzhiyun <> - states 47*4882a593Smuzhiyun 48*4882a593SmuzhiyunWith these hooks we provide the following statistics: 49*4882a593Smuzhiyun 50*4882a593Smuzhiyun con-bounces 51*4882a593Smuzhiyun - number of lock contention that involved x-cpu data 52*4882a593Smuzhiyun contentions 53*4882a593Smuzhiyun - number of lock acquisitions that had to wait 54*4882a593Smuzhiyun wait time 55*4882a593Smuzhiyun min 56*4882a593Smuzhiyun - shortest (non-0) time we ever had to wait for a lock 57*4882a593Smuzhiyun max 58*4882a593Smuzhiyun - longest time we ever had to wait for a lock 59*4882a593Smuzhiyun total 60*4882a593Smuzhiyun - total time we spend waiting on this lock 61*4882a593Smuzhiyun avg 62*4882a593Smuzhiyun - average time spent waiting on this lock 63*4882a593Smuzhiyun acq-bounces 64*4882a593Smuzhiyun - number of lock acquisitions that involved x-cpu data 65*4882a593Smuzhiyun acquisitions 66*4882a593Smuzhiyun - number of times we took the lock 67*4882a593Smuzhiyun hold time 68*4882a593Smuzhiyun min 69*4882a593Smuzhiyun - shortest (non-0) time we ever held the lock 70*4882a593Smuzhiyun max 71*4882a593Smuzhiyun - longest time we ever held the lock 72*4882a593Smuzhiyun total 73*4882a593Smuzhiyun - total time this lock was held 74*4882a593Smuzhiyun avg 75*4882a593Smuzhiyun - average time this lock was held 76*4882a593Smuzhiyun 77*4882a593SmuzhiyunThese numbers are gathered per lock class, per read/write state (when 78*4882a593Smuzhiyunapplicable). 79*4882a593Smuzhiyun 80*4882a593SmuzhiyunIt also tracks 4 contention points per class. A contention point is a call site 81*4882a593Smuzhiyunthat had to wait on lock acquisition. 82*4882a593Smuzhiyun 83*4882a593SmuzhiyunConfiguration 84*4882a593Smuzhiyun------------- 85*4882a593Smuzhiyun 86*4882a593SmuzhiyunLock statistics are enabled via CONFIG_LOCK_STAT. 87*4882a593Smuzhiyun 88*4882a593SmuzhiyunUsage 89*4882a593Smuzhiyun----- 90*4882a593Smuzhiyun 91*4882a593SmuzhiyunEnable collection of statistics:: 92*4882a593Smuzhiyun 93*4882a593Smuzhiyun # echo 1 >/proc/sys/kernel/lock_stat 94*4882a593Smuzhiyun 95*4882a593SmuzhiyunDisable collection of statistics:: 96*4882a593Smuzhiyun 97*4882a593Smuzhiyun # echo 0 >/proc/sys/kernel/lock_stat 98*4882a593Smuzhiyun 99*4882a593SmuzhiyunLook at the current lock statistics:: 100*4882a593Smuzhiyun 101*4882a593Smuzhiyun ( line numbers not part of actual output, done for clarity in the explanation 102*4882a593Smuzhiyun below ) 103*4882a593Smuzhiyun 104*4882a593Smuzhiyun # less /proc/lock_stat 105*4882a593Smuzhiyun 106*4882a593Smuzhiyun 01 lock_stat version 0.4 107*4882a593Smuzhiyun 02----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 108*4882a593Smuzhiyun 03 class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg 109*4882a593Smuzhiyun 04----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 110*4882a593Smuzhiyun 05 111*4882a593Smuzhiyun 06 &mm->mmap_sem-W: 46 84 0.26 939.10 16371.53 194.90 47291 2922365 0.16 2220301.69 17464026916.32 5975.99 112*4882a593Smuzhiyun 07 &mm->mmap_sem-R: 37 100 1.31 299502.61 325629.52 3256.30 212344 34316685 0.10 7744.91 95016910.20 2.77 113*4882a593Smuzhiyun 08 --------------- 114*4882a593Smuzhiyun 09 &mm->mmap_sem 1 [<ffffffff811502a7>] khugepaged_scan_mm_slot+0x57/0x280 115*4882a593Smuzhiyun 10 &mm->mmap_sem 96 [<ffffffff815351c4>] __do_page_fault+0x1d4/0x510 116*4882a593Smuzhiyun 11 &mm->mmap_sem 34 [<ffffffff81113d77>] vm_mmap_pgoff+0x87/0xd0 117*4882a593Smuzhiyun 12 &mm->mmap_sem 17 [<ffffffff81127e71>] vm_munmap+0x41/0x80 118*4882a593Smuzhiyun 13 --------------- 119*4882a593Smuzhiyun 14 &mm->mmap_sem 1 [<ffffffff81046fda>] dup_mmap+0x2a/0x3f0 120*4882a593Smuzhiyun 15 &mm->mmap_sem 60 [<ffffffff81129e29>] SyS_mprotect+0xe9/0x250 121*4882a593Smuzhiyun 16 &mm->mmap_sem 41 [<ffffffff815351c4>] __do_page_fault+0x1d4/0x510 122*4882a593Smuzhiyun 17 &mm->mmap_sem 68 [<ffffffff81113d77>] vm_mmap_pgoff+0x87/0xd0 123*4882a593Smuzhiyun 18 124*4882a593Smuzhiyun 19............................................................................................................................................................................................................................. 125*4882a593Smuzhiyun 20 126*4882a593Smuzhiyun 21 unix_table_lock: 110 112 0.21 49.24 163.91 1.46 21094 66312 0.12 624.42 31589.81 0.48 127*4882a593Smuzhiyun 22 --------------- 128*4882a593Smuzhiyun 23 unix_table_lock 45 [<ffffffff8150ad8e>] unix_create1+0x16e/0x1b0 129*4882a593Smuzhiyun 24 unix_table_lock 47 [<ffffffff8150b111>] unix_release_sock+0x31/0x250 130*4882a593Smuzhiyun 25 unix_table_lock 15 [<ffffffff8150ca37>] unix_find_other+0x117/0x230 131*4882a593Smuzhiyun 26 unix_table_lock 5 [<ffffffff8150a09f>] unix_autobind+0x11f/0x1b0 132*4882a593Smuzhiyun 27 --------------- 133*4882a593Smuzhiyun 28 unix_table_lock 39 [<ffffffff8150b111>] unix_release_sock+0x31/0x250 134*4882a593Smuzhiyun 29 unix_table_lock 49 [<ffffffff8150ad8e>] unix_create1+0x16e/0x1b0 135*4882a593Smuzhiyun 30 unix_table_lock 20 [<ffffffff8150ca37>] unix_find_other+0x117/0x230 136*4882a593Smuzhiyun 31 unix_table_lock 4 [<ffffffff8150a09f>] unix_autobind+0x11f/0x1b0 137*4882a593Smuzhiyun 138*4882a593Smuzhiyun 139*4882a593SmuzhiyunThis excerpt shows the first two lock class statistics. Line 01 shows the 140*4882a593Smuzhiyunoutput version - each time the format changes this will be updated. Line 02-04 141*4882a593Smuzhiyunshow the header with column descriptions. Lines 05-18 and 20-31 show the actual 142*4882a593Smuzhiyunstatistics. These statistics come in two parts; the actual stats separated by a 143*4882a593Smuzhiyunshort separator (line 08, 13) from the contention points. 144*4882a593Smuzhiyun 145*4882a593SmuzhiyunLines 09-12 show the first 4 recorded contention points (the code 146*4882a593Smuzhiyunwhich tries to get the lock) and lines 14-17 show the first 4 recorded 147*4882a593Smuzhiyuncontended points (the lock holder). It is possible that the max 148*4882a593Smuzhiyuncon-bounces point is missing in the statistics. 149*4882a593Smuzhiyun 150*4882a593SmuzhiyunThe first lock (05-18) is a read/write lock, and shows two lines above the 151*4882a593Smuzhiyunshort separator. The contention points don't match the column descriptors, 152*4882a593Smuzhiyunthey have two: contentions and [<IP>] symbol. The second set of contention 153*4882a593Smuzhiyunpoints are the points we're contending with. 154*4882a593Smuzhiyun 155*4882a593SmuzhiyunThe integer part of the time values is in us. 156*4882a593Smuzhiyun 157*4882a593SmuzhiyunDealing with nested locks, subclasses may appear:: 158*4882a593Smuzhiyun 159*4882a593Smuzhiyun 32........................................................................................................................................................................................................................... 160*4882a593Smuzhiyun 33 161*4882a593Smuzhiyun 34 &rq->lock: 13128 13128 0.43 190.53 103881.26 7.91 97454 3453404 0.00 401.11 13224683.11 3.82 162*4882a593Smuzhiyun 35 --------- 163*4882a593Smuzhiyun 36 &rq->lock 645 [<ffffffff8103bfc4>] task_rq_lock+0x43/0x75 164*4882a593Smuzhiyun 37 &rq->lock 297 [<ffffffff8104ba65>] try_to_wake_up+0x127/0x25a 165*4882a593Smuzhiyun 38 &rq->lock 360 [<ffffffff8103c4c5>] select_task_rq_fair+0x1f0/0x74a 166*4882a593Smuzhiyun 39 &rq->lock 428 [<ffffffff81045f98>] scheduler_tick+0x46/0x1fb 167*4882a593Smuzhiyun 40 --------- 168*4882a593Smuzhiyun 41 &rq->lock 77 [<ffffffff8103bfc4>] task_rq_lock+0x43/0x75 169*4882a593Smuzhiyun 42 &rq->lock 174 [<ffffffff8104ba65>] try_to_wake_up+0x127/0x25a 170*4882a593Smuzhiyun 43 &rq->lock 4715 [<ffffffff8103ed4b>] double_rq_lock+0x42/0x54 171*4882a593Smuzhiyun 44 &rq->lock 893 [<ffffffff81340524>] schedule+0x157/0x7b8 172*4882a593Smuzhiyun 45 173*4882a593Smuzhiyun 46........................................................................................................................................................................................................................... 174*4882a593Smuzhiyun 47 175*4882a593Smuzhiyun 48 &rq->lock/1: 1526 11488 0.33 388.73 136294.31 11.86 21461 38404 0.00 37.93 109388.53 2.84 176*4882a593Smuzhiyun 49 ----------- 177*4882a593Smuzhiyun 50 &rq->lock/1 11526 [<ffffffff8103ed58>] double_rq_lock+0x4f/0x54 178*4882a593Smuzhiyun 51 ----------- 179*4882a593Smuzhiyun 52 &rq->lock/1 5645 [<ffffffff8103ed4b>] double_rq_lock+0x42/0x54 180*4882a593Smuzhiyun 53 &rq->lock/1 1224 [<ffffffff81340524>] schedule+0x157/0x7b8 181*4882a593Smuzhiyun 54 &rq->lock/1 4336 [<ffffffff8103ed58>] double_rq_lock+0x4f/0x54 182*4882a593Smuzhiyun 55 &rq->lock/1 181 [<ffffffff8104ba65>] try_to_wake_up+0x127/0x25a 183*4882a593Smuzhiyun 184*4882a593SmuzhiyunLine 48 shows statistics for the second subclass (/1) of &rq->lock class 185*4882a593Smuzhiyun(subclass starts from 0), since in this case, as line 50 suggests, 186*4882a593Smuzhiyundouble_rq_lock actually acquires a nested lock of two spinlocks. 187*4882a593Smuzhiyun 188*4882a593SmuzhiyunView the top contending locks:: 189*4882a593Smuzhiyun 190*4882a593Smuzhiyun # grep : /proc/lock_stat | head 191*4882a593Smuzhiyun clockevents_lock: 2926159 2947636 0.15 46882.81 1784540466.34 605.41 3381345 3879161 0.00 2260.97 53178395.68 13.71 192*4882a593Smuzhiyun tick_broadcast_lock: 346460 346717 0.18 2257.43 39364622.71 113.54 3642919 4242696 0.00 2263.79 49173646.60 11.59 193*4882a593Smuzhiyun &mapping->i_mmap_mutex: 203896 203899 3.36 645530.05 31767507988.39 155800.21 3361776 8893984 0.17 2254.15 14110121.02 1.59 194*4882a593Smuzhiyun &rq->lock: 135014 136909 0.18 606.09 842160.68 6.15 1540728 10436146 0.00 728.72 17606683.41 1.69 195*4882a593Smuzhiyun &(&zone->lru_lock)->rlock: 93000 94934 0.16 59.18 188253.78 1.98 1199912 3809894 0.15 391.40 3559518.81 0.93 196*4882a593Smuzhiyun tasklist_lock-W: 40667 41130 0.23 1189.42 428980.51 10.43 270278 510106 0.16 653.51 3939674.91 7.72 197*4882a593Smuzhiyun tasklist_lock-R: 21298 21305 0.20 1310.05 215511.12 10.12 186204 241258 0.14 1162.33 1179779.23 4.89 198*4882a593Smuzhiyun rcu_node_1: 47656 49022 0.16 635.41 193616.41 3.95 844888 1865423 0.00 764.26 1656226.96 0.89 199*4882a593Smuzhiyun &(&dentry->d_lockref.lock)->rlock: 39791 40179 0.15 1302.08 88851.96 2.21 2790851 12527025 0.10 1910.75 3379714.27 0.27 200*4882a593Smuzhiyun rcu_node_0: 29203 30064 0.16 786.55 1555573.00 51.74 88963 244254 0.00 398.87 428872.51 1.76 201*4882a593Smuzhiyun 202*4882a593SmuzhiyunClear the statistics:: 203*4882a593Smuzhiyun 204*4882a593Smuzhiyun # echo 0 > /proc/lock_stat 205