1*4882a593Smuzhiyun========================================= 2*4882a593SmuzhiyunUprobe-tracer: Uprobe-based Event Tracing 3*4882a593Smuzhiyun========================================= 4*4882a593Smuzhiyun 5*4882a593Smuzhiyun:Author: Srikar Dronamraju 6*4882a593Smuzhiyun 7*4882a593Smuzhiyun 8*4882a593SmuzhiyunOverview 9*4882a593Smuzhiyun-------- 10*4882a593SmuzhiyunUprobe based trace events are similar to kprobe based trace events. 11*4882a593SmuzhiyunTo enable this feature, build your kernel with CONFIG_UPROBE_EVENTS=y. 12*4882a593Smuzhiyun 13*4882a593SmuzhiyunSimilar to the kprobe-event tracer, this doesn't need to be activated via 14*4882a593Smuzhiyuncurrent_tracer. Instead of that, add probe points via 15*4882a593Smuzhiyun/sys/kernel/debug/tracing/uprobe_events, and enable it via 16*4882a593Smuzhiyun/sys/kernel/debug/tracing/events/uprobes/<EVENT>/enable. 17*4882a593Smuzhiyun 18*4882a593SmuzhiyunHowever unlike kprobe-event tracer, the uprobe event interface expects the 19*4882a593Smuzhiyunuser to calculate the offset of the probepoint in the object. 20*4882a593Smuzhiyun 21*4882a593SmuzhiyunYou can also use /sys/kernel/debug/tracing/dynamic_events instead of 22*4882a593Smuzhiyunuprobe_events. That interface will provide unified access to other 23*4882a593Smuzhiyundynamic events too. 24*4882a593Smuzhiyun 25*4882a593SmuzhiyunSynopsis of uprobe_tracer 26*4882a593Smuzhiyun------------------------- 27*4882a593Smuzhiyun:: 28*4882a593Smuzhiyun 29*4882a593Smuzhiyun p[:[GRP/]EVENT] PATH:OFFSET [FETCHARGS] : Set a uprobe 30*4882a593Smuzhiyun r[:[GRP/]EVENT] PATH:OFFSET [FETCHARGS] : Set a return uprobe (uretprobe) 31*4882a593Smuzhiyun p[:[GRP/]EVENT] PATH:OFFSET%return [FETCHARGS] : Set a return uprobe (uretprobe) 32*4882a593Smuzhiyun -:[GRP/]EVENT : Clear uprobe or uretprobe event 33*4882a593Smuzhiyun 34*4882a593Smuzhiyun GRP : Group name. If omitted, "uprobes" is the default value. 35*4882a593Smuzhiyun EVENT : Event name. If omitted, the event name is generated based 36*4882a593Smuzhiyun on PATH+OFFSET. 37*4882a593Smuzhiyun PATH : Path to an executable or a library. 38*4882a593Smuzhiyun OFFSET : Offset where the probe is inserted. 39*4882a593Smuzhiyun OFFSET%return : Offset where the return probe is inserted. 40*4882a593Smuzhiyun 41*4882a593Smuzhiyun FETCHARGS : Arguments. Each probe can have up to 128 args. 42*4882a593Smuzhiyun %REG : Fetch register REG 43*4882a593Smuzhiyun @ADDR : Fetch memory at ADDR (ADDR should be in userspace) 44*4882a593Smuzhiyun @+OFFSET : Fetch memory at OFFSET (OFFSET from same file as PATH) 45*4882a593Smuzhiyun $stackN : Fetch Nth entry of stack (N >= 0) 46*4882a593Smuzhiyun $stack : Fetch stack address. 47*4882a593Smuzhiyun $retval : Fetch return value.(\*1) 48*4882a593Smuzhiyun $comm : Fetch current task comm. 49*4882a593Smuzhiyun +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*2)(\*3) 50*4882a593Smuzhiyun \IMM : Store an immediate value to the argument. 51*4882a593Smuzhiyun NAME=FETCHARG : Set NAME as the argument name of FETCHARG. 52*4882a593Smuzhiyun FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types 53*4882a593Smuzhiyun (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types 54*4882a593Smuzhiyun (x8/x16/x32/x64), "string" and bitfield are supported. 55*4882a593Smuzhiyun 56*4882a593Smuzhiyun (\*1) only for return probe. 57*4882a593Smuzhiyun (\*2) this is useful for fetching a field of data structures. 58*4882a593Smuzhiyun (\*3) Unlike kprobe event, "u" prefix will just be ignored, becuse uprobe 59*4882a593Smuzhiyun events can access only user-space memory. 60*4882a593Smuzhiyun 61*4882a593SmuzhiyunTypes 62*4882a593Smuzhiyun----- 63*4882a593SmuzhiyunSeveral types are supported for fetch-args. Uprobe tracer will access memory 64*4882a593Smuzhiyunby given type. Prefix 's' and 'u' means those types are signed and unsigned 65*4882a593Smuzhiyunrespectively. 'x' prefix implies it is unsigned. Traced arguments are shown 66*4882a593Smuzhiyunin decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32' 67*4882a593Smuzhiyunor 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and 68*4882a593Smuzhiyunx86-64 uses x64). 69*4882a593SmuzhiyunString type is a special type, which fetches a "null-terminated" string from 70*4882a593Smuzhiyunuser space. 71*4882a593SmuzhiyunBitfield is another special type, which takes 3 parameters, bit-width, bit- 72*4882a593Smuzhiyunoffset, and container-size (usually 32). The syntax is:: 73*4882a593Smuzhiyun 74*4882a593Smuzhiyun b<bit-width>@<bit-offset>/<container-size> 75*4882a593Smuzhiyun 76*4882a593SmuzhiyunFor $comm, the default type is "string"; any other type is invalid. 77*4882a593Smuzhiyun 78*4882a593Smuzhiyun 79*4882a593SmuzhiyunEvent Profiling 80*4882a593Smuzhiyun--------------- 81*4882a593SmuzhiyunYou can check the total number of probe hits per event via 82*4882a593Smuzhiyun/sys/kernel/debug/tracing/uprobe_profile. The first column is the filename, 83*4882a593Smuzhiyunthe second is the event name, the third is the number of probe hits. 84*4882a593Smuzhiyun 85*4882a593SmuzhiyunUsage examples 86*4882a593Smuzhiyun-------------- 87*4882a593Smuzhiyun * Add a probe as a new uprobe event, write a new definition to uprobe_events 88*4882a593Smuzhiyun as below (sets a uprobe at an offset of 0x4245c0 in the executable /bin/bash):: 89*4882a593Smuzhiyun 90*4882a593Smuzhiyun echo 'p /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events 91*4882a593Smuzhiyun 92*4882a593Smuzhiyun * Add a probe as a new uretprobe event:: 93*4882a593Smuzhiyun 94*4882a593Smuzhiyun echo 'r /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events 95*4882a593Smuzhiyun 96*4882a593Smuzhiyun * Unset registered event:: 97*4882a593Smuzhiyun 98*4882a593Smuzhiyun echo '-:p_bash_0x4245c0' >> /sys/kernel/debug/tracing/uprobe_events 99*4882a593Smuzhiyun 100*4882a593Smuzhiyun * Print out the events that are registered:: 101*4882a593Smuzhiyun 102*4882a593Smuzhiyun cat /sys/kernel/debug/tracing/uprobe_events 103*4882a593Smuzhiyun 104*4882a593Smuzhiyun * Clear all events:: 105*4882a593Smuzhiyun 106*4882a593Smuzhiyun echo > /sys/kernel/debug/tracing/uprobe_events 107*4882a593Smuzhiyun 108*4882a593SmuzhiyunFollowing example shows how to dump the instruction pointer and %ax register 109*4882a593Smuzhiyunat the probed text address. Probe zfree function in /bin/zsh:: 110*4882a593Smuzhiyun 111*4882a593Smuzhiyun # cd /sys/kernel/debug/tracing/ 112*4882a593Smuzhiyun # cat /proc/`pgrep zsh`/maps | grep /bin/zsh | grep r-xp 113*4882a593Smuzhiyun 00400000-0048a000 r-xp 00000000 08:03 130904 /bin/zsh 114*4882a593Smuzhiyun # objdump -T /bin/zsh | grep -w zfree 115*4882a593Smuzhiyun 0000000000446420 g DF .text 0000000000000012 Base zfree 116*4882a593Smuzhiyun 117*4882a593Smuzhiyun0x46420 is the offset of zfree in object /bin/zsh that is loaded at 118*4882a593Smuzhiyun0x00400000. Hence the command to uprobe would be:: 119*4882a593Smuzhiyun 120*4882a593Smuzhiyun # echo 'p:zfree_entry /bin/zsh:0x46420 %ip %ax' > uprobe_events 121*4882a593Smuzhiyun 122*4882a593SmuzhiyunAnd the same for the uretprobe would be:: 123*4882a593Smuzhiyun 124*4882a593Smuzhiyun # echo 'r:zfree_exit /bin/zsh:0x46420 %ip %ax' >> uprobe_events 125*4882a593Smuzhiyun 126*4882a593Smuzhiyun.. note:: User has to explicitly calculate the offset of the probe-point 127*4882a593Smuzhiyun in the object. 128*4882a593Smuzhiyun 129*4882a593SmuzhiyunWe can see the events that are registered by looking at the uprobe_events file. 130*4882a593Smuzhiyun:: 131*4882a593Smuzhiyun 132*4882a593Smuzhiyun # cat uprobe_events 133*4882a593Smuzhiyun p:uprobes/zfree_entry /bin/zsh:0x00046420 arg1=%ip arg2=%ax 134*4882a593Smuzhiyun r:uprobes/zfree_exit /bin/zsh:0x00046420 arg1=%ip arg2=%ax 135*4882a593Smuzhiyun 136*4882a593SmuzhiyunFormat of events can be seen by viewing the file events/uprobes/zfree_entry/format. 137*4882a593Smuzhiyun:: 138*4882a593Smuzhiyun 139*4882a593Smuzhiyun # cat events/uprobes/zfree_entry/format 140*4882a593Smuzhiyun name: zfree_entry 141*4882a593Smuzhiyun ID: 922 142*4882a593Smuzhiyun format: 143*4882a593Smuzhiyun field:unsigned short common_type; offset:0; size:2; signed:0; 144*4882a593Smuzhiyun field:unsigned char common_flags; offset:2; size:1; signed:0; 145*4882a593Smuzhiyun field:unsigned char common_preempt_count; offset:3; size:1; signed:0; 146*4882a593Smuzhiyun field:int common_pid; offset:4; size:4; signed:1; 147*4882a593Smuzhiyun field:int common_padding; offset:8; size:4; signed:1; 148*4882a593Smuzhiyun 149*4882a593Smuzhiyun field:unsigned long __probe_ip; offset:12; size:4; signed:0; 150*4882a593Smuzhiyun field:u32 arg1; offset:16; size:4; signed:0; 151*4882a593Smuzhiyun field:u32 arg2; offset:20; size:4; signed:0; 152*4882a593Smuzhiyun 153*4882a593Smuzhiyun print fmt: "(%lx) arg1=%lx arg2=%lx", REC->__probe_ip, REC->arg1, REC->arg2 154*4882a593Smuzhiyun 155*4882a593SmuzhiyunRight after definition, each event is disabled by default. For tracing these 156*4882a593Smuzhiyunevents, you need to enable it by:: 157*4882a593Smuzhiyun 158*4882a593Smuzhiyun # echo 1 > events/uprobes/enable 159*4882a593Smuzhiyun 160*4882a593SmuzhiyunLets start tracing, sleep for some time and stop tracing. 161*4882a593Smuzhiyun:: 162*4882a593Smuzhiyun 163*4882a593Smuzhiyun # echo 1 > tracing_on 164*4882a593Smuzhiyun # sleep 20 165*4882a593Smuzhiyun # echo 0 > tracing_on 166*4882a593Smuzhiyun 167*4882a593SmuzhiyunAlso, you can disable the event by:: 168*4882a593Smuzhiyun 169*4882a593Smuzhiyun # echo 0 > events/uprobes/enable 170*4882a593Smuzhiyun 171*4882a593SmuzhiyunAnd you can see the traced information via /sys/kernel/debug/tracing/trace. 172*4882a593Smuzhiyun:: 173*4882a593Smuzhiyun 174*4882a593Smuzhiyun # cat trace 175*4882a593Smuzhiyun # tracer: nop 176*4882a593Smuzhiyun # 177*4882a593Smuzhiyun # TASK-PID CPU# TIMESTAMP FUNCTION 178*4882a593Smuzhiyun # | | | | | 179*4882a593Smuzhiyun zsh-24842 [006] 258544.995456: zfree_entry: (0x446420) arg1=446420 arg2=79 180*4882a593Smuzhiyun zsh-24842 [007] 258545.000270: zfree_exit: (0x446540 <- 0x446420) arg1=446540 arg2=0 181*4882a593Smuzhiyun zsh-24842 [002] 258545.043929: zfree_entry: (0x446420) arg1=446420 arg2=79 182*4882a593Smuzhiyun zsh-24842 [004] 258547.046129: zfree_exit: (0x446540 <- 0x446420) arg1=446540 arg2=0 183*4882a593Smuzhiyun 184*4882a593SmuzhiyunOutput shows us uprobe was triggered for a pid 24842 with ip being 0x446420 185*4882a593Smuzhiyunand contents of ax register being 79. And uretprobe was triggered with ip at 186*4882a593Smuzhiyun0x446540 with counterpart function entry at 0x446420. 187