1*4882a593Smuzhiyun===================== 2*4882a593SmuzhiyunDAWR issues on POWER9 3*4882a593Smuzhiyun===================== 4*4882a593Smuzhiyun 5*4882a593SmuzhiyunOn POWER9 the Data Address Watchpoint Register (DAWR) can cause a checkstop 6*4882a593Smuzhiyunif it points to cache inhibited (CI) memory. Currently Linux has no way to 7*4882a593Smuzhiyundisinguish CI memory when configuring the DAWR, so (for now) the DAWR is 8*4882a593Smuzhiyundisabled by this commit:: 9*4882a593Smuzhiyun 10*4882a593Smuzhiyun commit 9654153158d3e0684a1bdb76dbababdb7111d5a0 11*4882a593Smuzhiyun Author: Michael Neuling <mikey@neuling.org> 12*4882a593Smuzhiyun Date: Tue Mar 27 15:37:24 2018 +1100 13*4882a593Smuzhiyun powerpc: Disable DAWR in the base POWER9 CPU features 14*4882a593Smuzhiyun 15*4882a593SmuzhiyunTechnical Details: 16*4882a593Smuzhiyun================== 17*4882a593Smuzhiyun 18*4882a593SmuzhiyunDAWR has 6 different ways of being set. 19*4882a593Smuzhiyun1) ptrace 20*4882a593Smuzhiyun2) h_set_mode(DAWR) 21*4882a593Smuzhiyun3) h_set_dabr() 22*4882a593Smuzhiyun4) kvmppc_set_one_reg() 23*4882a593Smuzhiyun5) xmon 24*4882a593Smuzhiyun 25*4882a593SmuzhiyunFor ptrace, we now advertise zero breakpoints on POWER9 via the 26*4882a593SmuzhiyunPPC_PTRACE_GETHWDBGINFO call. This results in GDB falling back to 27*4882a593Smuzhiyunsoftware emulation of the watchpoint (which is slow). 28*4882a593Smuzhiyun 29*4882a593Smuzhiyunh_set_mode(DAWR) and h_set_dabr() will now return an error to the 30*4882a593Smuzhiyunguest on a POWER9 host. Current Linux guests ignore this error, so 31*4882a593Smuzhiyunthey will silently not get the DAWR. 32*4882a593Smuzhiyun 33*4882a593Smuzhiyunkvmppc_set_one_reg() will store the value in the vcpu but won't 34*4882a593Smuzhiyunactually set it on POWER9 hardware. This is done so we don't break 35*4882a593Smuzhiyunmigration from POWER8 to POWER9, at the cost of silently losing the 36*4882a593SmuzhiyunDAWR on the migration. 37*4882a593Smuzhiyun 38*4882a593SmuzhiyunFor xmon, the 'bd' command will return an error on P9. 39*4882a593Smuzhiyun 40*4882a593SmuzhiyunConsequences for users 41*4882a593Smuzhiyun====================== 42*4882a593Smuzhiyun 43*4882a593SmuzhiyunFor GDB watchpoints (ie 'watch' command) on POWER9 bare metal , GDB 44*4882a593Smuzhiyunwill accept the command. Unfortunately since there is no hardware 45*4882a593Smuzhiyunsupport for the watchpoint, GDB will software emulate the watchpoint 46*4882a593Smuzhiyunmaking it run very slowly. 47*4882a593Smuzhiyun 48*4882a593SmuzhiyunThe same will also be true for any guests started on a POWER9 49*4882a593Smuzhiyunhost. The watchpoint will fail and GDB will fall back to software 50*4882a593Smuzhiyunemulation. 51*4882a593Smuzhiyun 52*4882a593SmuzhiyunIf a guest is started on a POWER8 host, GDB will accept the watchpoint 53*4882a593Smuzhiyunand configure the hardware to use the DAWR. This will run at full 54*4882a593Smuzhiyunspeed since it can use the hardware emulation. Unfortunately if this 55*4882a593Smuzhiyunguest is migrated to a POWER9 host, the watchpoint will be lost on the 56*4882a593SmuzhiyunPOWER9. Loads and stores to the watchpoint locations will not be 57*4882a593Smuzhiyuntrapped in GDB. The watchpoint is remembered, so if the guest is 58*4882a593Smuzhiyunmigrated back to the POWER8 host, it will start working again. 59*4882a593Smuzhiyun 60*4882a593SmuzhiyunForce enabling the DAWR 61*4882a593Smuzhiyun======================= 62*4882a593SmuzhiyunKernels (since ~v5.2) have an option to force enable the DAWR via:: 63*4882a593Smuzhiyun 64*4882a593Smuzhiyun echo Y > /sys/kernel/debug/powerpc/dawr_enable_dangerous 65*4882a593Smuzhiyun 66*4882a593SmuzhiyunThis enables the DAWR even on POWER9. 67*4882a593Smuzhiyun 68*4882a593SmuzhiyunThis is a dangerous setting, USE AT YOUR OWN RISK. 69*4882a593Smuzhiyun 70*4882a593SmuzhiyunSome users may not care about a bad user crashing their box 71*4882a593Smuzhiyun(ie. single user/desktop systems) and really want the DAWR. This 72*4882a593Smuzhiyunallows them to force enable DAWR. 73*4882a593Smuzhiyun 74*4882a593SmuzhiyunThis flag can also be used to disable DAWR access. Once this is 75*4882a593Smuzhiyuncleared, all DAWR access should be cleared immediately and your 76*4882a593Smuzhiyunmachine once again safe from crashing. 77*4882a593Smuzhiyun 78*4882a593SmuzhiyunUserspace may get confused by toggling this. If DAWR is force 79*4882a593Smuzhiyunenabled/disabled between getting the number of breakpoints (via 80*4882a593SmuzhiyunPTRACE_GETHWDBGINFO) and setting the breakpoint, userspace will get an 81*4882a593Smuzhiyuninconsistent view of what's available. Similarly for guests. 82*4882a593Smuzhiyun 83*4882a593SmuzhiyunFor the DAWR to be enabled in a KVM guest, the DAWR needs to be force 84*4882a593Smuzhiyunenabled in the host AND the guest. For this reason, this won't work on 85*4882a593SmuzhiyunPOWERVM as it doesn't allow the HCALL to work. Writes of 'Y' to the 86*4882a593Smuzhiyundawr_enable_dangerous file will fail if the hypervisor doesn't support 87*4882a593Smuzhiyunwriting the DAWR. 88*4882a593Smuzhiyun 89*4882a593SmuzhiyunTo double check the DAWR is working, run this kernel selftest: 90*4882a593Smuzhiyun 91*4882a593Smuzhiyun tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c 92*4882a593Smuzhiyun 93*4882a593SmuzhiyunAny errors/failures/skips mean something is wrong. 94