feat(cpufeat): add support for PMUv3p9

Armv8.9 introduced the FEAT_PMUV3P9 extension, which allows finer
grained control over EL0 usage of PMU registers. This is controlled by
the new PMUACR_EL1 sys

feat(cpufeat): add support for PMUv3p9

Armv8.9 introduced the FEAT_PMUV3P9 extension, which allows finer
grained control over EL0 usage of PMU registers. This is controlled by
the new PMUACR_EL1 system register, access to which is guarded by the
MDCR_EL3.EnPM2 bit. We should set this bit to avoid a trap into EL3 when
lower ELs access this register.

Add the required bits and pieces to make this feature usable:
- Add the CPUID and MDCR_EL3 bit definitions associated with
FEAT_PMUV3P9.
- Extend the existing PMU feature check to allow v9 now as well. This
is fine since we don't context switch PMU registers at all, so we
don't need to do much except to flip the MDCR bit:
- Set the EnPM2 bit in pmuv3_enable, so the feature is usuable in
non-secure world (and there only).
- Handle the MDCR bit for the ARCH_FEATURE_AVAILABILITY feature.

Please note that MDCR_EL3.EnPM2 guards other system registers as well,
for other PMU related new architecture features.

Change-Id: I288ca15f5c9efd336c64477d1c6fe9543613e238
Signed-off-by: Andre Przywara <andre.przywara@arm.com>

show more ...

fix(plat): remove fvp_r

The platform has not been maintained for some years and is generally
broken. Remove it to avoid confusion.

Change-Id: I93d832d51e114689ec79969af5d96071a03f4a88
Signed-off-by

fix(plat): remove fvp_r

The platform has not been maintained for some years and is generally
broken. Remove it to avoid confusion.

Change-Id: I93d832d51e114689ec79969af5d96071a03f4a88
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "chore(cm): add MDCR_EL3.RLTE to context management" into integration

Merge changes from topic "mec" into integration

* changes:
feat(qemu): add plat_rmmd_mecid_key_update()
feat(rmmd): add RMM_MECID_KEY_UPDATE call

chore(cm): add MDCR_EL3.RLTE to context management

The bit is already implicitly zero so no functional change. Adding it
helps fully describe how we expect FEAT_TRF to behave.

Change-Id: If7a7881e2

chore(cm): add MDCR_EL3.RLTE to context management

The bit is already implicitly zero so no functional change. Adding it
helps fully describe how we expect FEAT_TRF to behave.

Change-Id: If7a7881e2b50188222ce46265b432d658a664c75
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(rmmd): add RMM_MECID_KEY_UPDATE call

With this addition, TF-A now has an SMC call to handle the
update of MEC keys associated to MECIDs.

The behavior of this newly added call is empty for now

feat(rmmd): add RMM_MECID_KEY_UPDATE call

With this addition, TF-A now has an SMC call to handle the
update of MEC keys associated to MECIDs.

The behavior of this newly added call is empty for now until an
implementation for the MPE (Memory Protection Engine) driver is
available. Only parameter sanitization has been implemented.

Signed-off-by: Tushar Khandelwal <tushar.khandelwal@arm.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Juan Pablo Conde <juanpablo.conde@arm.com>
Change-Id: I2a969310b47e8c6da1817a79be0cd56158c6efc3

show more ...

Merge "feat(rmmd): add FEAT_MEC support" into integration

feat(rmmd): add FEAT_MEC support

This patch provides architectural support for further use of
Memory Encryption Contexts (MEC) by declaring the necessary
registers, bits, masks, helpers and values a

feat(rmmd): add FEAT_MEC support

This patch provides architectural support for further use of
Memory Encryption Contexts (MEC) by declaring the necessary
registers, bits, masks, helpers and values and modifying the
necessary registers to enable FEAT_MEC.

Signed-off-by: Tushar Khandelwal <tushar.khandelwal@arm.com>
Signed-off-by: Juan Pablo Conde <juanpablo.conde@arm.com>
Change-Id: I670dbfcef46e131dcbf3a0b927467ebf6f438fa4

show more ...

fix(cpufeat): include FEAT_MOPS declaration in aarch32 header

This patch adds the missing is_feat_mops_supported() declaration
in aarch32 header.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakas

fix(cpufeat): include FEAT_MOPS declaration in aarch32 header

This patch adds the missing is_feat_mops_supported() declaration
in aarch32 header.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I875f65defe23912351f9ef18555a5b0a0e53717d

show more ...

fix(cpufeat): avoid using mrrs/msrr for tspd

tspd compiles with `arch_helpers.h` and when FEAT_D128 is enabled
read/writes to D128 impacted registers will provide 128-bit
mrrs/msrr read/write implem

fix(cpufeat): avoid using mrrs/msrr for tspd

tspd compiles with `arch_helpers.h` and when FEAT_D128 is enabled
read/writes to D128 impacted registers will provide 128-bit
mrrs/msrr read/write implementation.

However FEAT_D128 implementation with SCR_EL3.D128en is set only
for lower-EL Non-Secure world. When tspd is chosen as the SPD target,
it builds tsp as well. This S-EL1 payload, used for testing,
inadvertently uses mrrs/msrr read/write implementation in
`modify_el1_common_regs` helper function. This eventually leads
to a panic.

Group all D128 impacted registers and avoid using mrrs/msrr read/write
implementation for tspd builds.

Change-Id: Ic0ed3a901ffa65f9447cae08951defbadee3e02a
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context

SVE and SME aren't enabled symmetrically for all worlds, but EL3 needs
to context switch them nonetheless. Previously,

perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context

SVE and SME aren't enabled symmetrically for all worlds, but EL3 needs
to context switch them nonetheless. Previously, this had to happen by
writing the enable bits just before reading/writing the relevant
context. But since the introduction of root context, this need not be
the case. We can have these enables always be present for EL3 and save
on some work (and ISBs!) on every context switch.

We can also hoist ZCR_EL3 to a never changing register, as we set its
value to be identical for every world, which happens to be the one we
want for EL3 too.

Change-Id: I3d950e72049a298008205ba32f230d5a5c02f8b0
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness in that this is rather slow. A large part of it is written in
assembly, making it opaque to the compiler for optimisations. The
future proofness requires reading registers that are effectively
`volatile`, making it even harder for the compiler, as well as adding
lots of implicit barriers, making it hard for the microarchitecutre to
optimise as well.

We can make a few assumptions, checked by a few well placed asserts, and
remove a lot of this burden. For a start, at the moment there are 4
group 0 counters with static assignments. Contexting them is a trivial
affair that doesn't need a loop. Similarly, there can only be up to 16
group 1 counters. Contexting them is a bit harder, but we can do with a
single branch with a falling through switch. If/when both of these
change, we have a pair of asserts and the feature detection mechanism to
guard us against pretending that we support something we don't.

We can drop contexting of the offset registers. They are fully
accessible by EL2 and as such are its responsibility to preserve on
powerdown.

Another small thing we can do, is pass the core_pos into the hook.
The caller already knows which core we're running on, we don't need to
call this non-trivial function again.

Finally, knowing this, we don't really need the auxiliary AMUs to be
described by the device tree. Linux doesn't care at the moment, and any
information we need for EL3 can be neatly placed in a simple array.

All of this, combined with lifting the actual saving out of assembly,
reduces the instructions to save the context from 180 to 40, including a
lot fewer branches. The code is also much shorter and easier to read.

Also propagate to aarch32 so that the two don't diverge too much.

Change-Id: Ib62e6e9ba5be7fb9fb8965c8eee148d5598a5361
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the s

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the same way. Despite that, it is enabled more like an
architectural feature with a top level enable flag. This utilised the
identical implementation.

This duality has left MPMM in an awkward place, where its enablement
should be generic, like an architectural feature, but since it is not,
it should also be core-specific if it ever changes. One choice to do
this has been through the device tree.

This has worked just fine so far, however, recent implementations expose
a weakness in that this is rather slow - the device tree has to be read,
there's a long call stack of functions with many branches, and system
registers are read. In the hot path of PSCI CPU powerdown, this has a
significant and measurable impact. Besides it being a rather large
amount of code that is difficult to understand.

Since MPMM is a microarchitectural feature, its correct placement is in
the reset function. The essence of the current enablement is to write
CPUPPMCR_EL3.MPMM_EN if CPUPPMCR_EL3.MPMMPINCTL == 0. Replacing the C
enablement with an assembly macro in each CPU's reset function achieves
the same effect with just a single close branch and a grand total of 6
instructions (versus the old 2 branches and 32 instructions).

Having done this, the device tree entry becomes redundant. Should a core
that doesn't support MPMM arise, this can cleanly be handled in the
reset function. As such, the whole ENABLE_MPMM_FCONF and platform hooks
mechanisms become obsolete and are removed.

Change-Id: I1d0475b21a1625bb3519f513ba109284f973ffdf
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes from topic "bk/errata_speed" into integration

* changes:
refactor(cpus): declare runtime errata correctly
perf(cpus): make reset errata do fewer branches
perf(cpus): inline the i

Merge changes from topic "bk/errata_speed" into integration

* changes:
refactor(cpus): declare runtime errata correctly
perf(cpus): make reset errata do fewer branches
perf(cpus): inline the init_cpu_data_ptr function
perf(cpus): inline the reset function
perf(cpus): inline the cpu_get_rev_var call
perf(cpus): inline cpu_rev_var checks
refactor(cpus): register DSU errata with the errata framework's wrappers
refactor(cpus): convert checker functions to standard helpers
refactor(cpus): convert the Cortex-A65 to use the errata framework
fix(cpus): declare reset errata correctly

show more ...

perf(cpus): inline the init_cpu_data_ptr function

Similar to the reset function inline, inline this too to not do a costly
branch with no extra cost.

Change-Id: I54cc399e570e9d0f373ae13c7224d32dbdf

perf(cpus): inline the init_cpu_data_ptr function

Similar to the reset function inline, inline this too to not do a costly
branch with no extra cost.

Change-Id: I54cc399e570e9d0f373ae13c7224d32dbdfae1e5
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(cpus): inline the reset function

Similar to the cpu_rev_var and cpu_ger_rev_var functions, inline the
call_reset_handler handler. This way we skip the costly branch at no
extra cost as this is

perf(cpus): inline the reset function

Similar to the cpu_rev_var and cpu_ger_rev_var functions, inline the
call_reset_handler handler. This way we skip the costly branch at no
extra cost as this is the only place where this is called.

While we're at it, drop the options for CPU_NO_RESET_FUNC. The only cpus
that need that are virtual cpus which can spare the tiny bit of
performance lost. The rest are real cores which can save on the check
for zero.

Now is a good time to put the assert for a missing cpu in the
get_cpu_ops_ptr function so that it's a bit better encapsulated.

Change-Id: Ia7c3dcd13b75e5d7c8bafad4698994ea65f42406
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(rmm): add support for BRBCR_EL2 register for feat_brbe

Currently BRBE is being disabled for Realm world in EL3 by
switching the SBRBE bit in mdcr_el3 register to 0b00.
The patch removes the swit

fix(rmm): add support for BRBCR_EL2 register for feat_brbe

Currently BRBE is being disabled for Realm world in EL3 by
switching the SBRBE bit in mdcr_el3 register to 0b00.
The patch removes the switching of SBRBE bits, and adds
context switch of BRBCR_EL2 register.

Change-Id: I66ca13edefc37e40fa265fd438b0b66f7d09b4bb
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

Merge changes I765a7fa0,Ic33f0b6d,I8d1a88c7,I381f96be,I698fa849, ... into integration

* changes:
fix(cpus): clear CPUPWRCTLR_EL1.CORE_PWRDN_EN_BIT on reset
chore(docs): drop the "wfi" from `pwr_

Merge changes I765a7fa0,Ic33f0b6d,I8d1a88c7,I381f96be,I698fa849, ... into integration

* changes:
fix(cpus): clear CPUPWRCTLR_EL1.CORE_PWRDN_EN_BIT on reset
chore(docs): drop the "wfi" from `pwr_domain_pwr_down_wfi`
chore(psci): drop skip_wfi variable
feat(arm): convert arm platforms to expect a wakeup
fix(cpus): avoid SME related loss of context on powerdown
feat(psci): allow cores to wake up from powerdown
refactor: panic after calling psci_power_down_wfi()
refactor(cpus): undo errata mitigations
feat(cpus): add sysreg_bit_toggle

show more ...

fix: add support for 128-bit sysregs to EL3 crash handler

The following changes have been made:
* Add new sysreg definitions and ASM macro is_feat_sysreg128_present_asm
* Add registers TTBR0_EL2 and

fix: add support for 128-bit sysregs to EL3 crash handler

The following changes have been made:
* Add new sysreg definitions and ASM macro is_feat_sysreg128_present_asm
* Add registers TTBR0_EL2 and VTTBR_EL2 to EL3 crash handler output
* Use MRRS instead of MRS for registers TTBR0_EL1, TTBR0_EL2, TTBR1_EL1,
VTTBR_EL2 and PAR_EL1

Change-Id: I0e20b2c35251f3afba2df794c1f8bc0c46c197ff
Signed-off-by: Igor Podgainõi <igor.podgainoi@arm.com>

show more ...

fix(cpus): avoid SME related loss of context on powerdown

Travis' and Gelas' TRMs tell us to disable SME (set PSTATE.{ZA, SM} to
0) when we're attempting to power down. What they don't tell us is th

fix(cpus): avoid SME related loss of context on powerdown

Travis' and Gelas' TRMs tell us to disable SME (set PSTATE.{ZA, SM} to
0) when we're attempting to power down. What they don't tell us is that
if this isn't done, the powerdown request will be rejected. On the
CPU_OFF path that's not a problem - we can force SVCR to 0 and be
certain the core will power off.

On the suspend to powerdown path, however, we cannot do this. The TRM
also tells us that the sequence could also be aborted on eg. GIC
interrupts. If this were to happen when we have overwritten SVCR to 0,
upon a return to the caller they would experience a loss of context. We
know that at least Linux may call into PSCI with SVCR != 0. One option
is to save the entire SME context which would be quite expensive just to
work around. Another option is to downgrade the request to a normal
suspend when SME was left on. This option is better as this is expected
to happen rarely enough to ignore the wasted power and we don't want to
burden the generic (correct) path with needless context management.

Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: I698fa8490ebf51461f6aa8bba84f9827c5c46ad4

show more ...

feat(mops): enable FEAT_MOPS in EL3 when INIT_UNUSED_NS_EL2=1

FEAT_MOPS, mandatory from Arm v8.8, is typically managed in EL2.
However, in configurations where NS_EL2 is not enabled,
EL3 must set th

feat(mops): enable FEAT_MOPS in EL3 when INIT_UNUSED_NS_EL2=1

FEAT_MOPS, mandatory from Arm v8.8, is typically managed in EL2.
However, in configurations where NS_EL2 is not enabled,
EL3 must set the HCRX_EL2.MSCEn bit to 1 to enable the feature.

This patch ensures FEAT_MOPS is enabled by setting HCRX_EL2.MSCEn to 1.

Change-Id: Ic4960e0cc14a44279156b79ded50de475b3b21c5
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>

show more ...

refactor(cm): clean up per-world context

In preparation for SMCCC_ARCH_FEATURE_AVAILABILITY, it is useful for
context to be directly related to the underlying system. Currently,
certain bits like SC

refactor(cm): clean up per-world context

In preparation for SMCCC_ARCH_FEATURE_AVAILABILITY, it is useful for
context to be directly related to the underlying system. Currently,
certain bits like SCR_EL3.APK are always set with the understanding that
they will only take effect if the feature is present.

However, that is problematic for SMCCC_ARCH_FEATURE_AVAILABILITY (an
SMCCC call to report which features firmware enables), as simply reading
the enable bit may contradict the ID register, like the APK bit above
for a system with no Pauth present.

This patch is to clean up these cases. Add a check for PAuth's presence
so that the APK bit remains unset if not present. Also move SPE and TRBE
enablement to only the NS context. They already only enable the features
for NS only and disable them for Secure and Realm worlds. This change
only makes these worlds' context read 0 for easy bitmasking.

There's only a single snag on SPE and TRBE. Currently, their fields have
the same values and any world asymmetry is handled by hardware. Since we
don't want to do that, the buffers' ownership will change if we just set
the fields to 0 for non-NS worlds. Doing that, however, exposes Secure
state to a potential denial of service attack - a malicious NS can
enable profiling and call an SMC. Then, the owning security state will
change and since no SPE/TRBE registers are contexted, Secure state will
start generating records. Always have NS world own the buffers to
prevent this.

Finally, get rid of manage_extensions_common() as it's just a level of
indirection to enable a single feature.

Change-Id: I487bd4c70ac3e2105583917a0e5499e0ee248ed9
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(cm): change owning security state when a feature is disabled

SPE and TRBE don't have an outright EL3 disable, there are only
constraints on what's allowed. Since we only enable them for NS

refactor(cm): change owning security state when a feature is disabled

SPE and TRBE don't have an outright EL3 disable, there are only
constraints on what's allowed. Since we only enable them for NS at the
moment, we want NS to own the buffers even when the feature should be
"disabled" for a world. This means that when we're running in NS
everything is as normal but when running in S/RL then tracing is
prohibited (since the buffers are owned by NS). This allows us to fiddle
with context a bit more without having to context switch registers.

Change-Id: Ie1dc7c00e4cf9bcc746f02ae43633acca32d3758
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(aarch64): add DBGPRCR_EL1 register accessors

This is a small change adding accessor functions for the Debug Power
Control register (DBGPRCR_EL1) to the common architectural helpers.

Change-Id:

feat(aarch64): add DBGPRCR_EL1 register accessors

This is a small change adding accessor functions for the Debug Power
Control register (DBGPRCR_EL1) to the common architectural helpers.

Change-Id: I72261fbf0395d900347b46af320093ed946aa73d
Signed-off-by: Chris Kay <chris.kay@arm.com>

show more ...

fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not d

fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not done, the in-flight entries might use any piece of now incorrect
context as there are no implicit ordering requirements.

Prior to root context, the buffer drain hooks would have done that. But
now that must happen much earlier. So add a tsb to prepare_el3_entry as
well.

Annoyingly, the barrier can be reordered relative to other instructions
by default (rule RCKVWP). So add an isb after the psb/tsb to assure that
they are ordered, at least as far as context is concerned.

Then, drop the buffer draining hooks. Everything they need to do is
already done by now. There's a notable difference in that there are no
dsb-s now. Since EL3 does not access the buffers or the feature
specific context, we don't need to wait for them to finish.

Finally, drop a stray isb in the context saving macro. It is now
absorbed into root context, but was missed.

Change-Id: I30797a40ac7f91d0bb71ad271a1597e85092ccd5
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...