perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness in that this is rather slow. A large part of it is written in
assembly, making it opaque to the compiler for optimisations. The
future proofness requires reading registers that are effectively
`volatile`, making it even harder for the compiler, as well as adding
lots of implicit barriers, making it hard for the microarchitecutre to
optimise as well.

We can make a few assumptions, checked by a few well placed asserts, and
remove a lot of this burden. For a start, at the moment there are 4
group 0 counters with static assignments. Contexting them is a trivial
affair that doesn't need a loop. Similarly, there can only be up to 16
group 1 counters. Contexting them is a bit harder, but we can do with a
single branch with a falling through switch. If/when both of these
change, we have a pair of asserts and the feature detection mechanism to
guard us against pretending that we support something we don't.

We can drop contexting of the offset registers. They are fully
accessible by EL2 and as such are its responsibility to preserve on
powerdown.

Another small thing we can do, is pass the core_pos into the hook.
The caller already knows which core we're running on, we don't need to
call this non-trivial function again.

Finally, knowing this, we don't really need the auxiliary AMUs to be
described by the device tree. Linux doesn't care at the moment, and any
information we need for EL3 can be neatly placed in a simple array.

All of this, combined with lifting the actual saving out of assembly,
reduces the instructions to save the context from 180 to 40, including a
lot fewer branches. The code is also much shorter and easier to read.

Also propagate to aarch32 so that the two don't diverge too much.

Change-Id: Ib62e6e9ba5be7fb9fb8965c8eee148d5598a5361
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the s

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the same way. Despite that, it is enabled more like an
architectural feature with a top level enable flag. This utilised the
identical implementation.

This duality has left MPMM in an awkward place, where its enablement
should be generic, like an architectural feature, but since it is not,
it should also be core-specific if it ever changes. One choice to do
this has been through the device tree.

This has worked just fine so far, however, recent implementations expose
a weakness in that this is rather slow - the device tree has to be read,
there's a long call stack of functions with many branches, and system
registers are read. In the hot path of PSCI CPU powerdown, this has a
significant and measurable impact. Besides it being a rather large
amount of code that is difficult to understand.

Since MPMM is a microarchitectural feature, its correct placement is in
the reset function. The essence of the current enablement is to write
CPUPPMCR_EL3.MPMM_EN if CPUPPMCR_EL3.MPMMPINCTL == 0. Replacing the C
enablement with an assembly macro in each CPU's reset function achieves
the same effect with just a single close branch and a grand total of 6
instructions (versus the old 2 branches and 32 instructions).

Having done this, the device tree entry becomes redundant. Should a core
that doesn't support MPMM arise, this can cleanly be handled in the
reset function. As such, the whole ENABLE_MPMM_FCONF and platform hooks
mechanisms become obsolete and are removed.

Change-Id: I1d0475b21a1625bb3519f513ba109284f973ffdf
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix: add support for 128-bit sysregs to EL3 crash handler

The following changes have been made:
* Add new sysreg definitions and ASM macro is_feat_sysreg128_present_asm
* Add registers TTBR0_EL2 and

fix: add support for 128-bit sysregs to EL3 crash handler

The following changes have been made:
* Add new sysreg definitions and ASM macro is_feat_sysreg128_present_asm
* Add registers TTBR0_EL2 and VTTBR_EL2 to EL3 crash handler output
* Use MRRS instead of MRS for registers TTBR0_EL1, TTBR0_EL2, TTBR1_EL1,
VTTBR_EL2 and PAR_EL1

Change-Id: I0e20b2c35251f3afba2df794c1f8bc0c46c197ff
Signed-off-by: Igor Podgainõi <igor.podgainoi@arm.com>

show more ...

feat(pmuv3): setup per world MDCR_EL3

MDCR_EL3 register will context switch across all worlds. Thus the pmuv3
init has to be part of context management initialization.

Change-Id: I10ef7a3071c0fc5c1

feat(pmuv3): setup per world MDCR_EL3

MDCR_EL3 register will context switch across all worlds. Thus the pmuv3
init has to be part of context management initialization.

Change-Id: I10ef7a3071c0fc5c11a93d3c9c2a95ec8c6493bf
Signed-off-by: Mateusz Sulimowicz <matsul@google.com>

show more ...

fix(cm): change back owning security state when a feature is disabled

Patch fc7dca72ba656e5f147487b20f9f0fb6eb39e116 changed the owning
security states of the TRBE and SPE buffers to NS. The thinkin

fix(cm): change back owning security state when a feature is disabled

Patch fc7dca72ba656e5f147487b20f9f0fb6eb39e116 changed the owning
security states of the TRBE and SPE buffers to NS. The thinking was that
this would assist SMCCC feature availability to more easily determine
if the feature is enabled or disabled. However, that only changed bit 0
while the SMCCC feature only looks at bit 1 so this change is redundant.

It was also meant to tighten security but that was done by
73d98e37593f4a4044dd28f52127cdc890911c0c instead.

Annoyingly, FEAT_TRBE has TRBIDR_EL1 which reports that programming is
allowed when the current security state owns the buffer even when the
MDCR_EL3 setting disallows this in practice.

So revert the functional aspect of the patch as it causes linux panics
with ERRATA_A520_2938996. Keep the defines as they are used elsewhere.

Change-Id: I39463d585df89aee44d1996137616da85d678f41
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(cm): change owning security state when a feature is disabled

SPE and TRBE don't have an outright EL3 disable, there are only
constraints on what's allowed. Since we only enable them for NS

refactor(cm): change owning security state when a feature is disabled

SPE and TRBE don't have an outright EL3 disable, there are only
constraints on what's allowed. Since we only enable them for NS at the
moment, we want NS to own the buffers even when the feature should be
"disabled" for a world. This means that when we're running in NS
everything is as normal but when running in S/RL then tracing is
prohibited (since the buffers are owned by NS). This allows us to fiddle
with context a bit more without having to context switch registers.

Change-Id: Ie1dc7c00e4cf9bcc746f02ae43633acca32d3758
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not d

fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not done, the in-flight entries might use any piece of now incorrect
context as there are no implicit ordering requirements.

Prior to root context, the buffer drain hooks would have done that. But
now that must happen much earlier. So add a tsb to prepare_el3_entry as
well.

Annoyingly, the barrier can be reordered relative to other instructions
by default (rule RCKVWP). So add an isb after the psb/tsb to assure that
they are ordered, at least as far as context is concerned.

Then, drop the buffer draining hooks. Everything they need to do is
already done by now. There's a notable difference in that there are no
dsb-s now. Since EL3 does not access the buffers or the feature
specific context, we don't need to wait for them to finish.

Finally, drop a stray isb in the context saving macro. It is now
absorbed into root context, but was missed.

Change-Id: I30797a40ac7f91d0bb71ad271a1597e85092ccd5
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disab

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disabling sampling
(writing PMBLIMITR_EL1.E to 0) is redundant and can be removed. The only
reason we save/restore SPE context is because of that disable, so those
can be removed too.

There's the issue of draining the profiling buffer though. No new
samples will have been generated since entering EL3. However, old
samples might still be in-flight. Unless synchronised by a psb csync,
those might be affected by our extensive context mutation. Adding a psb
in prepare_el3_entry should cater for that. Note that prior to the
introduction of root context this was not a problem as context remained
unchanged and the hooks took care of the rest.

Then, the only time we care about the buffer actually making it to
memory is when we exit coherency. On HW_ASSISTED_COHERENCY systems we
don't have to do anything, it should be handled for us. Systems without
it need a dsb to wait for them to complete. There should be one already
in each cpu's powerdown hook which should work.

While on the topic of barriers, the esb barrier is no longer used.
Remove it.

Change-Id: I9736fc7d109702c63e7d403dc9e2a4272828afb2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for N

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for NS world only.

Reference:
https://developer.arm.com/documentation/109697/2024_09/
Feature-descriptions/The-Armv9-5-architecture-extension?lang=en

Change-Id: I580c409b9b22f8ead0737502280fb9093a3d5dd2
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>

show more ...

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEA

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEAT_SYSREG128 is implemented.
With FEAT_SYSREG128 certain system registers are treated as 128-bit,
so we should be context saving and restoring 128-bits instead of 64-bit
when FEAT_D128 is enabled.

FEAT_SYSREG128 adds support for MRRS and MSRR instruction which
helps us to read write to 128-bit system register.
Refer to Arm Architecture Manual for further details.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I1a53db5eac29e56c8fbdcd4961ede3abfcb2411a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

fix(brbe): allow RME builds with BRBE

It used to be the case that a FEAT_RME build could not be built with
FEAT_BRBE support. BRBE doesn't have a 3-world aware disable and
MDCR_EL3 was not context s

fix(brbe): allow RME builds with BRBE

It used to be the case that a FEAT_RME build could not be built with
FEAT_BRBE support. BRBE doesn't have a 3-world aware disable and
MDCR_EL3 was not context switched to allow for disabling in Realm world.

As of commit 123002f9171384d976d95935b7f566740d69cc68 MDCR_EL3 is
context switched. Since the flag for BRBE support is
ENABLE_BRBE_FOR_NS, move brbe_enable() to only happen for NS world. The
other worlds will see BRBE disabled and branch recording prohibited.
This allows for a build with both RME and BRBE.

Note that EL2 BRBE registers are not context switched. Further work is
needed if non-NS support is required.

Change-Id: I82f0f08399dcd080902477dc9636bc4541685f89
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across core

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across cores and the respective support is added here.

Adding a function to handle this asymmetry by re-visting the
feature presence on running core.
There are two possible cases:
- If the primary core has the feature and secondary does not have it
then the feature is disabled.
- If the primary does not have the feature and secondary has it then,
the feature need to be enabled in secondary cores.

Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Change-Id: I73a70891d52268ddfa4effe40edf04115f5821ca

show more ...

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure,

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure, we hold a copy of EL1, EL2
system registers, per world per PE. This context structure is
enormous and will continue to grow bigger with the addition of
new features incorporating new registers.

* Ideally, EL3 should save and restore the system registers at its next
lower exception level, which is EL2 in majority of the configurations.

* This patch aims at optimising the memory allocation in cases, when
the members from the context structure are unused. So el1 system
register context must be omitted when lower EL is always x-EL2.

* "CTX_INCLUDE_EL2_REGS" is the internal build flag which gets set,
when SPD=spmd and SPMD_SPM_AT_SEL2=1 or ENABLE_RME=1.
It indicates, the system registers at EL2 are context switched for
the respective build configuration. Here, there is no need to save
and restore EL1 system registers, while x-EL2 is enabled.

Henceforth, this patch addresses this issue, by taking out the EL1
context at all possible places, while EL2 (CTX_INCLUDE_EL2_REGS) is
enabled, there by saving memory.

Change-Id: Ifddc497d3c810e22a15b1c227a731bcc133c2f4a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

feat(trbe): introduce trbe_disable() function

This patch adds trbe_disable() which disables Trace buffer access
from lower ELs in all security state. This function makes Secure
state the owner of Tr

feat(trbe): introduce trbe_disable() function

This patch adds trbe_disable() which disables Trace buffer access
from lower ELs in all security state. This function makes Secure
state the owner of Trace buffer and access from EL2/EL1 generate
trap exceptions to EL3.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: If3e3bd621684b3c28f44c3ed2fe3df30b143f8cd

show more ...

feat(spe): introduce spe_disable() function

Introduce a function to disable SPE feature for Non-secure state and do
the default setting of making Secure state the owner of profiling
buffers and trap

feat(spe): introduce spe_disable() function

Introduce a function to disable SPE feature for Non-secure state and do
the default setting of making Secure state the owner of profiling
buffers and trap access of profiling and profiling buffer control
registers from lower ELs to EL3.

This functionality is required to handle asymmetric cores where SPE has
to disabled at runtime.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I2f99e922e8df06bfc900c153137aef7c9dcfd759

show more ...

chore(spe): rename spe_disable() to spe_stop()

During CPU power down, we stop the profiling by calling spe_disable()
function. From TF-A point of view, enable/disable means the avaibility
of the fea

chore(spe): rename spe_disable() to spe_stop()

During CPU power down, we stop the profiling by calling spe_disable()
function. From TF-A point of view, enable/disable means the avaibility
of the feature for lower EL. In this case we are not actully disabling
the feautre but stoping it before power down.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I6e3b39c5c35d330c51e7ac715446a8b36bf9531f

show more ...

feat(fgt2): add support for FEAT_FGT2

This patch disables trapping to EL3 when the FEAT_FGT2
specific trap registers are accessed by setting the
SCR_EL3.FGTEn2 bit

Signed-off-by: Arvind Ram Prakash

feat(fgt2): add support for FEAT_FGT2

This patch disables trapping to EL3 when the FEAT_FGT2
specific trap registers are accessed by setting the
SCR_EL3.FGTEn2 bit

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I6d2b614affb9067b2bc3d7bf0ae7d169d031592a

show more ...

feat(debugv8p9): add support for FEAT_Debugv8p9

This patch enables FEAT_Debugv8p9 and prevents EL1/0 from
trapping to EL3 when accessing MDSELR_EL1 register by
setting the MDCR_EL3.EBWE bit.

Signed

feat(debugv8p9): add support for FEAT_Debugv8p9

This patch enables FEAT_Debugv8p9 and prevents EL1/0 from
trapping to EL3 when accessing MDSELR_EL1 register by
setting the MDCR_EL3.EBWE bit.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I3613af1dd8cb8c0d3c33dc959f170846c0b9695a

show more ...

feat(cm): context switch MDCR_EL3 register

Currently MDCR_EL3 register value is same for all the
worlds(Non-secure, Secure, Realm and Root).

With this approach, features enable/disable settings
rem

feat(cm): context switch MDCR_EL3 register

Currently MDCR_EL3 register value is same for all the
worlds(Non-secure, Secure, Realm and Root).

With this approach, features enable/disable settings
remain same across all the worlds. This is not ideal as
there must be flexibility in controlling feature as per
the requirements for individual world.

The patch addresses this by providing MDCR_EL3 a per world
value. Features with identical values for all the worlds are
grouped under ``manage_extensions_common`` API.

Change-Id: Ibc068d985fe165d8cb6d0ffb84119bffd743b3d1
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

refactor(cpufeat): add macro to simplify is_feat_xx_present

In this patch, we are trying to introduce the wrapper macro
CREATE_FEATURE_PRESENT to get the following capability and
align it for all th

refactor(cpufeat): add macro to simplify is_feat_xx_present

In this patch, we are trying to introduce the wrapper macro
CREATE_FEATURE_PRESENT to get the following capability and
align it for all the features:

-> is_feat_xx_present(): Does Hardware implement the feature.
-> uniformity in naming the function across multiple features.
-> improved readability

The is_feat_xx_present() is implemented to check if the hardware
implements the feature and does not take into account the
ENABLE_FEAT_XXX flag enabled/disabled in software.

- CREATE_FEATURE_PRESENT(name, idreg, shift, mask, idval)
The wrapper macro reduces the function to a single line and
creates the is_feat_xx_present function that checks the
id register based on the shift and mask values and compares
this against a determined idvalue.

Change-Id: I7b91d2c9c6fbe55f94c693aa1b2c50be54fb9ecc
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

fix(pmu): fix breakage on ARMv7 CPUs with SP_min as BL32

While comments introduced with the original commit claim that
pmuv3_disable_el3()/pmuv3_init_el3() are compatible with PMUv2 and
PMUv1, this

fix(pmu): fix breakage on ARMv7 CPUs with SP_min as BL32

While comments introduced with the original commit claim that
pmuv3_disable_el3()/pmuv3_init_el3() are compatible with PMUv2 and
PMUv1, this is not true in practice: The function accesses the Secure
Debug Control Register (SDCR), which only available to ARMv8 CPUs.

ARMv8 CPUs executing in AArch32 mode would thus be able to disable
their PMUv3, while ARMv7 CPUs would hang trying to access the SDCR.

Fix this by only doing PMUv3 handling when we know a PMUv3 to be
available. This resolves boot hanging on all STM32MP15 platforms
that use SP_min as BL32 instead of OP-TEE.

Change-Id: I40f7611cf46b89a30243cc55bf55a8d9c9de93c8
Fixes: c73686a11cea ("feat(pmu): introduce pmuv3 lib/extensions folder")
Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

show more ...

refactor(cm): couple el2 registers with dependent feature flags

Currently the EL2 part of the context structure (el2_sysregs_t), is
mostly feature dependent.

For instance, CTX_HCRX_EL2 is only need

refactor(cm): couple el2 registers with dependent feature flags

Currently the EL2 part of the context structure (el2_sysregs_t), is
mostly feature dependent.

For instance, CTX_HCRX_EL2 is only needed when FEAT_HCX
(ENABLE_FEAT_HCX=1) is set, but the entry is unconditionally added
in the EL2 context structure and thereby consuming memory even in
build configurations where FEAT_HCX is disabled.

Henceforth, all such context entries should be coupled/tied with
their respective feature enables and be optimized away when unused.
This would reduce the context memory allocation for platforms, that
dont enable/support all the architectural features at once.

Further, converting the assembly context-offset entries into
a c structure relies on garbage collection of the linker
removing unreferenced structures from memory, as well as aiding
in readability and future maintenance.

Change-Id: I0cf49498ee3033cb6f3ee3810331121b26627783
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

fix(spe): invoke spe_disable during power domain off/suspend

spe_disable function, disables profiling and flushes all the buffers and
hence needs to be called on power-off/suspend path.
It needs to

fix(spe): invoke spe_disable during power domain off/suspend

spe_disable function, disables profiling and flushes all the buffers and
hence needs to be called on power-off/suspend path.
It needs to be invoked as SPE feature writes to memory as part of
regular operation and not disabling before exiting coherency
could potentially cause issues.

Currently, this is handled only for the FVP. Other platforms need
to replicate this behaviour and is covered as part of this patch.

Calling it from generic psci library code, before the platform specific
actions to turn off the CPUs, will make it applicable for all the
platforms which have ported the PSCI library.

Change-Id: I90b24c59480357e2ebfa3dfc356c719ca935c13d
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

refactor(cm): move MPAM3_EL3 reg to per world context

Refactor MPAM3_EL3 to be world-specific, eliminating redundant cross-CPU
value duplication and reducing memory footprint.

Signed-off-by: Arvind

refactor(cm): move MPAM3_EL3 reg to per world context

Refactor MPAM3_EL3 to be world-specific, eliminating redundant cross-CPU
value duplication and reducing memory footprint.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: Iddf020a5462737e01ac35e4f2b2b204a8759fafb

show more ...