feat(cpufeat): add support for FEAT_IDTE3

This patch adds support for FEAT_IDTE3, which introduces support
for handling the trapping of Group 3 and Group 5 (only GMID_EL1)
registers to EL3 (unless t

feat(cpufeat): add support for FEAT_IDTE3

This patch adds support for FEAT_IDTE3, which introduces support
for handling the trapping of Group 3 and Group 5 (only GMID_EL1)
registers to EL3 (unless trapped to EL2). IDTE3 allows EL3 to
modify the view of ID registers for lower ELs, and this capability
is used to disable fields of ID registers tied to disabled features.

The ID registers are initially read as-is and stored in context.
Then, based on the feature enablement status for each world, if a
particular feature is disabled, its corresponding field in the
cached ID register is set to Res0. When lower ELs attempt to read
an ID register, the cached ID register value is returned. This
allows EL3 to prevent lower ELs from accessing feature-specific
system registers that are disabled in EL3, even though the hardware
implements them.

The emulated ID register values are stored primarily in per-world
context, except for certain debug-related ID registers such as
ID_AA64DFR0_EL1 and ID_AA64DFR1_EL1, which are stored in the
cpu_data and are unique to each PE. This is done to support feature
asymmetry that is commonly seen in debug features.

FEAT_IDTE3 traps all Group 3 ID registers in the range
op0 == 3, op1 == 0, CRn == 0, CRm == {2–7}, op2 == {0–7} and the
Group 5 GMID_EL1 register. However, only a handful of ID registers
contain fields used to detect features enabled in EL3. Hence, we
only cache those ID registers, while the rest are transparently
returned as is to the lower EL.

This patch updates the CREATE_FEATURE_FUNCS macro to generate
update_feat_xyz_idreg_field() functions that disable ID register
fields on a per-feature basis. The enabled_worlds scope is used to
disable ID register fields for security states where the feature is
not enabled.

This EXPERIMENTAL feature is controlled by the ENABLE_FEAT_IDTE3
build flag and is currently disabled by default.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I5f998eeab81bb48c7595addc5595313a9ebb96d5

show more ...

fix(bl1): use per-world context correctly

Currently, the configuration with BL1 and BL2 at SEL1 will transition
via el3_exit which will restore per-world context. However, that context
is never writ

fix(bl1): use per-world context correctly

Currently, the configuration with BL1 and BL2 at SEL1 will transition
via el3_exit which will restore per-world context. However, that context
is never written to and so zeroes end up in registers, which is not
necessarily correct.

This patch gets BL1 to call cm_manage_extensions_per_world() whenever
BL2 runs in a lower EL. This allows the per-world registers to have the
reset values we intend. An accompanying call to
cm_manage_extensions_el3() is also added for completeness.

Doing this shows a small deficiency in cptr_el3 - bits TFP and TCPAC
change a lot. This patch makes them consistent by always setting TCPAC
and TFP to 0 which unconditionally enable access to CPTR_EL2 and FPCR by
default as they are always accessible. Other places that manipulate the
TFP bit are removed.

A nice side effect of all of this is that we're now in a position to
enable and use any architectural extension in BL2.

Change-Id: I070d62bbf8e9d9b472caf7e2c931c303523be308
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(cpufeat): enable FEAT_CPA2 for EL3

FEAT_CPA2 enables checked pointer arithmetic, which in the event
of an arithmetic overflow in pointer generation will result in a
non-canonical pointer being

feat(cpufeat): enable FEAT_CPA2 for EL3

FEAT_CPA2 enables checked pointer arithmetic, which in the event
of an arithmetic overflow in pointer generation will result in a
non-canonical pointer being generated and subsequent address fault.

Note that FEAT_CPA is a trivial implementation that exists in
some hardware purely so it can run CPA2-enabled instructions
without crashing but they don't actually have checked arithmetic,
so FEAT_CPA is not explicitly enabled in TF-A.

Change-Id: I6d2ca7a7e4b986bb9e917aa8baf8091a271c168b
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

refactor: unify blx_setup() and blx_main()

All BLs have a bl_setup() for things that need to happen early, a fall
back into assembly and then bl_main() for the main functionality. This
was necessary

refactor: unify blx_setup() and blx_main()

All BLs have a bl_setup() for things that need to happen early, a fall
back into assembly and then bl_main() for the main functionality. This
was necessary in order to fiddle with PAuth related things that tend to
break C calls. Since then PAuth's enablement has seen a lot of
refactoring and this is now worked around cleanly so the distinction can
be removed. The only tradeoff is that this requires pauth to not be used
for the top-level main function.

There are two main benefits to doing this: First, code is easier to
understand as it's all together and the entrypoint is smaller. Second,
the compiler gets to see more of the code and apply optimisations
(importantly LTO).

Change-Id: Iddb93551115a2048988017547eb7b8db441dbd37
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(cpufeat): enable FEAT_MPAM_PE_BW_CTRL support

Implement support for FEAT_MPAM_PE_BW_CTRL, allowing lower
Exception Levels to access MPAM_PE_BW_CTRL control registers
by disabling their traps to

feat(cpufeat): enable FEAT_MPAM_PE_BW_CTRL support

Implement support for FEAT_MPAM_PE_BW_CTRL, allowing lower
Exception Levels to access MPAM_PE_BW_CTRL control registers
by disabling their traps to EL3.

When INIT_UNUSED_NS_EL2=1, configure MPAMBW2_EL2 in EL3 so
that MPAM_PE_BW_CTRL accesses from EL0/EL1 do not trap to EL2.

At this stage, PE-side MPAM bandwidth controls remain disabled
in EL3.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I8e359b0eb912cff3bdda109b21727a627cac3a7e

show more ...

fix(cm): disable SPE/TRBE correctly

SPE and TRBE are unusual features. They have multi-bit enables whose
function is not immediately apparent and disabling them is not
straightforward.

While attemp

fix(cm): disable SPE/TRBE correctly

SPE and TRBE are unusual features. They have multi-bit enables whose
function is not immediately apparent and disabling them is not
straightforward.

While attempting to figure this out, the disables were made a mess of.
Patch fc7dca72b began changing the owning security state of SPE and
TRBE. This was first used in patch 79c0c7fac0 with calls to
spe_disable() and trbe_disbale(). However, patch 13f4a2525 reverted the
security state ownership, making the spe_disable() and trbe_disable()
redundant and their comments incorrect - the DoS protection is achieved
by the psb/tsb barriers on context switch, introduces separately in
f80887337 and 73d98e375.

Those patches got the behaviour full circle to what it was in fc7dca72b
so the disables can be fully removed for clarity.

However, the original method for disabling these features is not fully
correct - letting the "disabled" state be all zeroes made the features
seem enabled for secure world but they would trap. That is not a problem
while secure world doesn't use them, but could lead to some confusing
debugging in the future. NS and Realm worlds were not affected. This
patch fully establishes the pattern for SPE and TRBE's enablement,
documents it, and implements it such.

The description comments in the features boil down to 2 rules. There is
a third rule possible:
3. To enable TRBE/SPE for world X with a dirty buffer:
* world X owns the buffer
* trapping enabled
This is not listed as it would not work correctly with
SMCCC_ARCH_FEATURE_AVAILABILITY which relies on trapping to be disabled
to report correctly. If that is ever implemented, the SMCCC
implementation should be considered too.

Change-Id: I5588a3d5fc074c2445470954d8c3b172bec77d43
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(cpufeat): convert FEAT_PAuth setup to C

An oversimplified view of FEAT_PAuth is that it's a symmetric encryption
of the LR. PAC instructions execute as NOPs until explicitly turned on.
So i

refactor(cpufeat): convert FEAT_PAuth setup to C

An oversimplified view of FEAT_PAuth is that it's a symmetric encryption
of the LR. PAC instructions execute as NOPs until explicitly turned on.
So in a function that turns PAuth on, the signing would have executed as
a NOP and the authentication will encrypt the address, leading to a
failure. That's why enablement is in assembly - we have full control of
when pointer authentications happen.

However, assembly is hard to read, is opaque to the compiler for
optimisations, and we need to call into C anyway for the platform hook
to get the key. So convert it to C. We can instruct the compiler to not
generate branch protection for the enable function only and as long as
the caller doesn't do branch protection (and all callers are entrypoints
written in assembly) everything will work.

Change-Id: I8917a26e1293033c910e3058664e3ca9207359b7
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(ras): add eabort get helper function

Add EABORT get field helper function to obtain SET, AET (UET) values
from esr_el3/disr_el1 based on PE error state recording in the exception
syndrome refer

feat(ras): add eabort get helper function

Add EABORT get field helper function to obtain SET, AET (UET) values
from esr_el3/disr_el1 based on PE error state recording in the exception
syndrome refer to RAS PE architecture in
https://developer.arm.com/documentation/ddi0487/latest/

Change-Id: I0011f041a3089c9bbf670275687ad7c3362a07f9
Signed-off-by: Vinoj Soundararajan <vinojs@google.com>

show more ...

feat(ras): add asynchronous error type corrected

Add asynchronous error type Corrected (CE) to error status
AET based on PE error state recording in the exception syndrome
Refer to https://developer

feat(ras): add asynchronous error type corrected

Add asynchronous error type Corrected (CE) to error status
AET based on PE error state recording in the exception syndrome
Refer to https://developer.arm.com/documentation/ddi0487/latest/
RAS PE architecture.

Change-Id: I9f2525411b94c8fd397b4a0b8cf5dc47457a2771
Signed-off-by: Vinoj Soundararajan <vinojs@google.com>

show more ...

fix(ras): fix typo in uncorrectable error type UEO

Fix spelling for UEO from restable to restartable
based on PE error state recording in the exception syndrome
Refer to https://developer.arm.com/do

fix(ras): fix typo in uncorrectable error type UEO

Fix spelling for UEO from restable to restartable
based on PE error state recording in the exception syndrome
Refer to https://developer.arm.com/documentation/ddi0487/latest/
RAS PE architecture.

Change-Id: I4da419f2120a7385853d4da78b409c675cdfe1c8
Signed-off-by: Vinoj Soundararajan <vinojs@google.com>

show more ...

fix(ras): fix status synchronous error type fields

Based on SET bits of ISS encoding for an exception from Data or
Instruction Abort. (Refer to ESR_EL3)
1. Fix Synchronous error type restartable val

fix(ras): fix status synchronous error type fields

Based on SET bits of ISS encoding for an exception from Data or
Instruction Abort. (Refer to ESR_EL3)
1. Fix Synchronous error type restartable value from 1 to 3
2. Remove corrected CE field which is not applicable to SET

Change-Id: If357da9881bee962825bc3b9423ba7fc107f9b1d
Signed-off-by: Vinoj Soundararajan <vinojs@google.com>

show more ...

perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context

SVE and SME aren't enabled symmetrically for all worlds, but EL3 needs
to context switch them nonetheless. Previously,

perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context

SVE and SME aren't enabled symmetrically for all worlds, but EL3 needs
to context switch them nonetheless. Previously, this had to happen by
writing the enable bits just before reading/writing the relevant
context. But since the introduction of root context, this need not be
the case. We can have these enables always be present for EL3 and save
on some work (and ISBs!) on every context switch.

We can also hoist ZCR_EL3 to a never changing register, as we set its
value to be identical for every world, which happens to be the one we
want for EL3 too.

Change-Id: I3d950e72049a298008205ba32f230d5a5c02f8b0
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness in that this is rather slow. A large part of it is written in
assembly, making it opaque to the compiler for optimisations. The
future proofness requires reading registers that are effectively
`volatile`, making it even harder for the compiler, as well as adding
lots of implicit barriers, making it hard for the microarchitecutre to
optimise as well.

We can make a few assumptions, checked by a few well placed asserts, and
remove a lot of this burden. For a start, at the moment there are 4
group 0 counters with static assignments. Contexting them is a trivial
affair that doesn't need a loop. Similarly, there can only be up to 16
group 1 counters. Contexting them is a bit harder, but we can do with a
single branch with a falling through switch. If/when both of these
change, we have a pair of asserts and the feature detection mechanism to
guard us against pretending that we support something we don't.

We can drop contexting of the offset registers. They are fully
accessible by EL2 and as such are its responsibility to preserve on
powerdown.

Another small thing we can do, is pass the core_pos into the hook.
The caller already knows which core we're running on, we don't need to
call this non-trivial function again.

Finally, knowing this, we don't really need the auxiliary AMUs to be
described by the device tree. Linux doesn't care at the moment, and any
information we need for EL3 can be neatly placed in a simple array.

All of this, combined with lifting the actual saving out of assembly,
reduces the instructions to save the context from 180 to 40, including a
lot fewer branches. The code is also much shorter and easier to read.

Also propagate to aarch32 so that the two don't diverge too much.

Change-Id: Ib62e6e9ba5be7fb9fb8965c8eee148d5598a5361
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disab

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disabling sampling
(writing PMBLIMITR_EL1.E to 0) is redundant and can be removed. The only
reason we save/restore SPE context is because of that disable, so those
can be removed too.

There's the issue of draining the profiling buffer though. No new
samples will have been generated since entering EL3. However, old
samples might still be in-flight. Unless synchronised by a psb csync,
those might be affected by our extensive context mutation. Adding a psb
in prepare_el3_entry should cater for that. Note that prior to the
introduction of root context this was not a problem as context remained
unchanged and the hooks took care of the rest.

Then, the only time we care about the buffer actually making it to
memory is when we exit coherency. On HW_ASSISTED_COHERENCY systems we
don't have to do anything, it should be handled for us. Systems without
it need a dsb to wait for them to complete. There should be one already
in each cpu's powerdown hook which should work.

While on the topic of barriers, the esb barrier is no longer used.
Remove it.

Change-Id: I9736fc7d109702c63e7d403dc9e2a4272828afb2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for N

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for NS world only.

Reference:
https://developer.arm.com/documentation/109697/2024_09/
Feature-descriptions/The-Armv9-5-architecture-extension?lang=en

Change-Id: I580c409b9b22f8ead0737502280fb9093a3d5dd2
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>

show more ...

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEA

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEAT_SYSREG128 is implemented.
With FEAT_SYSREG128 certain system registers are treated as 128-bit,
so we should be context saving and restoring 128-bits instead of 64-bit
when FEAT_D128 is enabled.

FEAT_SYSREG128 adds support for MRRS and MSRR instruction which
helps us to read write to 128-bit system register.
Refer to Arm Architecture Manual for further details.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I1a53db5eac29e56c8fbdcd4961ede3abfcb2411a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across core

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across cores and the respective support is added here.

Adding a function to handle this asymmetry by re-visting the
feature presence on running core.
There are two possible cases:
- If the primary core has the feature and secondary does not have it
then the feature is disabled.
- If the primary does not have the feature and secondary has it then,
the feature need to be enabled in secondary cores.

Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Change-Id: I73a70891d52268ddfa4effe40edf04115f5821ca

show more ...

feat(simd): add routines to save, restore sve state

This adds assembly routines to save and restore SVE registers. In order
to share between FPU and SVE the code to save and restore FPCR and
FPSR, t

feat(simd): add routines to save, restore sve state

This adds assembly routines to save and restore SVE registers. In order
to share between FPU and SVE the code to save and restore FPCR and
FPSR, the patch converts code for those registers into macro.
Since we will be using simd_ctx_t to save and restore FPU also, we use
offsets in simd_ctx_t for FPSR and FPCR. Since simd_ctx_t has the same
structure at the beginning as fp_regs_t, those offsets should be the
same as CTX_FP_* offsets, when SVE is not enabled. Note that the code
also saves and restores FPEXC32 reg along with FPSR and FPCR.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: I120c02359794aa6bb6376a464a9afe98bd84ae60

show more ...