fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not d

fix(trbe): add a tsb before context switching

Just like for SPE, we need to synchronize TRBE samples before we change
the context to ensure everything goes where it was intended to. If that
is not done, the in-flight entries might use any piece of now incorrect
context as there are no implicit ordering requirements.

Prior to root context, the buffer drain hooks would have done that. But
now that must happen much earlier. So add a tsb to prepare_el3_entry as
well.

Annoyingly, the barrier can be reordered relative to other instructions
by default (rule RCKVWP). So add an isb after the psb/tsb to assure that
they are ordered, at least as far as context is concerned.

Then, drop the buffer draining hooks. Everything they need to do is
already done by now. There's a notable difference in that there are no
dsb-s now. Since EL3 does not access the buffers or the feature
specific context, we don't need to wait for them to finish.

Finally, drop a stray isb in the context saving macro. It is now
absorbed into root context, but was missed.

Change-Id: I30797a40ac7f91d0bb71ad271a1597e85092ccd5
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disab

fix(spe): add a psb before updating context and remove context saving

In the chapter about FEAT_SPE (D16.4 specifically) it is stated that
"Sampling is always disabled at EL3". That means that disabling sampling
(writing PMBLIMITR_EL1.E to 0) is redundant and can be removed. The only
reason we save/restore SPE context is because of that disable, so those
can be removed too.

There's the issue of draining the profiling buffer though. No new
samples will have been generated since entering EL3. However, old
samples might still be in-flight. Unless synchronised by a psb csync,
those might be affected by our extensive context mutation. Adding a psb
in prepare_el3_entry should cater for that. Note that prior to the
introduction of root context this was not a problem as context remained
unchanged and the hooks took care of the rest.

Then, the only time we care about the buffer actually making it to
memory is when we exit coherency. On HW_ASSISTED_COHERENCY systems we
don't have to do anything, it should be handled for us. Systems without
it need a dsb to wait for them to complete. There should be one already
in each cpu's powerdown hook which should work.

While on the topic of barriers, the esb barrier is no longer used.
Remove it.

Change-Id: I9736fc7d109702c63e7d403dc9e2a4272828afb2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for N

feat(fpmr): disable FPMR trap

This patch enables support of FEAT_FPMR by enabling access
to FPMR register. It achieves it by setting the EnFPM bit of
SCR_EL3. This feature is currently enabled for NS world only.

Reference:
https://developer.arm.com/documentation/109697/2024_09/
Feature-descriptions/The-Armv9-5-architecture-extension?lang=en

Change-Id: I580c409b9b22f8ead0737502280fb9093a3d5dd2
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>

show more ...

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest 32 bits of the data with a value taken from the ACCDATA_EL1
system register (so that EL0 cannot alter them).
Using that ST64BV0 instruction and accessing the ACCDATA_EL1 system
register is guarded by two SCR_EL3 bits, which we should set to avoid a
trap into EL3, when lower ELs use one of those.

Add the required bits and pieces to make this feature usable:
- Add the ENABLE_FEAT_LS64_ACCDATA build option (defaulting to 0).
- Add the CPUID and SCR_EL3 bit definitions associated with FEAT_LS64.
- Add a feature check to check for the existing four variants of the
LS64 feature and detect future extensions.
- Add code to save and restore the ACCDATA_EL1 register on
secure/non-secure context switches.
- Enable the feature with runtime detection for FVP and Arm FPGA.

Please note that the *basic* FEAT_LS64 feature does not feature any trap
bits, it's only the addition of the ACCDATA_EL1 system register that
adds these traps and the SCR_EL3 bits.

Change-Id: Ie3e2ca2d9c4fbbd45c0cc6089accbb825579138a
Signed-off-by: Andre Przywara <andre.przywara@arm.com>

show more ...

Merge "feat(feat_sctlr2): enable FEAT_SCTLR2 for Realm world" into integration

feat(feat_sctlr2): enable FEAT_SCTLR2 for Realm world

Change-Id: I62e769ae796bbeb41741c2c421a5f129d875f5fb
Signed-off-by: Javier Almansa Sobrino <javier.almansasobrino@arm.com>

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEA

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEAT_SYSREG128 is implemented.
With FEAT_SYSREG128 certain system registers are treated as 128-bit,
so we should be context saving and restoring 128-bits instead of 64-bit
when FEAT_D128 is enabled.

FEAT_SYSREG128 adds support for MRRS and MSRR instruction which
helps us to read write to 128-bit system register.
Refer to Arm Architecture Manual for further details.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I1a53db5eac29e56c8fbdcd4961ede3abfcb2411a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge "fix(brbe): allow RME builds with BRBE" into integration

Merge "feat(context-mgmt): introduce EL3/root context" into integration

fix(brbe): allow RME builds with BRBE

It used to be the case that a FEAT_RME build could not be built with
FEAT_BRBE support. BRBE doesn't have a 3-world aware disable and
MDCR_EL3 was not context s

fix(brbe): allow RME builds with BRBE

It used to be the case that a FEAT_RME build could not be built with
FEAT_BRBE support. BRBE doesn't have a 3-world aware disable and
MDCR_EL3 was not context switched to allow for disabling in Realm world.

As of commit 123002f9171384d976d95935b7f566740d69cc68 MDCR_EL3 is
context switched. Since the flag for BRBE support is
ENABLE_BRBE_FOR_NS, move brbe_enable() to only happen for NS world. The
other worlds will see BRBE disabled and branch recording prohibited.
This allows for a build with both RME and BRBE.

Note that EL2 BRBE registers are not context switched. Further work is
needed if non-NS support is required.

Change-Id: I82f0f08399dcd080902477dc9636bc4541685f89
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(context-mgmt): introduce EL3/root context

* This patch adds root context procedure to restore/configure
the registers, which are of importance during EL3 execution.

* EL3/Root context is a s

feat(context-mgmt): introduce EL3/root context

* This patch adds root context procedure to restore/configure
the registers, which are of importance during EL3 execution.

* EL3/Root context is a simple restore operation that overwrites
the following bits: (MDCR_EL3.SDD, SCR_EL3.{EA, SIF}, PMCR_EL0.DP
PSTATE.DIT) while the execution is in EL3.

* It ensures EL3 world maintains its own settings distinct
from other worlds (NS/Realm/SWd). With this in place, the EL3
system register settings is no longer influenced by settings of
incoming worlds. This allows the EL3/Root world to access features
for its own execution at EL3 (eg: Pauth).

* It should be invoked at cold and warm boot entry paths and also
at all the possible exception handlers routing to EL3 at runtime.
Cold and warm boot paths are handled by including setup_el3_context
function in "el3_entrypoint_common" macro, which gets invoked in
both the entry paths.

* At runtime, el3_context is setup at the stage, while we get prepared
to enter into EL3 via "prepare_el3_entry" routine.

Change-Id: I5c090978c54a53bc1c119d1bc5fa77cd8813cdc2
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

feat(sctlr2): add support for FEAT_SCTLR2

Arm v8.9 introduces FEAT_SCTLR2, adding SCTLR2_ELx registers.
Support this, context switching the registers and disabling
traps so lower ELs can access the

feat(sctlr2): add support for FEAT_SCTLR2

Arm v8.9 introduces FEAT_SCTLR2, adding SCTLR2_ELx registers.
Support this, context switching the registers and disabling
traps so lower ELs can access the new registers.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I0c4cba86917b6b065a7e8dd6af7daf64ee18dcda
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

feat(the): add support for FEAT_THE

Arm v8.9 introduces FEAT_THE, adding Translation Hardening Extension
Read-Check-Write mask registers, RCWMASK_EL1 and RCWSMASK_EL1.
Support this, context switchin

feat(the): add support for FEAT_THE

Arm v8.9 introduces FEAT_THE, adding Translation Hardening Extension
Read-Check-Write mask registers, RCWMASK_EL1 and RCWSMASK_EL1.
Support this, context switching the registers and disabling
traps so lower ELs can access the new registers.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I8775787f523639b39faf61d046ef482f73b2a562
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

fix(cpus): modify the fix for Cortex-A75 erratum 764081

Apply the mitigation only for the revision and variant
mentioned in the SDEN.

SDEN Documentation:
https://developer.arm.com/documentation/SDE

fix(cpus): modify the fix for Cortex-A75 erratum 764081

Apply the mitigation only for the revision and variant
mentioned in the SDEN.

SDEN Documentation:
https://developer.arm.com/documentation/SDEN859515/latest

Change-Id: Ifda1f4cb32bdec9a9af29397ddc03bf22a7a87fc
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across core

feat(cm): handle asymmetry for FEAT_TCR2

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. FEAT_TCR2 is one of the features which can be
asymmetric across cores and the respective support is added here.

Adding a function to handle this asymmetry by re-visting the
feature presence on running core.
There are two possible cases:
- If the primary core has the feature and secondary does not have it
then the feature is disabled.
- If the primary does not have the feature and secondary has it then,
the feature need to be enabled in secondary cores.

Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Change-Id: I73a70891d52268ddfa4effe40edf04115f5821ca

show more ...

feat(cm): enhance the cpu_context memory report

Currently, as part of the context_memory report, we explicitly
list the EL3, EL1 and EL2 registers and memory usage per CPU
for each world. The remain

feat(cm): enhance the cpu_context memory report

Currently, as part of the context_memory report, we explicitly
list the EL3, EL1 and EL2 registers and memory usage per CPU
for each world. The remaining bits in the cpu_context_t structure
are grouped and listed as other section.

This patch enhances this part, by individually listing all the
remaining bits (GPREGS, PAUTH_REGS) separately providing
a much detailed overview of the context memory consumption
amongst the registers.

The patch has been tested on the CI with the following patch
and the results are summarised precisely.
[https://review.trustedfirmware.org/c/ci/tf-a-ci-scripts/+/28849]

Change-Id: I16f210b605ddd7900600519520accf1ccd057bc7
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure,

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure, we hold a copy of EL1, EL2
system registers, per world per PE. This context structure is
enormous and will continue to grow bigger with the addition of
new features incorporating new registers.

* Ideally, EL3 should save and restore the system registers at its next
lower exception level, which is EL2 in majority of the configurations.

* This patch aims at optimising the memory allocation in cases, when
the members from the context structure are unused. So el1 system
register context must be omitted when lower EL is always x-EL2.

* "CTX_INCLUDE_EL2_REGS" is the internal build flag which gets set,
when SPD=spmd and SPMD_SPM_AT_SEL2=1 or ENABLE_RME=1.
It indicates, the system registers at EL2 are context switched for
the respective build configuration. Here, there is no need to save
and restore EL1 system registers, while x-EL2 is enabled.

Henceforth, this patch addresses this issue, by taking out the EL1
context at all possible places, while EL2 (CTX_INCLUDE_EL2_REGS) is
enabled, there by saving memory.

Change-Id: Ifddc497d3c810e22a15b1c227a731bcc133c2f4a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

chore(cm): reorganise sctlr_el1 and tcr_el1 ctx code

SCTLR_EL1 and TCR_EL1 regs are included either as part of errata
"ERRATA_SPECULATIVE_AT" or under el1_sysregs_t context structure.
The code to wr

chore(cm): reorganise sctlr_el1 and tcr_el1 ctx code

SCTLR_EL1 and TCR_EL1 regs are included either as part of errata
"ERRATA_SPECULATIVE_AT" or under el1_sysregs_t context structure.
The code to write and read into these context entries, looks
repetitive and is invoked at most places.
This section is refactored to bring them under a static procedure,
keeping the code neat and easier to maintain.

Change-Id: Ib0d8c51bee09e1600c5baaa7f9745083dca9fee1
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

Merge changes from topic "mp/simd_ctxt_mgmt" into integration

* changes:
feat(fvp): allow SIMD context to be put in TZC DRAM
docs(simd): introduce CTX_INCLUDE_SVE_REGS build flag
feat(fvp): ad

Merge changes from topic "mp/simd_ctxt_mgmt" into integration

* changes:
feat(fvp): allow SIMD context to be put in TZC DRAM
docs(simd): introduce CTX_INCLUDE_SVE_REGS build flag
feat(fvp): add Cactus partition manifest for EL3 SPMC
chore(simd): remove unused macros and utilities for FP
feat(el3-spmc): support simd context management upon world switch
feat(trusty): switch to simd_ctx_save/restore apis
feat(pncd): switch to simd_ctx_save/restore apis
feat(spm-mm): switch to simd_ctx_save/restore APIs
feat(simd): add rules to rationalize simd ctxt mgmt
feat(simd): introduce simd context helper APIs
feat(simd): add routines to save, restore sve state
feat(simd): add sve state to simd ctxt struct
feat(simd): add data struct for simd ctxt management

show more ...

feat(simd): introduce simd context helper APIs

This patch adds the common API to save and restore FP and SVE. When SVE
is enabled we save and restore SVE which automatically covers FP. If FP
is enab

feat(simd): introduce simd context helper APIs

This patch adds the common API to save and restore FP and SVE. When SVE
is enabled we save and restore SVE which automatically covers FP. If FP
is enabled while SVE is not, then we save and restore FP only.

The patch uses simd_ctx_t to save and restore both FP and SVE which
means developers need not use fp or sve routines directly. Once all the
calls to fpregs_context_* are replaced with simd_ctx_*, we can remove
fp_regs_t data structure and macros (taken care in a following patch).

simd_ctx_t is currently allocated in section of its own. This will go
into BSS section by default but platform will have option of relocating
it to a different section by overriding in plat.ld.S.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: I090f8b8fa3862e527b6c40385249adc69256bf24

show more ...

feat(simd): add routines to save, restore sve state

This adds assembly routines to save and restore SVE registers. In order
to share between FPU and SVE the code to save and restore FPCR and
FPSR, t

feat(simd): add routines to save, restore sve state

This adds assembly routines to save and restore SVE registers. In order
to share between FPU and SVE the code to save and restore FPCR and
FPSR, the patch converts code for those registers into macro.
Since we will be using simd_ctx_t to save and restore FPU also, we use
offsets in simd_ctx_t for FPSR and FPCR. Since simd_ctx_t has the same
structure at the beginning as fp_regs_t, those offsets should be the
same as CTX_FP_* offsets, when SVE is not enabled. Note that the code
also saves and restores FPEXC32 reg along with FPSR and FPCR.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: I120c02359794aa6bb6376a464a9afe98bd84ae60

show more ...

feat(cm): asymmetric feature support for trbe

This patch checks if the Errata 2938996(Cortex-A520) , 2726228(Cortex-X4)
applies to cores and if affected applies the errata workaround which
disables

feat(cm): asymmetric feature support for trbe

This patch checks if the Errata 2938996(Cortex-A520) , 2726228(Cortex-X4)
applies to cores and if affected applies the errata workaround which
disables TRBE.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I53b037839820c8b3a869f393588302a365d5b97c

show more ...

feat(cm): handle asymmetry for SPE feature

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. SPE is one of the features which can be asymmetric
across cores.

feat(cm): handle asymmetry for SPE feature

With introduction of FEAT_STATE_CHECK_ASYMMETRIC, the asymmetry of cores
can be handled. SPE is one of the features which can be asymmetric
across cores.

Add a function to handle this asymmetry by re-visting the feature
presence on running core.
There are two possible cases:
- If the primary has the feature and secondary does not have it then,
the feature needs to be disabled.
- If the primary does not have the feature and secondary has it then,
the feature need to be enabled

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: Ibb2b88b5ef63b3efcb80801898ae8d8967e5c271

show more ...

feat(cm): support for asymmetric feature among cores

TF-A assumes that all the cores in a platform has architecture feature
parity, this is evident by the fact that primary sets up the
Non-secure co

feat(cm): support for asymmetric feature among cores

TF-A assumes that all the cores in a platform has architecture feature
parity, this is evident by the fact that primary sets up the
Non-secure context of secondary cores.

With changing landscape of platforms (e.g. big/little/mid cores), we are
seeing more and more platforms which has feature asymmetry among cores.
There is also a scenario where certain CPU erratum only applies to one
type of cores and requires a feature to be disabled even it supports
the feature.

To handle these scenarios, introduce a hook in warmboot path which would
be called on the running CPU to override any feature disparity in the
NS context stashed up by primary. Note that, re-checking of feature for
Secure/Realm context is not required as the context is created on
running core itself.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I5a01dbda528fa8481a00fdd098b58a7463ed0e22

show more ...

feat(cpufeat): add new feature state for asymmetric features

Introduce a new feature state CHECK_ASYMMETRIC to cater for the features
which are asymmetric across cores. This state is useful for plat

feat(cpufeat): add new feature state for asymmetric features

Introduce a new feature state CHECK_ASYMMETRIC to cater for the features
which are asymmetric across cores. This state is useful for platforms
which has architectural asymmetric cores (A feature is only present in
one type of core e.g. big).
This state is similar to FEAT_STATE_CHECK (dynamic detection) except
that feature state is also checked on each core during warmboot path and
override the context (just for asymmetric features) which was setup by
core executing CPU_ON call.

Only Non-secure context will be re-checked as secure and realm context
is created on same core.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: Ic78a0b6ca996e0d7881c43da1a6a0c422f528ef3

show more ...