Merge changes from topic "bk/errata_speed" into integration

* changes:
refactor(cpus): declare runtime errata correctly
perf(cpus): make reset errata do fewer branches
perf(cpus): inline the i

Merge changes from topic "bk/errata_speed" into integration

* changes:
refactor(cpus): declare runtime errata correctly
perf(cpus): make reset errata do fewer branches
perf(cpus): inline the init_cpu_data_ptr function
perf(cpus): inline the reset function
perf(cpus): inline the cpu_get_rev_var call
perf(cpus): inline cpu_rev_var checks
refactor(cpus): register DSU errata with the errata framework's wrappers
refactor(cpus): convert checker functions to standard helpers
refactor(cpus): convert the Cortex-A65 to use the errata framework
fix(cpus): declare reset errata correctly

show more ...

perf(cpus): inline the init_cpu_data_ptr function

Similar to the reset function inline, inline this too to not do a costly
branch with no extra cost.

Change-Id: I54cc399e570e9d0f373ae13c7224d32dbdf

perf(cpus): inline the init_cpu_data_ptr function

Similar to the reset function inline, inline this too to not do a costly
branch with no extra cost.

Change-Id: I54cc399e570e9d0f373ae13c7224d32dbdfae1e5
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(rmm): add support for BRBCR_EL2 register for feat_brbe

Currently BRBE is being disabled for Realm world in EL3 by
switching the SBRBE bit in mdcr_el3 register to 0b00.
The patch removes the swit

fix(rmm): add support for BRBCR_EL2 register for feat_brbe

Currently BRBE is being disabled for Realm world in EL3 by
switching the SBRBE bit in mdcr_el3 register to 0b00.
The patch removes the switching of SBRBE bits, and adds
context switch of BRBCR_EL2 register.

Change-Id: I66ca13edefc37e40fa265fd438b0b66f7d09b4bb
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

Merge "fix(pubsub): make sure LTO doesn't garbage collect the handlers" into integration

fix(pubsub): make sure LTO doesn't garbage collect the handlers

We never directly reference the event handlers so they look like fair
game to be garbage collected when building with LTO.

Tell the c

fix(pubsub): make sure LTO doesn't garbage collect the handlers

We never directly reference the event handlers so they look like fair
game to be garbage collected when building with LTO.

Tell the compiler that we definitely need them and to leave them alone.

Change-Id: Iac672ce85e20328d25acbc3f5e544ad157eebf48
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(cm): fix context management SYSREG128 write macros

This patch fixes a bug which was introduced in commit
3065513 related to improper saving of EL1 context in the
context management library code

fix(cm): fix context management SYSREG128 write macros

This patch fixes a bug which was introduced in commit
3065513 related to improper saving of EL1 context in the
context management library code when using 128-bit
system registers.

Bug explanation:
The function el1_sysregs_context_save still used the normal
macros that read all the system registers related to the EL1
context, which then involved casting them to uint64_t and
eventually writing them to a memory structure. This means that
the context management library was saving EL1-related SYSREG128
registers with the upper 64 bits zeroed out.

Alternative macros had previously been introduced for the EL2
context in the aforementioned commit, but not for EL1.

Some refactoring has also been done as part of this patch:
- Re-added "common" back to write_el2_ctx_common_sysreg128
- Added dummy SYSREG128 macros for cases when some features
are disabled
- Removed some newlines

Change-Id: I15aa2190794ac099a493e5f430220b1c81e1b558
Signed-off-by: Igor Podgainõi <igor.podgainoi@arm.com>

show more ...

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest 32 bits of the data with a value taken from the ACCDATA_EL1
system register (so that EL0 cannot alter them).
Using that ST64BV0 instruction and accessing the ACCDATA_EL1 system
register is guarded by two SCR_EL3 bits, which we should set to avoid a
trap into EL3, when lower ELs use one of those.

Add the required bits and pieces to make this feature usable:
- Add the ENABLE_FEAT_LS64_ACCDATA build option (defaulting to 0).
- Add the CPUID and SCR_EL3 bit definitions associated with FEAT_LS64.
- Add a feature check to check for the existing four variants of the
LS64 feature and detect future extensions.
- Add code to save and restore the ACCDATA_EL1 register on
secure/non-secure context switches.
- Enable the feature with runtime detection for FVP and Arm FPGA.

Please note that the *basic* FEAT_LS64 feature does not feature any trap
bits, it's only the addition of the ACCDATA_EL1 system register that
adds these traps and the SCR_EL3 bits.

Change-Id: Ie3e2ca2d9c4fbbd45c0cc6089accbb825579138a
Signed-off-by: Andre Przywara <andre.przywara@arm.com>

show more ...

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEA

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEAT_SYSREG128 is implemented.
With FEAT_SYSREG128 certain system registers are treated as 128-bit,
so we should be context saving and restoring 128-bits instead of 64-bit
when FEAT_D128 is enabled.

FEAT_SYSREG128 adds support for MRRS and MSRR instruction which
helps us to read write to 128-bit system register.
Refer to Arm Architecture Manual for further details.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I1a53db5eac29e56c8fbdcd4961ede3abfcb2411a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge "fix(el3-runtime): correct CASSERT for cpu data size" into integration

fix(el3-runtime): correct CASSERT for cpu data size

Build breaks when EL3_EXCEPTION_HANDLING option is enabled. The
CPU_DATA_SIZE macro does not consider the size required to save the
ehf_data field

fix(el3-runtime): correct CASSERT for cpu data size

Build breaks when EL3_EXCEPTION_HANDLING option is enabled. The
CPU_DATA_SIZE macro does not consider the size required to save the
ehf_data field of cpu_data structure.

include/lib/el3_runtime/cpu_data.h:163:17: error: size of array
'assert_cpu_data_size_mismatch' is negative
assert_cpu_data_size_mismatch);

This patch adds support to consider ehf_data field to calculate the
CPU_DATA_SIZE macro. Also adds relevant checks and asserts if the
ehf_data field is not considered.

Signed-off-by: Omkar Anand Kulkarni <omkar.kulkarni@arm.com>
Change-Id: I3c11b2982f4a612ce28e46848b5c5035a8f8efc2

show more ...

feat(sctlr2): add support for FEAT_SCTLR2

Arm v8.9 introduces FEAT_SCTLR2, adding SCTLR2_ELx registers.
Support this, context switching the registers and disabling
traps so lower ELs can access the

feat(sctlr2): add support for FEAT_SCTLR2

Arm v8.9 introduces FEAT_SCTLR2, adding SCTLR2_ELx registers.
Support this, context switching the registers and disabling
traps so lower ELs can access the new registers.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I0c4cba86917b6b065a7e8dd6af7daf64ee18dcda
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

feat(the): add support for FEAT_THE

Arm v8.9 introduces FEAT_THE, adding Translation Hardening Extension
Read-Check-Write mask registers, RCWMASK_EL1 and RCWSMASK_EL1.
Support this, context switchin

feat(the): add support for FEAT_THE

Arm v8.9 introduces FEAT_THE, adding Translation Hardening Extension
Read-Check-Write mask registers, RCWMASK_EL1 and RCWSMASK_EL1.
Support this, context switching the registers and disabling
traps so lower ELs can access the new registers.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I8775787f523639b39faf61d046ef482f73b2a562
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure,

refactor(cm): remove el1 context when SPMD_SPM_AT_SEL2=1

* Currently, EL1 context is included in cpu_context_t by default
for all the build configurations.
As part of the cpu context structure, we hold a copy of EL1, EL2
system registers, per world per PE. This context structure is
enormous and will continue to grow bigger with the addition of
new features incorporating new registers.

* Ideally, EL3 should save and restore the system registers at its next
lower exception level, which is EL2 in majority of the configurations.

* This patch aims at optimising the memory allocation in cases, when
the members from the context structure are unused. So el1 system
register context must be omitted when lower EL is always x-EL2.

* "CTX_INCLUDE_EL2_REGS" is the internal build flag which gets set,
when SPD=spmd and SPMD_SPM_AT_SEL2=1 or ENABLE_RME=1.
It indicates, the system registers at EL2 are context switched for
the respective build configuration. Here, there is no need to save
and restore EL1 system registers, while x-EL2 is enabled.

Henceforth, this patch addresses this issue, by taking out the EL1
context at all possible places, while EL2 (CTX_INCLUDE_EL2_REGS) is
enabled, there by saving memory.

Change-Id: Ifddc497d3c810e22a15b1c227a731bcc133c2f4a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

chore(cm): reorganise sctlr_el1 and tcr_el1 ctx code

SCTLR_EL1 and TCR_EL1 regs are included either as part of errata
"ERRATA_SPECULATIVE_AT" or under el1_sysregs_t context structure.
The code to wr

chore(cm): reorganise sctlr_el1 and tcr_el1 ctx code

SCTLR_EL1 and TCR_EL1 regs are included either as part of errata
"ERRATA_SPECULATIVE_AT" or under el1_sysregs_t context structure.
The code to write and read into these context entries, looks
repetitive and is invoked at most places.
This section is refactored to bring them under a static procedure,
keeping the code neat and easier to maintain.

Change-Id: Ib0d8c51bee09e1600c5baaa7f9745083dca9fee1
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

Merge changes from topic "mp/simd_ctxt_mgmt" into integration

* changes:
feat(fvp): allow SIMD context to be put in TZC DRAM
docs(simd): introduce CTX_INCLUDE_SVE_REGS build flag
feat(fvp): ad

Merge changes from topic "mp/simd_ctxt_mgmt" into integration

* changes:
feat(fvp): allow SIMD context to be put in TZC DRAM
docs(simd): introduce CTX_INCLUDE_SVE_REGS build flag
feat(fvp): add Cactus partition manifest for EL3 SPMC
chore(simd): remove unused macros and utilities for FP
feat(el3-spmc): support simd context management upon world switch
feat(trusty): switch to simd_ctx_save/restore apis
feat(pncd): switch to simd_ctx_save/restore apis
feat(spm-mm): switch to simd_ctx_save/restore APIs
feat(simd): add rules to rationalize simd ctxt mgmt
feat(simd): introduce simd context helper APIs
feat(simd): add routines to save, restore sve state
feat(simd): add sve state to simd ctxt struct
feat(simd): add data struct for simd ctxt management

show more ...

chore(simd): remove unused macros and utilities for FP

Now that we have switched to simd context save/restore apis
which uses simd_context_t data structures, we can safely remove
any macros, helpers

chore(simd): remove unused macros and utilities for FP

Now that we have switched to simd context save/restore apis
which uses simd_context_t data structures, we can safely remove
any macros, helpers and utilities used in the old way of
FPU context management.

Change-Id: I27a636dd07bf5c4cb99fd25b9a204d55b525b677
Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>

show more ...

feat(simd): introduce simd context helper APIs

This patch adds the common API to save and restore FP and SVE. When SVE
is enabled we save and restore SVE which automatically covers FP. If FP
is enab

feat(simd): introduce simd context helper APIs

This patch adds the common API to save and restore FP and SVE. When SVE
is enabled we save and restore SVE which automatically covers FP. If FP
is enabled while SVE is not, then we save and restore FP only.

The patch uses simd_ctx_t to save and restore both FP and SVE which
means developers need not use fp or sve routines directly. Once all the
calls to fpregs_context_* are replaced with simd_ctx_*, we can remove
fp_regs_t data structure and macros (taken care in a following patch).

simd_ctx_t is currently allocated in section of its own. This will go
into BSS section by default but platform will have option of relocating
it to a different section by overriding in plat.ld.S.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: I090f8b8fa3862e527b6c40385249adc69256bf24

show more ...

feat(simd): add sve state to simd ctxt struct

This patch extends the simd data structure to hold the SVE state. A new
build flag CTX_INCLUDE_SVE_REGS is introduced in this patch to enable
SVE contex

feat(simd): add sve state to simd ctxt struct

This patch extends the simd data structure to hold the SVE state. A new
build flag CTX_INCLUDE_SVE_REGS is introduced in this patch to enable
SVE context management.

Necessary precautions are taken such as ensuring the field offsets are
not changed and necessary padding is added for alignment reasons.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: I54f5222c7d8c68638105003f68caa511d347cd60

show more ...

feat(simd): add data struct for simd ctxt management

From a context management perspective, we refer to SVE and FP state
as SIMD. SVE can be considered as a superset of FP/NEON functionality
and the

feat(simd): add data struct for simd ctxt management

From a context management perspective, we refer to SVE and FP state
as SIMD. SVE can be considered as a superset of FP/NEON functionality
and there is a architectural dependency between the two due to the fact
that FP registers occupy the bottom 128 bits of SVE Z vector registers.

This patch introduces a new data structure to hold the FP state of a CPU
as needed for context management operations during world switch events.
It will evolve in next patches to capture the SVE state as well.

Having this unified data structure is necessary for us to implement
SIMD context management while catering to various build configurations.

Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Signed-off-by: Okash Khawaja <okash@google.com>
Change-Id: If2a9a7526883780df49bbd3dade2f09f7c3d95ae

show more ...

feat(cm): support for asymmetric feature among cores

TF-A assumes that all the cores in a platform has architecture feature
parity, this is evident by the fact that primary sets up the
Non-secure co

feat(cm): support for asymmetric feature among cores

TF-A assumes that all the cores in a platform has architecture feature
parity, this is evident by the fact that primary sets up the
Non-secure context of secondary cores.

With changing landscape of platforms (e.g. big/little/mid cores), we are
seeing more and more platforms which has feature asymmetry among cores.
There is also a scenario where certain CPU erratum only applies to one
type of cores and requires a feature to be disabled even it supports
the feature.

To handle these scenarios, introduce a hook in warmboot path which would
be called on the running CPU to override any feature disparity in the
NS context stashed up by primary. Note that, re-checking of feature for
Secure/Realm context is not required as the context is created on
running core itself.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I5a01dbda528fa8481a00fdd098b58a7463ed0e22

show more ...

refactor(cm): convert el1-ctx assembly offset entries to c structure

Currently the EL1 part of the context structure (el1_sysregs_t),
is coupled with feature flags reducing the context memory alloca

refactor(cm): convert el1-ctx assembly offset entries to c structure

Currently the EL1 part of the context structure (el1_sysregs_t),
is coupled with feature flags reducing the context memory allocation
for platforms, that don't enable/support all the architectural
features at once.

Similar to the el2 context optimization commit-"d6af234" this patch
further improves this section by converting the assembly context-offset
entries into a c structure. It relies on garbage collection of the
linker removing unreferenced structures from memory, as well as aiding
in readability and future maintenance. Additionally, it eliminates
the #ifs usage in 'context_mgmt.c' source file.

Change-Id: If6075931cec994bc89231241337eccc7042c5ede
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

feat(cm): add explicit context entries for ERRATA_SPECULATIVE_AT

* Currently, "ERRATA_SPECUALTIVE_AT" errata is enabled by default
for few cores and they need context entries for saving and
rest

feat(cm): add explicit context entries for ERRATA_SPECULATIVE_AT

* Currently, "ERRATA_SPECUALTIVE_AT" errata is enabled by default
for few cores and they need context entries for saving and
restoring EL1 regs "SCTLR_EL1 and TCR_EL1" registers at all times.

* This prevents the mechanism of decoupling EL1 and EL2 registers,
as EL3 firmware shouldn't be handling both simultaneously.

* Depending on the build configuration either EL1 or EL2 context
structures need to included, which would result in saving a good
amount of context memory.

* In order to achieve this it's essential to have explicit context
entries for registers supporting "ERRATA_SPECULATIVE_AT".

* This patch adds two context entries under "errata_speculative_at"
structure to assist this errata and thereby allows decoupling
EL1 and EL2 context structures.

Change-Id: Ia50626eea8fb64899a2e2d81622adbe07fe77d65
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

feat(fgt2): add support for FEAT_FGT2

This patch disables trapping to EL3 when the FEAT_FGT2
specific trap registers are accessed by setting the
SCR_EL3.FGTEn2 bit

Signed-off-by: Arvind Ram Prakash

feat(fgt2): add support for FEAT_FGT2

This patch disables trapping to EL3 when the FEAT_FGT2
specific trap registers are accessed by setting the
SCR_EL3.FGTEn2 bit

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I6d2b614affb9067b2bc3d7bf0ae7d169d031592a

show more ...

feat(cm): context switch MDCR_EL3 register

Currently MDCR_EL3 register value is same for all the
worlds(Non-secure, Secure, Realm and Root).

With this approach, features enable/disable settings
rem

feat(cm): context switch MDCR_EL3 register

Currently MDCR_EL3 register value is same for all the
worlds(Non-secure, Secure, Realm and Root).

With this approach, features enable/disable settings
remain same across all the worlds. This is not ideal as
there must be flexibility in controlling feature as per
the requirements for individual world.

The patch addresses this by providing MDCR_EL3 a per world
value. Features with identical values for all the worlds are
grouped under ``manage_extensions_common`` API.

Change-Id: Ibc068d985fe165d8cb6d0ffb84119bffd743b3d1
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

refactor(cm): move mpam registers into el2 context

* FEAT_MPAM related EL2 registers are placed explicitly outside
the EL2 context in the cpu_context_t structure.

* With EL2 registers now coupled

refactor(cm): move mpam registers into el2 context

* FEAT_MPAM related EL2 registers are placed explicitly outside
the EL2 context in the cpu_context_t structure.

* With EL2 registers now coupled with dependent features, this
patch moves them to the el2_context structure "el2_sysregs_t".

* Further, converting the assembly context-offset entries into a
c structure. It relies on garbage collection of the linker
removing unreferenced structures from memory, as well as aiding
in readability and future maintenance.

Change-Id: Ib784bc8d2fbe35a8a47a569426d8663282ec06aa
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...