Merge "feat(cpufeat): enable USE_SPINLOCK_CAS to FEAT_STATE_CHECKED" into integration

feat(cpufeat): enable USE_SPINLOCK_CAS to FEAT_STATE_CHECKED

The FEAT_LSE enablement predates the FEAT_STATE framework and has never
been converted. Since the introduction of USE_SPINLOCK_CAS we've

feat(cpufeat): enable USE_SPINLOCK_CAS to FEAT_STATE_CHECKED

The FEAT_LSE enablement predates the FEAT_STATE framework and has never
been converted. Since the introduction of USE_SPINLOCK_CAS we've gained
lots of quality of life features that allow for better feature
enablement. This patch converts USE_SPINLOCK_CAS to tri-state and adds
it to FEATURE_DETECTION to align with all other features.

Instead of introducing the assembly checking for tri-state, this patch
translates all locking routines to C inline assembly and uses the
standard C helpers. The main benefit is that this gives greater
visibility to the compiler about what the functions are doing and lets
it optimise better. Namely, it is able to allocate registers itself and
inline the functions when LTO is enabled.

An unsuccessful attempt was made to use the instructions directly and
have even flow control in C. This, however, made code very complicated
and less efficient in the tight loops of the spinlock.

The last use of ARM_ARCH_AT_LEAST goes away with this change and so this
macro is removed. It has now been fully superseded by the FEAT_STATE
framework.

This change exposes a limitation - RME_GPT_BITLOCK_BLOCK requires
USE_SPINLOCK_CAS. This patch does not address this in any way but makes
the relationship explicit.

Change-Id: I580081549aceded2dca3e0f4564ee7510a7e56ae
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes Id942c20c,Idd286bea,I8917a26e,Iec8c3477,If3c25dcd, ... into integration

* changes:
feat(cpufeat): enable FEAT_PAuth to FEAT_STATE_CHECKED
perf(cpufeat): centralise PAuth key saving

Merge changes Id942c20c,Idd286bea,I8917a26e,Iec8c3477,If3c25dcd, ... into integration

* changes:
feat(cpufeat): enable FEAT_PAuth to FEAT_STATE_CHECKED
perf(cpufeat): centralise PAuth key saving
refactor(cpufeat): convert FEAT_PAuth setup to C
refactor(cpufeat): prepare FEAT_PAuth for FEATURE_DETECTION
chore(cpufeat): remove PAuth presence checks
feat(cpufeat): enable FEAT_BTI to FEAT_STATE_CHECKED

show more ...

refactor(cpufeat): prepare FEAT_PAuth for FEATURE_DETECTION

Convert the old style is_armv8_3_pauth_present() to the new style
is_feat_pauth_{present, supported}() helpers and hook FEATURE_DETECTION

refactor(cpufeat): prepare FEAT_PAuth for FEATURE_DETECTION

Convert the old style is_armv8_3_pauth_present() to the new style
is_feat_pauth_{present, supported}() helpers and hook FEATURE_DETECTION
into it. This is in preparation for converting FEAT_PAuth to FEAT_STATE.

Change-Id: Iec8c3477fafb2cdae67d39ae4da2cca76a67511a
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(cpufeat): enable FEAT_BTI to FEAT_STATE_CHECKED

Introduce the is_feat_bti_{supported, present}() helpers and replace
checks for ENABLE_BTI with it. Also factor out the setting of
SCTLR_EL3.BT o

feat(cpufeat): enable FEAT_BTI to FEAT_STATE_CHECKED

Introduce the is_feat_bti_{supported, present}() helpers and replace
checks for ENABLE_BTI with it. Also factor out the setting of
SCTLR_EL3.BT out of the PAuth enablement and place it in the respective
entrypoints where we initialise SCTLR_EL3. This makes PAuth
self-contained and SCTLR_EL3 initialisation centralised.

Change-Id: I0c0657ff1e78a9652cd2cf1603478283dc01f17b
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "fix(cpufeat): include FEAT_MOPS declaration in aarch32 header" into integration

fix(cpufeat): include FEAT_MOPS declaration in aarch32 header

This patch adds the missing is_feat_mops_supported() declaration
in aarch32 header.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakas

fix(cpufeat): include FEAT_MOPS declaration in aarch32 header

This patch adds the missing is_feat_mops_supported() declaration
in aarch32 header.

Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: I875f65defe23912351f9ef18555a5b0a0e53717d

show more ...

Merge changes I3d950e72,Id315a8fe,Ib62e6e9b,I1d0475b2 into integration

* changes:
perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context
perf(psci): get PMF timestamps wi

Merge changes I3d950e72,Id315a8fe,Ib62e6e9b,I1d0475b2 into integration

* changes:
perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context
perf(psci): get PMF timestamps with no cache flushes if possible
perf(amu): greatly simplify AMU context management
perf(mpmm): greatly simplify MPMM enablement

show more ...

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness in that this is rather slow. A large part of it is written in
assembly, making it opaque to the compiler for optimisations. The
future proofness requires reading registers that are effectively
`volatile`, making it even harder for the compiler, as well as adding
lots of implicit barriers, making it hard for the microarchitecutre to
optimise as well.

We can make a few assumptions, checked by a few well placed asserts, and
remove a lot of this burden. For a start, at the moment there are 4
group 0 counters with static assignments. Contexting them is a trivial
affair that doesn't need a loop. Similarly, there can only be up to 16
group 1 counters. Contexting them is a bit harder, but we can do with a
single branch with a falling through switch. If/when both of these
change, we have a pair of asserts and the feature detection mechanism to
guard us against pretending that we support something we don't.

We can drop contexting of the offset registers. They are fully
accessible by EL2 and as such are its responsibility to preserve on
powerdown.

Another small thing we can do, is pass the core_pos into the hook.
The caller already knows which core we're running on, we don't need to
call this non-trivial function again.

Finally, knowing this, we don't really need the auxiliary AMUs to be
described by the device tree. Linux doesn't care at the moment, and any
information we need for EL3 can be neatly placed in a simple array.

All of this, combined with lifting the actual saving out of assembly,
reduces the instructions to save the context from 180 to 40, including a
lot fewer branches. The code is also much shorter and easier to read.

Also propagate to aarch32 so that the two don't diverge too much.

Change-Id: Ib62e6e9ba5be7fb9fb8965c8eee148d5598a5361
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA" into integration

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest

feat(cpufeat): add ENABLE_FEAT_LS64_ACCDATA

Armv8.6 introduced the FEAT_LS64 extension, which provides a 64 *byte*
store instruction. A related instruction is ST64BV0, which will replace
the lowest 32 bits of the data with a value taken from the ACCDATA_EL1
system register (so that EL0 cannot alter them).
Using that ST64BV0 instruction and accessing the ACCDATA_EL1 system
register is guarded by two SCR_EL3 bits, which we should set to avoid a
trap into EL3, when lower ELs use one of those.

Add the required bits and pieces to make this feature usable:
- Add the ENABLE_FEAT_LS64_ACCDATA build option (defaulting to 0).
- Add the CPUID and SCR_EL3 bit definitions associated with FEAT_LS64.
- Add a feature check to check for the existing four variants of the
LS64 feature and detect future extensions.
- Add code to save and restore the ACCDATA_EL1 register on
secure/non-secure context switches.
- Enable the feature with runtime detection for FVP and Arm FPGA.

Please note that the *basic* FEAT_LS64 feature does not feature any trap
bits, it's only the addition of the ACCDATA_EL1 system register that
adds these traps and the SCR_EL3 bits.

Change-Id: Ie3e2ca2d9c4fbbd45c0cc6089accbb825579138a
Signed-off-by: Andre Przywara <andre.przywara@arm.com>

show more ...

Merge "feat(d128): add support for FEAT_D128" into integration

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEA

feat(d128): add support for FEAT_D128

This patch disables trapping to EL3 when the FEAT_D128
specific registers are accessed by setting the SCR_EL3.D128En bit.

If FEAT_D128 is implemented, then FEAT_SYSREG128 is implemented.
With FEAT_SYSREG128 certain system registers are treated as 128-bit,
so we should be context saving and restoring 128-bits instead of 64-bit
when FEAT_D128 is enabled.

FEAT_SYSREG128 adds support for MRRS and MSRR instruction which
helps us to read write to 128-bit system register.
Refer to Arm Architecture Manual for further details.

Change the FVP platform to default to handling this as a dynamic option
so the right decision can be made by the code at runtime.

Change-Id: I1a53db5eac29e56c8fbdcd4961ede3abfcb2411a
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge "refactor(cpufeat): feat detect helpers inlining" into integration

refactor(cpufeat): feat detect helpers inlining

Force inlining feat detect helpers such that context save/restore
operations are flattened with sequences of ID regs read and conditional
instructions

refactor(cpufeat): feat detect helpers inlining

Force inlining feat detect helpers such that context save/restore
operations are flattened with sequences of ID regs read and conditional
instructions for system registers read/write. This is opposed to current
situation where with -Os optimization level, feat detect helpers get
called through non-inlined sequences of branch-link+ret.

Signed-off-by: Olivier Deprez <olivier.deprez@arm.com>
Change-Id: I2633442fb0e69e4a4ed13467e65846fb66d214f6

show more ...

Merge changes from topic "sm/feat_detect" into integration

* changes:
refactor(cpufeat): restore functions in detect_arch_features
refactor(cpufeat): add macro to simplify is_feat_xx_present
c

Merge changes from topic "sm/feat_detect" into integration

* changes:
refactor(cpufeat): restore functions in detect_arch_features
refactor(cpufeat): add macro to simplify is_feat_xx_present
chore: simplify the macro names in ENABLE_FEAT mechanism

show more ...

refactor(cpufeat): add macro to simplify is_feat_xx_present

In this patch, we are trying to introduce the wrapper macro
CREATE_FEATURE_PRESENT to get the following capability and
align it for all th

refactor(cpufeat): add macro to simplify is_feat_xx_present

In this patch, we are trying to introduce the wrapper macro
CREATE_FEATURE_PRESENT to get the following capability and
align it for all the features:

-> is_feat_xx_present(): Does Hardware implement the feature.
-> uniformity in naming the function across multiple features.
-> improved readability

The is_feat_xx_present() is implemented to check if the hardware
implements the feature and does not take into account the
ENABLE_FEAT_XXX flag enabled/disabled in software.

- CREATE_FEATURE_PRESENT(name, idreg, shift, mask, idval)
The wrapper macro reduces the function to a single line and
creates the is_feat_xx_present function that checks the
id register based on the shift and mask values and compares
this against a determined idvalue.

Change-Id: I7b91d2c9c6fbe55f94c693aa1b2c50be54fb9ecc
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

chore: simplify the macro names in ENABLE_FEAT mechanism

Currently, the macros used to denote feature implementation
in hardware follow a random pattern with a few macros having
suffix as SUPPORTED

chore: simplify the macro names in ENABLE_FEAT mechanism

Currently, the macros used to denote feature implementation
in hardware follow a random pattern with a few macros having
suffix as SUPPORTED and a few using the suffix IMPLEMENTED.
This patch aligns the macro names uniformly using the suffix
IMPLEMENTED across all the features and removes unused macros
pertaining to the Enable feat mechanism.

FEAT_SUPPORTED --> FEAT_IMPLEMENTED
FEAT_NOT_SUPPORTED --> FEAT_NOT_IMPLEMENTED

Change-Id: I61bb7d154b23f677b80756a4b6a81f74b10cd24f
Signed-off-by: Sona Mathew <sonarebecca.mathew@arm.com>

show more ...

Merge changes from topic "mte_fixes" into integration

* changes:
build(changelog): move mte to mte2
refactor(mte): remove mte, mte_perm

refactor(mte): remove mte, mte_perm

Currently both FEAT_MTE and FEAT_MTE_PERM aren't used for enabling
of any feature bits in EL3. So remove both FEAT handling.

All mte regs that are currently cont

refactor(mte): remove mte, mte_perm

Currently both FEAT_MTE and FEAT_MTE_PERM aren't used for enabling
of any feature bits in EL3. So remove both FEAT handling.

All mte regs that are currently context saved/restored are needed
only when FEAT_MTE2 is enabled, so move to usage of FEAT_MTE2 and
remove FEAT_MTE usage.

BREAKING CHANGE: Any platform or downstream code trying to use
SCR_EL3.ATA bit(26) will see failures as this is now moved to be
used only with FEAT_MTE2 with
commit@ef0d0e5478a3f19cbe70a378b9b184036db38fe2

Change-Id: Id01e154156571f7792135639e17dc5c8d0e17cf8
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge changes from topic "mp/undef_injection" into integration

* changes:
feat(el3-runtime): introduce UNDEF injection to lower EL
feat(cpufeat): added few helper functions

feat(cpufeat): added few helper functions

Following utility functions/bit definitions done
- Write a helper function to return the presence of following features
- FEAT_UAO
- FEAT_EBEP

feat(cpufeat): added few helper functions

Following utility functions/bit definitions done
- Write a helper function to return the presence of following features
- FEAT_UAO
- FEAT_EBEP
- FEAT_SEBEP
- FEAT_SSBS
- FEAT_NMI
- FEAT_PAN
- Add definition of some missing bits of SPSR.
- Add GCSCR_EL1 register encoding and accessor function.

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: Ifcead0dd8e3b32096e4ab810dde5d582a889785a

show more ...

Merge "feat(mte): add mte2 feat" into integration

feat(mte): add mte2 feat

Add support for feat mte2. tfsr_el2 is available only with mte2,
however currently its context_save/restore is done with mte rather than
mte2, so introduce 'is_feat_mte2_sup

feat(mte): add mte2 feat

Add support for feat mte2. tfsr_el2 is available only with mte2,
however currently its context_save/restore is done with mte rather than
mte2, so introduce 'is_feat_mte2_supported' to check mte2.

Change-Id: I108d9989a8f5b4d1d2f3b9865a914056fa566cf2
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge "feat(cpufeat): add feature detection for FEAT_CSV2_3" into integration