Merge "fix(cpufeat): require FEAT_AMUv1p1 to enable the auxiliary counters" into integration

fix(cpufeat): require FEAT_AMUv1p1 to enable the auxiliary counters

The auxiliary counters are a feature of FEAT_AMUv1p1 but it's possible
to enable them (ENABLE_AMU_AUXILIARY_COUNTERS=1) without en

fix(cpufeat): require FEAT_AMUv1p1 to enable the auxiliary counters

The auxiliary counters are a feature of FEAT_AMUv1p1 but it's possible
to enable them (ENABLE_AMU_AUXILIARY_COUNTERS=1) without enabling
FEAT_AMUv1p1. As a result, the AMU_RESTRICT_COUNTERS may not take
effect, making this configuration potentially insecure.

Fix this by adding a constraints and rejigging auxiliary counter enables
such that they only happen when FEAT_AMUv1p1 has been enabled so that's
more apparent.

Change-Id: I5b5061d603013598f07d70401d68915c016a1a1b
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes from topic "NUMA_AWARE_PER_CPU" into integration

* changes:
docs(maintainers): add per-cpu framework into maintainers.rst
feat(per-cpu): add documentation for per-cpu framework
f

Merge changes from topic "NUMA_AWARE_PER_CPU" into integration

* changes:
docs(maintainers): add per-cpu framework into maintainers.rst
feat(per-cpu): add documentation for per-cpu framework
feat(rdv3): enable numa aware per-cpu for RD-V3-Cfg2
feat(per-cpu): migrate amu_ctx to per-cpu framework
feat(per-cpu): migrate spm_core_context to per-cpu framework
feat(per-cpu): migrate psci_ns_context to per-cpu framework
feat(per-cpu): migrate psci_cpu_pd_nodes to per-cpu framework
feat(per-cpu): migrate rmm_context to per-cpu framework
feat(per-cpu): integrate per-cpu framework into BL31/BL32
feat(per-cpu): introduce framework accessors/definers
feat(per-cpu): introduce linker changes for NUMA aware per-cpu framework
docs(changelog): add scope for per-cpu framework

show more ...

feat(per-cpu): migrate amu_ctx to per-cpu framework

migrate amu_ctx object to the NUMA-aware per-cpu framework
to optimize memory access and to efficiently utilize memory.

Signed-off-by: Sammit Jos

feat(per-cpu): migrate amu_ctx to per-cpu framework

migrate amu_ctx object to the NUMA-aware per-cpu framework
to optimize memory access and to efficiently utilize memory.

Signed-off-by: Sammit Joshi <sammit.joshi@arm.com>
Signed-off-by: Rohit Mathew <rohit.mathew@arm.com>
Change-Id: I55e5e923ddd7307039e0cd9a35e91d908f910d10

show more ...

Merge changes I3d950e72,Id315a8fe,Ib62e6e9b,I1d0475b2 into integration

* changes:
perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context
perf(psci): get PMF timestamps wi

Merge changes I3d950e72,Id315a8fe,Ib62e6e9b,I1d0475b2 into integration

* changes:
perf(cm): drop ZCR_EL3 saving and some ISBs and replace them with root context
perf(psci): get PMF timestamps with no cache flushes if possible
perf(amu): greatly simplify AMU context management
perf(mpmm): greatly simplify MPMM enablement

show more ...

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness

perf(amu): greatly simplify AMU context management

The current code is incredibly resilient to updates to the spec and
has worked quite well so far. However, recent implementations expose a
weakness in that this is rather slow. A large part of it is written in
assembly, making it opaque to the compiler for optimisations. The
future proofness requires reading registers that are effectively
`volatile`, making it even harder for the compiler, as well as adding
lots of implicit barriers, making it hard for the microarchitecutre to
optimise as well.

We can make a few assumptions, checked by a few well placed asserts, and
remove a lot of this burden. For a start, at the moment there are 4
group 0 counters with static assignments. Contexting them is a trivial
affair that doesn't need a loop. Similarly, there can only be up to 16
group 1 counters. Contexting them is a bit harder, but we can do with a
single branch with a falling through switch. If/when both of these
change, we have a pair of asserts and the feature detection mechanism to
guard us against pretending that we support something we don't.

We can drop contexting of the offset registers. They are fully
accessible by EL2 and as such are its responsibility to preserve on
powerdown.

Another small thing we can do, is pass the core_pos into the hook.
The caller already knows which core we're running on, we don't need to
call this non-trivial function again.

Finally, knowing this, we don't really need the auxiliary AMUs to be
described by the device tree. Linux doesn't care at the moment, and any
information we need for EL3 can be neatly placed in a simple array.

All of this, combined with lifting the actual saving out of assembly,
reduces the instructions to save the context from 180 to 40, including a
lot fewer branches. The code is also much shorter and easier to read.

Also propagate to aarch32 so that the two don't diverge too much.

Change-Id: Ib62e6e9ba5be7fb9fb8965c8eee148d5598a5361
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the s

perf(mpmm): greatly simplify MPMM enablement

MPMM is a core-specific microarchitectural feature. It has been present
in every Arm core since the Cortex-A510 and has been implemented in
exactly the same way. Despite that, it is enabled more like an
architectural feature with a top level enable flag. This utilised the
identical implementation.

This duality has left MPMM in an awkward place, where its enablement
should be generic, like an architectural feature, but since it is not,
it should also be core-specific if it ever changes. One choice to do
this has been through the device tree.

This has worked just fine so far, however, recent implementations expose
a weakness in that this is rather slow - the device tree has to be read,
there's a long call stack of functions with many branches, and system
registers are read. In the hot path of PSCI CPU powerdown, this has a
significant and measurable impact. Besides it being a rather large
amount of code that is difficult to understand.

Since MPMM is a microarchitectural feature, its correct placement is in
the reset function. The essence of the current enablement is to write
CPUPPMCR_EL3.MPMM_EN if CPUPPMCR_EL3.MPMMPINCTL == 0. Replacing the C
enablement with an assembly macro in each CPU's reset function achieves
the same effect with just a single close branch and a grand total of 6
instructions (versus the old 2 branches and 32 instructions).

Having done this, the device tree entry becomes redundant. Should a core
that doesn't support MPMM arise, this can cleanly be handled in the
reset function. As such, the whole ENABLE_MPMM_FCONF and platform hooks
mechanisms become obsolete and are removed.

Change-Id: I1d0475b21a1625bb3519f513ba109284f973ffdf
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "refactor(cm): move EL3 registers to global context" into integration

refactor(cm): move EL3 registers to global context

Currently, EL3 context registers are duplicated per-world per-cpu.
Some registers have the same value across all CPUs, so this patch
moves these re

refactor(cm): move EL3 registers to global context

Currently, EL3 context registers are duplicated per-world per-cpu.
Some registers have the same value across all CPUs, so this patch
moves these registers out into a per-world context to reduce
memory usage.

Change-Id: I91294e3d5f4af21a58c23599af2bdbd2a747c54a
Signed-off-by: Elizabeth Ho <elizabeth.ho@arm.com>
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>

show more ...

Merge changes from topic "bk/context_refactor" into integration

* changes:
refactor(cm): enable S-EL2 globally for all contexts
refactor(cm): remove world differentiation for EL2 context restore

Merge changes from topic "bk/context_refactor" into integration

* changes:
refactor(cm): enable S-EL2 globally for all contexts
refactor(cm): remove world differentiation for EL2 context restore
fix(cm): make ICC_SRE_EL2 fixup generic to all worlds
refactor(cm): clean up SCR_EL3 and CPTR_EL3 initialization

show more ...

refactor(cm): clean up SCR_EL3 and CPTR_EL3 initialization

As with MDCR_EL3, setting some bits of these registers is redundant at
reset since they do not matter for EL3 execution and the registers g

refactor(cm): clean up SCR_EL3 and CPTR_EL3 initialization

As with MDCR_EL3, setting some bits of these registers is redundant at
reset since they do not matter for EL3 execution and the registers get
context switched so they get overwritten anyway.

The SCR_EL3.{TWE, TWI, SMD, API, APK} bits only affect lower ELs so
their place is in context management. The API and APK bits are a bit
special as they would get implicitly unset for secure world when
CTX_INCLUDE_PAUTH_REGS is unset. This is now explicit with their normal
world values being always set as PAuth defaults to enabled. The same
sequence is also added to realm world too. The reasoning is the same as
for Secure world - PAuth will be enabled for NS, and unless explicitly
handled by firmware, it should not leak to realm.

The CPTR_EL3.{ESM, EZ, TAM} bits are set by the relevant
feat_enable()s in lib/extensions so they can be skipped too.

CPTR_EL3.TFP is special as it's needed for access to generic floating
point registers even when SVE is not present. So keep it but move to
context management.

This leaves CPTR_EL3.TCPAC which affects several extensions. This bit
was set centrally at reset, however the earliest need for it is in BL2.
So set it in cm_setup_context_common(). However, this CPTR_EL3 is only
restored for BL31 which is clearly not the case. So always restore it.

Finally, setting CPTR_EL3 to a fresh RESET_VAL for each security state
prevents any bits from leaking between them.

Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Change-Id: Ie7095e967bd4a6d6ca6acf314c7086d89fec8900

show more ...

Merge "chore: update to use Arm word across TF-A" into integration

chore: update to use Arm word across TF-A

Align entire TF-A to use Arm in copyright header.

Change-Id: Ief9992169efdab61d0da6bd8c5180de7a4bc2244
Signed-off-by: Govindraj Raja <govindraj.raja@arm.co

chore: update to use Arm word across TF-A

Align entire TF-A to use Arm in copyright header.

Change-Id: Ief9992169efdab61d0da6bd8c5180de7a4bc2244
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>

show more ...

Merge changes from topic "bk/context_refactor" into integration

* changes:
refactor(amu): separate the EL2 and EL3 enablement code
refactor(cpufeat): separate the EL2 and EL3 enablement code

refactor(amu): separate the EL2 and EL3 enablement code

Combining the EL2 and EL3 enablement code necessitates that it must be
called at el3_exit, which is the only place with enough context to make

refactor(amu): separate the EL2 and EL3 enablement code

Combining the EL2 and EL3 enablement code necessitates that it must be
called at el3_exit, which is the only place with enough context to make
the decision of what needs to be set.
Decouple them to allow them to be called from elsewhere. Also take
some time to clarify and simplify AMU code.

The sanity check in the context_restore() is now wrong, as the cpu may
turn off on suspend, thus resetting the value of the counter enables.
Remove it.

Finally, this completes the migration to cm_manage_extensions_el3() and
manage_extensions_nonsecure() so manage_extensions_nonsecure_mixed() is
being removed.

Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: I66399132364c32be66017506bb54cbadd8485577

show more ...

Merge changes from topic "feat_amu_rework" into integration

* changes:
refactor(amu): use new AMU feature check routines
refactor(amu): unify ENABLE_AMU and ENABLE_FEAT_AMUv1

refactor(amu): use new AMU feature check routines

The AMU extension code was using its own feature detection routines.
Replace them with the generic CPU feature handlers (defined in
arch_features.h)

refactor(amu): use new AMU feature check routines

The AMU extension code was using its own feature detection routines.
Replace them with the generic CPU feature handlers (defined in
arch_features.h), which get updated to cover the v1p1 variant as well.

Change-Id: I8540f1e745d7b02a25a6c6cdf2a39d6f5e21f2aa
Signed-off-by: Andre Przywara <andre.przywara@arm.com>

show more ...

Merge "fix(amu): limit virtual offset register access to NS world" into integration

fix(amu): limit virtual offset register access to NS world

Previously the SCR_EL3.AMVOFFEN bit was set for all contexts, this
behavior is incorrect as it allows secure world to access the virtual
of

fix(amu): limit virtual offset register access to NS world

Previously the SCR_EL3.AMVOFFEN bit was set for all contexts, this
behavior is incorrect as it allows secure world to access the virtual
offset registers when it should not be able to. This patch only sets
AMVOFFEN for non-secure world.

Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: I2c61fe0a8a0092df089f1cb2c0d8a45c8c8ad0d3

show more ...

Merge "fix: libc: use long for 64-bit types on aarch64" into integration

fix: libc: use long for 64-bit types on aarch64

Use long instead of long long on aarch64 for 64_t stdint types.
Introduce inttypes.h to properly support printf format specifiers for
fixed width type

fix: libc: use long for 64-bit types on aarch64

Use long instead of long long on aarch64 for 64_t stdint types.
Introduce inttypes.h to properly support printf format specifiers for
fixed width types for such change.

Change-Id: I0bca594687a996fde0a9702d7a383055b99f10a1
Signed-off-by: Scott Branden <scott.branden@broadcom.com>

show more ...

Merge changes from topic "ck/mpmm" into integration

* changes:
docs(maintainers): add Chris Kay to AMU and MPMM
feat(tc): enable MPMM
feat(mpmm): add support for MPMM
feat(amu): enable per-c

Merge changes from topic "ck/mpmm" into integration

* changes:
docs(maintainers): add Chris Kay to AMU and MPMM
feat(tc): enable MPMM
feat(mpmm): add support for MPMM
feat(amu): enable per-core AMU auxiliary counters
docs(amu): add AMU documentation
refactor(amu): refactor enablement and context switching
refactor(amu): detect auxiliary counters at runtime
refactor(amu): detect architected counters at runtime
refactor(amu): conditionally compile auxiliary counter support
refactor(amu): factor out register accesses
refactor(amu)!: privatize unused AMU APIs
refactor(amu)!: remove `PLAT_AMU_GROUP1_COUNTERS_MASK`
build(amu): introduce `amu.mk`
build(fconf)!: clean up source collection
feat(fdt-wrappers): add CPU enumeration utility function
build(fdt-wrappers): introduce FDT wrappers makefile
build(bl2): deduplicate sources
build(bl1): deduplicate sources

show more ...

feat(mpmm): add support for MPMM

MPMM - the Maximum Power Mitigation Mechanism - is an optional
microarchitectural feature present on some Armv9-A cores, introduced
with the Cortex-X2, Cortex-A710 a

feat(mpmm): add support for MPMM

MPMM - the Maximum Power Mitigation Mechanism - is an optional
microarchitectural feature present on some Armv9-A cores, introduced
with the Cortex-X2, Cortex-A710 and Cortex-A510 cores.

MPMM allows the SoC firmware to detect and limit high activity events
to assist in SoC processor power domain dynamic power budgeting and
limit the triggering of whole-rail (i.e. clock chopping) responses to
overcurrent conditions.

This feature is enabled via the `ENABLE_MPMM` build option.
Configuration can be done via FCONF by enabling `ENABLE_MPMM_FCONF`, or
by via the plaform-implemented `plat_mpmm_topology` function.

Change-Id: I77da82808ad4744ece8263f0bf215c5a091c3167
Signed-off-by: Chris Kay <chris.kay@arm.com>

show more ...

feat(amu): enable per-core AMU auxiliary counters

This change makes AMU auxiliary counters configurable on a per-core
basis, controlled by `ENABLE_AMU_AUXILIARY_COUNTERS`.

Auxiliary counters can be

feat(amu): enable per-core AMU auxiliary counters

This change makes AMU auxiliary counters configurable on a per-core
basis, controlled by `ENABLE_AMU_AUXILIARY_COUNTERS`.

Auxiliary counters can be described via the `HW_CONFIG` device tree if
the `ENABLE_AMU_FCONF` build option is enabled, or the platform must
otherwise implement the `plat_amu_topology` function.

A new phandle property for `cpu` nodes (`amu`) has been introduced to
the `HW_CONFIG` specification to allow CPUs to describe the view of
their own AMU:

```
cpu0: cpu@0 {
...

amu = <&cpu0_amu>;
};
```

Multiple cores may share an `amu` handle if they implement the
same set of auxiliary counters.

AMU counters are described for one or more AMUs through the use of a new
`amus` node:

```
amus {
cpu0_amu: amu-0 {
#address-cells = <1>;
#size-cells = <0>;

counter@0 {
reg = <0>;

enable-at-el3;
};

counter@n {
reg = <n>;

...
};
};
};
```

This structure describes the **auxiliary** (group 1) AMU counters.
Architected counters have architecturally-defined behaviour, and as
such do not require DTB entries.

These `counter` nodes support two properties:

- The `reg` property represents the counter register index.
- The presence of the `enable-at-el3` property determines whether
the firmware should enable the counter prior to exiting EL3.

Change-Id: Ie43aee010518c5725a3b338a4899b0857caf4c28
Signed-off-by: Chris Kay <chris.kay@arm.com>

show more ...

refactor(amu): refactor enablement and context switching

This change represents a general refactoring to clean up old code that
has been adapted to account for changes required to enable dynamic
aux

refactor(amu): refactor enablement and context switching

This change represents a general refactoring to clean up old code that
has been adapted to account for changes required to enable dynamic
auxiliary counters.

Change-Id: Ia85e0518f3f65c765f07b34b67744fc869b9070d
Signed-off-by: Chris Kay <chris.kay@arm.com>

show more ...