refactor(aarch64): remove crash reporting's dependency on cpu_data

Crash reporting is useful as early as possible, even before most of the
runtime has been set up. This means that all of its depende

refactor(aarch64): remove crash reporting's dependency on cpu_data

Crash reporting is useful as early as possible, even before most of the
runtime has been set up. This means that all of its dependencies,
currently only cpu_data, must be set up as early as possible too. This
can be constraining as fiddling with the general EL3 runtime from the
early entrypoint is very difficult. So remove the cpu_data dependency.
Further benefits are that crash reporting will work even earlier (during
cpu reset functions!) and also in other BLs.

Change-Id: I92bb6b3921c6dec10560f8341b3bca5cdacfb492
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(el3-runtime): resolve essential-type mismatch

This resolves MISRA C:2012 Rule 10.7 violation
where a composite expression involved operands of differing
essential types, causing unintended impli

fix(el3-runtime): resolve essential-type mismatch

This resolves MISRA C:2012 Rule 10.7 violation
where a composite expression involved operands of differing
essential types, causing unintended implicit conversions.

The fix ensures all operands in the expression have matching
essential types by introducing explicit casts,
preventing unsafe or inconsistent arithmetic operations.

Change-Id: If01dfe78e7a5cffc8b0efa6ac969b262e236852b
Signed-off-by: Taticharla Venkata Sai <venkatasai.taticharla@amd.com>

show more ...

fix(cpufeat): prevent FEAT_AMU counters 2 and 3 from counting across worlds

FEAT_AMU has 4 architected counters. The lower 2, CPU_CYCLES and
CNT_CYCLES, are not considered to be side channels due to

fix(cpufeat): prevent FEAT_AMU counters 2 and 3 from counting across worlds

FEAT_AMU has 4 architected counters. The lower 2, CPU_CYCLES and
CNT_CYCLES, are not considered to be side channels due to their low
resolution and general availability of the data elsewhere. As such, they
are used for critical performance tuning and are expected to never be
turned off or context switched when switching worlds.

The upper 2 counters, INST_RETIRED and STALL_BACKEND_MEM, are different.
The data they provide is non-critical and expose new information that
could be used as a timing side channel, especially of Secure world. This
patch adds context switching of these two counters to prevent any such
side channel.

This is not done for group 1 auxiliary counters as those are IMP DEF and
are inaccessible by default unless overriden by the platform (with
AMU_RESTRICT_COUNTERS).

Change-Id: Ib4b946abb810e36736cabb9b84cd837308b4e761
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(psci): make CMOs target the whole psci_cpu_data_t

psci_cpu_data_t is tiny - on AArch64 it's 12 bytes. Cache maintenance
operations (CMOs) operate on cache lines which are much bigger - usua

refactor(psci): make CMOs target the whole psci_cpu_data_t

psci_cpu_data_t is tiny - on AArch64 it's 12 bytes. Cache maintenance
operations (CMOs) operate on cache lines which are much bigger - usually
64 bytes long. As such, issuing a cache clean for a member in the middle
of psci_cpu_data_t won't necessarily have the expected effect. The
member will be cleaned, sure, but so will the rest of the cache line
along with it. If the struct happens to straddle cache lines this will
lead to the next 52 bytes, most of which not belonging to
psci_cpu_data_t, being cleaned as well and the start of psci_cpu_data_t
not being cleaned at all.

This is not a problem because of the per-cpu (and cpu_data before it)
section - it is cache size aligned and all data within a single section
belongs to the same core so overdoing cache cleans won't have strange
side effects.

Regardless, this patch clarifies CMOs around psci_cpu_data_t by always
targeting the whole structure. To make sure there is never a situation
where it straddles cache lines and this causes weird side effect, its
alignment is set to the size of the structure to make sure it is always
on the same cache line.

Change-Id: I5d82ee6bb2ce0ed3c6a7e4abb7aa890f5e3bd0af
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

perf(cpus): reduce the footprint of errata reporting

Since the advent of spin_trylock() it's possible to combine the spinlock
with the errata_reported field. If the spinlock is only acquired with a

perf(cpus): reduce the footprint of errata reporting

Since the advent of spin_trylock() it's possible to combine the spinlock
with the errata_reported field. If the spinlock is only acquired with a
non-blocking call then a successful call means reporting should be done
and an unsuccessful one means that reporting would have been done by
whoever acquired it. This relies on the lock never being released which
this patch does. The effect is a smaller memory footprint and a smaller
runtime.

Change-Id: I215a84bd2c91e33703349c41fc59f654f7764b2f
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes I1a57de22,If97ea5fd into integration

* changes:
feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock
fix(locks): restore spin_trylock's ability

Merge changes I1a57de22,If97ea5fd into integration

* changes:
feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock
fix(locks): restore spin_trylock's ability to acquire a lock

show more ...

Merge changes I411af9d1,I89813759 into integration

* changes:
feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry
refactor(el3-runtime): factor out handler

Merge changes I411af9d1,I89813759 into integration

* changes:
feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry
refactor(el3-runtime): factor out handler fetching code

show more ...

Merge changes I20c97011,Ia1facabb into integration

* changes:
fix(rk3576): shorten names to fit into the allocated space
fix(debugfs): allocate enough space to fit all names

Merge changes from topic "xl/x925-errata" into integration

* changes:
fix(cpus): workaround for Cortex-X925 erratum 3865185
fix(cpus): workaround for Cortex-X925 erratum 3730893
fix(cpus): wor

Merge changes from topic "xl/x925-errata" into integration

* changes:
fix(cpus): workaround for Cortex-X925 erratum 3865185
fix(cpus): workaround for Cortex-X925 erratum 3730893
fix(cpus): workaround for Cortex-X925 erratum 3692980
fix(cpus): workaround for Cortex-X925 erratum 3324334
fix(cpus): workaround for Cortex-X925 erratum 2933290
fix(cpus): workaround for Cortex-X925 erratum 2922378
fix(cpus): workaround for Cortex-X925 erratum 2921199

show more ...

fix(cpus): workaround for Cortex-X925 erratum 3865185

Cortex-X925 erratum 3865185 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

Load issued to Non-Cacheable or De

fix(cpus): workaround for Cortex-X925 erratum 3865185

Cortex-X925 erratum 3865185 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

Load issued to Non-Cacheable or Device GRE memory can
read stale data brought in by an earlier load to the
same cache-line thereby violating ordering requirements.
This erratum can be avoided by setting CPUACTLR2[22] to 1'b1,
which will disable linking multiple Non-Cacheable or Device
GRE loads to the same read request for the cache-line. This
might have a significant performance impact to Non-cacheable
and Device GRE read bandwidth for streaming scenarios.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: Iff224ef82bd1cb9aff8d6b11451e2ac1d048149f
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 3730893

Cortex-X925 erratum 3730893 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

PE executing a load instruction th

fix(cpus): workaround for Cortex-X925 erratum 3730893

Cortex-X925 erratum 3730893 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

PE executing a load instruction that accesses a memory
region which crosses a 4K boundary might cause a deadlock.
This erratum can be avoided by setting CPUACTLR_EL1[60:58]
to 3'b001, which has a small perf impact.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I0245183669255afb0d3ec71cafa058aa72129de0
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when swit

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when switching from low
to high EL, this erratum can be avoided by setting the CPUACTLR4[10]
to 1 and CPUACTLR4[11] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: Ieb5fe278821d85382af60be25e9546e65ba9a629
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

Merge "feat(lib): add mmio_setbits_64 inline function" into integration

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two or more STG
instructions that access the same cache line but different 32-bytes
might not write the MTE allocation tag to memory. This erratum can
be avoided by setting CPUACTLR5_EL1[14] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I8eb8bbdd6f99f69c8713400191ac66f55ffedc8b
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

feat(lib): add mmio_setbits_64 inline function

Add mmio 64 bits set utility function.

Change-Id: I35c54d1c0e981f6a68eb8fd1101947451dcf253f
Signed-off-by: Runyang Chen <runyang.chen@mediatek.com>

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaroun

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaround for C1-Nano erratum 3630925
fix(cpus): workaround for C1-Nano erratum 3616450
fix(cpus): workaround for C1-Nano erratum 3516455
fix(cpus): workaround for C1-Nano erratum 3437202
fix(cpus): workaround for C1-Nano erratum 3392149

show more ...

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workarou

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workaround for Neoverse V3 erratum 3864536
fix(cpus): workaround for Neoverse V3 erratum 3782181
fix(cpus): workaround for Neoverse V3 erratum 3734562
fix(cpus): workaround for Neoverse V3 erratum 3696307

show more ...

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_EL1[27] to
1, which disable write streaming for MTE stores when MTE
feature is enabled.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Ib5103483163a1f93cbb2df8c3b3fcfb2c6d487c6
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full
retention mode by setting both IMP_CPUPWRCTLR_EL1[9:7] and
IMP_CPUPWRCTLR_EL1[6:4] to 3'b000.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I61cdf21b50dfb534ce2a1e74c22b06bde9a7c0a7
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
str

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
streaming mode when Non-SME instruction abort.
Which can be avoided by restricts address generation
based on speculatively produced data for vector
load/stores accessing 4 vector registers in streaming SVE
mode. The workaround can have a minor impact on
performance in heavy streaming SVE workloads, depending
on the density of the affected instructions

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Id97fbfd1d76e9dc1a3488ce33e353c032c41e0f1
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, wh

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, which
can be avoided by seting IMP_CPUACTLR_EL1[26] to 1.
The workaround is expected to have negligible performance
and power impact.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: If6c12a7a26ccd67496909481a9683151d30d4339
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an I-cache
invalidation, which can be avoided by seting
IMP_CPUACTLR3_EL1[39] to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I530c75acf25ee57efaf7ff58ef4a43508fb6d52a
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - t

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - the `cas` will directly return the
state of the lock (held or not held). However, when using exclusives,
there's a third state - failed to hold the lock. This happens when the
store exclusive couldn't complete the write and bailed. The current
implementation will pigeonhole this state into a "not held" state which
loses this subtlety and can return with no one holding the lock.

This patch makes it so that the operation is retried until the core is
certain it either holds the lock or someone else does. This keeps the
nonblocking nature of the trylock call but may incur a delay until the
state of the lock settles.

Change-Id: I1a57de22557e13c22f6a6afdef4c28f679dbe7f2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code paths it has thus far largely been
written in assembly. However, assembly is extremely tedious to read and
modify. Similar to context management, it is desirable to have as much
as possible in C. C code is generally easier to follow and can enable
the compiler to do more optimisations on surrounding code.

Most exceptions that BL31 deals with are the synchronous exceptions and
those are processed within BL31. They already get prepared for EL3 entry
and after the initial dispatch end up in C. So the dispatch can also be
converted in C. Interrupt exceptions are very similar so are converted
too. Finally, asynchronous external aborts share some code with
synchronous external aborts and may end up being processed deeper in
BL31. So they can safely be prepared for EL3 entry too and converted to
C so that they can share code properly.

The IMP DEF exceptions are not part of this refactor as their speed may
be important. There is currently little that uses them, but they can be
converted to C too once their use expands and usage allows it.

This refactor allows to expand the responsibilities of
prepare_el3_entry(). Its role is already to prepare context for
executing within EL3 but with this patch EL3 execution is synonymous
with C runtime execution. So it's given the responsibility of saving
spsr and elr as well as putting the runtime stack in.

When a synchronous exception happens, the only possible paths are to
enter the C EL3 runtime, exiting via el3_exit(), or to panic. In the EL3
runtime case, we always need prepare_el3_entry() and the runtime stack,
whereas in the panic case, this doesn't matter as we will never return.
So hoist the prepare_el3_entry() call and the changing of the stacks as
early as possible and make the rest of the code agnostic of this.

This patch also gets rid of smc_prohibited. It is an optimisation by
skipping prepare_el3_entry() when a bad smc call happens. However, speed
doesn't matter in this case as this is an erroneous case.

Change-Id: I411af9d17ef4046a736b1f4f5f8fbc9c28e66106
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "fix(cpus): workaround for Neoverse-V2 erratum 3442699" into integration