fix(cpus): workaround for Cortex-X925 erratum 2933290

Cortex-X925 erratum 2933290 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

MTE Tag checking may not be performed if an access th

fix(cpus): workaround for Cortex-X925 erratum 2933290

Cortex-X925 erratum 2933290 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

MTE Tag checking may not be performed if an access that crosses
a 16-byte boundary encounters a poisoned Allocation tag. This
erratum can be avoided by setting CPUACTLR5_EL1[42] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I6d5c680a3d5156b3b17d59de79f8b650d56deff3
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when swit

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when switching from low
to high EL, this erratum can be avoided by setting the CPUACTLR4[10]
to 1 and CPUACTLR4[11] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: Ieb5fe278821d85382af60be25e9546e65ba9a629
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two or more STG
instructions that access the same cache line but different 32-bytes
might not write the MTE allocation tag to memory. This erratum can
be avoided by setting CPUACTLR5_EL1[14] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I8eb8bbdd6f99f69c8713400191ac66f55ffedc8b
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaroun

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaround for C1-Nano erratum 3630925
fix(cpus): workaround for C1-Nano erratum 3616450
fix(cpus): workaround for C1-Nano erratum 3516455
fix(cpus): workaround for C1-Nano erratum 3437202
fix(cpus): workaround for C1-Nano erratum 3392149

show more ...

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workarou

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workaround for Neoverse V3 erratum 3864536
fix(cpus): workaround for Neoverse V3 erratum 3782181
fix(cpus): workaround for Neoverse V3 erratum 3734562
fix(cpus): workaround for Neoverse V3 erratum 3696307

show more ...

fix(cpus): workaround for C1-Nano erratum 3754876

C1-Nano erratum 3754876 is a Cat B erratum that applies
to revisions r0p0, and r0p1, and is fixed in r0p2.

This errata can be avoided by executing

fix(cpus): workaround for C1-Nano erratum 3754876

C1-Nano erratum 3754876 is a Cat B erratum that applies
to revisions r0p0, and r0p1, and is fixed in r0p2.

This errata can be avoided by executing a TSB CSYNC
before executing a WFI instruction for power down.
Which prevents core deadlock during power down if
Trace Buffer Extension (TRBE) is enabled.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I3528f08c028b50be848b8d6113d370414261ad48
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_EL1[27] to
1, which disable write streaming for MTE stores when MTE
feature is enabled.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Ib5103483163a1f93cbb2df8c3b3fcfb2c6d487c6
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full
retention mode by setting both IMP_CPUPWRCTLR_EL1[9:7] and
IMP_CPUPWRCTLR_EL1[6:4] to 3'b000.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I61cdf21b50dfb534ce2a1e74c22b06bde9a7c0a7
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3616450

C1-Nano erratum 3616450 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might result in data corruption under
u

fix(cpus): workaround for C1-Nano erratum 3616450

C1-Nano erratum 3616450 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might result in data corruption under
uncommon micro-architectural conditions for SME, SIMD&FP,
or SVE load or store. This can be avoided by setting
IMP_CPUACTLR_EL1[29] to 1. This workaround will reduce
the effectiveness of internal clock gating, and can have
a small impact on power efficiency. During power testing
of sample silicon, Arm recommends not applying the
workaround.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I542be9a051a1f17e93c21bef725f7ede429555f9
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
str

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
streaming mode when Non-SME instruction abort.
Which can be avoided by restricts address generation
based on speculatively produced data for vector
load/stores accessing 4 vector registers in streaming SVE
mode. The workaround can have a minor impact on
performance in heavy streaming SVE workloads, depending
on the density of the affected instructions

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Id97fbfd1d76e9dc1a3488ce33e353c032c41e0f1
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, wh

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, which
can be avoided by seting IMP_CPUACTLR_EL1[26] to 1.
The workaround is expected to have negligible performance
and power impact.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: If6c12a7a26ccd67496909481a9683151d30d4339
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an I-cache
invalidation, which can be avoided by seting
IMP_CPUACTLR3_EL1[39] to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I530c75acf25ee57efaf7ff58ef4a43508fb6d52a
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - t

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - the `cas` will directly return the
state of the lock (held or not held). However, when using exclusives,
there's a third state - failed to hold the lock. This happens when the
store exclusive couldn't complete the write and bailed. The current
implementation will pigeonhole this state into a "not held" state which
loses this subtlety and can return with no one holding the lock.

This patch makes it so that the operation is retried until the core is
certain it either holds the lock or someone else does. This keeps the
nonblocking nature of the trylock call but may incur a delay until the
state of the lock settles.

Change-Id: I1a57de22557e13c22f6a6afdef4c28f679dbe7f2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

fix(locks): restore spin_trylock's ability to acquire a lock

The FEAT_STATE conversion of spinlocks (patch
38e580e6411c7a2eb2801a6aacb0a19bb9b1ac46) missed a branch between the
two `mov` instruction

fix(locks): restore spin_trylock's ability to acquire a lock

The FEAT_STATE conversion of spinlocks (patch
38e580e6411c7a2eb2801a6aacb0a19bb9b1ac46) missed a branch between the
two `mov` instructions and as a result the `mov rx, #1` is always
followed by `mov rx, #0` causing the spin_trylock() function to always
return failure.

Putting the branch back into the inline assembly is difficult without
extra C control logic that would add extra instructions. So instead
keep only the exclusives in assembly and convert all control logic to C.
This emits code that is functionally equivalent to the original with the
same number of instructions.

Change-Id: If97ea5fd46321ec18c8f92368355b1bc004a7cf0
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code paths it has thus far largely been
written in assembly. However, assembly is extremely tedious to read and
modify. Similar to context management, it is desirable to have as much
as possible in C. C code is generally easier to follow and can enable
the compiler to do more optimisations on surrounding code.

Most exceptions that BL31 deals with are the synchronous exceptions and
those are processed within BL31. They already get prepared for EL3 entry
and after the initial dispatch end up in C. So the dispatch can also be
converted in C. Interrupt exceptions are very similar so are converted
too. Finally, asynchronous external aborts share some code with
synchronous external aborts and may end up being processed deeper in
BL31. So they can safely be prepared for EL3 entry too and converted to
C so that they can share code properly.

The IMP DEF exceptions are not part of this refactor as their speed may
be important. There is currently little that uses them, but they can be
converted to C too once their use expands and usage allows it.

This refactor allows to expand the responsibilities of
prepare_el3_entry(). Its role is already to prepare context for
executing within EL3 but with this patch EL3 execution is synonymous
with C runtime execution. So it's given the responsibility of saving
spsr and elr as well as putting the runtime stack in.

When a synchronous exception happens, the only possible paths are to
enter the C EL3 runtime, exiting via el3_exit(), or to panic. In the EL3
runtime case, we always need prepare_el3_entry() and the runtime stack,
whereas in the panic case, this doesn't matter as we will never return.
So hoist the prepare_el3_entry() call and the changing of the stacks as
early as possible and make the rest of the code agnostic of this.

This patch also gets rid of smc_prohibited. It is an optimisation by
skipping prepare_el3_entry() when a bad smc call happens. However, speed
doesn't matter in this case as this is an erroneous case.

Change-Id: I411af9d17ef4046a736b1f4f5f8fbc9c28e66106
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Neoverse-N3 erratum 3456111
fix(cpus): workaround for Neoverse-N2 erratum 3324339
fix(cpus)

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Neoverse-N3 erratum 3456111
fix(cpus): workaround for Neoverse-N2 erratum 3324339
fix(cpus): workaround for Neoverse-N1 erratum 3324349

show more ...

Merge "fix(cpus): workaround for Neoverse-V2 erratum 3442699" into integration

fix(cpus): workaround for Neoverse-N3 erratum 3456111

Neoverse-N3 erratum 3456111 is a Cat B erratum that applies
to revisions r0p0 and r0p1 and is still open.

This errata can be avoided by adding

fix(cpus): workaround for Neoverse-N3 erratum 3456111

Neoverse-N3 erratum 3456111 is a Cat B erratum that applies
to revisions r0p0 and r0p1 and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3050973

Change-Id: I1685c2cacbe64ddf70501e8cce94b4fbf03f0ba0
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

Merge "fix(cpus): workaround for C1-Pro erratum 3619847" into integration

Merge changes from topic "xl/c1pro-errata" into integration

* changes:
fix(cpus): workaround for C1-Pro erratum 3686597
fix(cpus): workaround for C1-Pro erratum 3300099
fix(cpus): workaround f

Merge changes from topic "xl/c1pro-errata" into integration

* changes:
fix(cpus): workaround for C1-Pro erratum 3686597
fix(cpus): workaround for C1-Pro erratum 3300099
fix(cpus): workaround for C1-Pro erratum 3338470
fix(cpus): workaround for C1-Pro erratum 3362007
fix(cpus): workaround for C1-Pro erratum 3684268
fix(cpus): workaround for C1-Pro erratum 3694158
fix(cpus): workaround for C1-Pro erratum 3706576

show more ...

fix(cpus): workaround for Neoverse-N2 erratum 3324339

Neoverse-N2 erratum 3324339 is a Cat B erratum that applies
to revisions r0p0, r0p1, r0p2 and r0p3 and is still open.

This errata can be avoide

fix(cpus): workaround for Neoverse-N2 erratum 3324339

Neoverse-N2 erratum 3324339 is a Cat B erratum that applies
to revisions r0p0, r0p1, r0p2 and r0p3 and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-1982442

Change-Id: I6b023279816005cfa459bc6947f60b1a3c0f2380
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse-N1 erratum 3324349

Neoverse-N1 erratum 3324349 is a Cat B erratum that applies
to all revisions <= r4p1, and is still open.

This errata can be avoided by adding a

fix(cpus): workaround for Neoverse-N1 erratum 3324349

Neoverse-N1 erratum 3324349 is a Cat B erratum that applies
to all revisions <= r4p1, and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-885747

Change-Id: I1f142027ed73135d78c368be926072c2f73eab46
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.
As workaround, Set CPUACTLR_EL1[36] before enabling icache.

SDEN: https://developer.arm.com/documentation/SDEN-2332927/latest

Change-Id: I38edc6ba445223091c3933cbca35b56db491c926
Signed-off-by: Jaiprakash Singh <jaiprakashs@marvell.com>
Signed-off-by: Chandrakala Chavva <cchavva@cavium.com>
Reviewed-by: Chandrakala Chavva <cchavva@marvell.com>
Tested-by: Chandrakala Chavva <cchavva@marvell.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3619847

C1-Pro erratum 3619847 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_E

fix(cpus): workaround for C1-Pro erratum 3619847

C1-Pro erratum 3619847 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_EL1[42] to 1.
Only a minor performance drop is expected when mixing SME and
non-SME store instructions.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: Id92e7180df20d973e4e2d112c4f187a561a4d924
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUE

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUECTLR_EL1[57]
to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I59a5d9316bf66793eae5dac08102231d0e2640fb
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...