Merge changes I411af9d1,I89813759 into integration

* changes:
feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry
refactor(el3-runtime): factor out handler

Merge changes I411af9d1,I89813759 into integration

* changes:
feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry
refactor(el3-runtime): factor out handler fetching code

show more ...

Merge changes I20c97011,Ia1facabb into integration

* changes:
fix(rk3576): shorten names to fit into the allocated space
fix(debugfs): allocate enough space to fit all names

Merge changes from topic "xl/x925-errata" into integration

* changes:
fix(cpus): workaround for Cortex-X925 erratum 3865185
fix(cpus): workaround for Cortex-X925 erratum 3730893
fix(cpus): wor

Merge changes from topic "xl/x925-errata" into integration

* changes:
fix(cpus): workaround for Cortex-X925 erratum 3865185
fix(cpus): workaround for Cortex-X925 erratum 3730893
fix(cpus): workaround for Cortex-X925 erratum 3692980
fix(cpus): workaround for Cortex-X925 erratum 3324334
fix(cpus): workaround for Cortex-X925 erratum 2933290
fix(cpus): workaround for Cortex-X925 erratum 2922378
fix(cpus): workaround for Cortex-X925 erratum 2921199

show more ...

fix(cpus): workaround for Cortex-X925 erratum 3865185

Cortex-X925 erratum 3865185 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

Load issued to Non-Cacheable or De

fix(cpus): workaround for Cortex-X925 erratum 3865185

Cortex-X925 erratum 3865185 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

Load issued to Non-Cacheable or Device GRE memory can
read stale data brought in by an earlier load to the
same cache-line thereby violating ordering requirements.
This erratum can be avoided by setting CPUACTLR2[22] to 1'b1,
which will disable linking multiple Non-Cacheable or Device
GRE loads to the same read request for the cache-line. This
might have a significant performance impact to Non-cacheable
and Device GRE read bandwidth for streaming scenarios.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: Iff224ef82bd1cb9aff8d6b11451e2ac1d048149f
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 3730893

Cortex-X925 erratum 3730893 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

PE executing a load instruction th

fix(cpus): workaround for Cortex-X925 erratum 3730893

Cortex-X925 erratum 3730893 is a Cat B erratum that
applies to revisions r0p0 and r0p1, it is fixed in r0p2.

PE executing a load instruction that accesses a memory
region which crosses a 4K boundary might cause a deadlock.
This erratum can be avoided by setting CPUACTLR_EL1[60:58]
to 3'b001, which has a small perf impact.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I0245183669255afb0d3ec71cafa058aa72129de0
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when swit

fix(cpus): workaround for Cortex-X925 erratum 2922378

Cortex-X925 erratum 2922378 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Branch prediction history is not suppressed when switching from low
to high EL, this erratum can be avoided by setting the CPUACTLR4[10]
to 1 and CPUACTLR4[11] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: Ieb5fe278821d85382af60be25e9546e65ba9a629
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

Merge "feat(lib): add mmio_setbits_64 inline function" into integration

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two

fix(cpus): workaround for Cortex-X925 erratum 2921199

Cortex-X925 erratum 2921199 is a Cat B erratum that
applies to r0p0 and is fixed in r0p1.

Under certain rare microarchitectural conditions, two or more STG
instructions that access the same cache line but different 32-bytes
might not write the MTE allocation tag to memory. This erratum can
be avoided by setting CPUACTLR5_EL1[14] to 1.

SDEN documentation:
https://developer.arm.com/documentation/109180/latest/

Change-Id: I8eb8bbdd6f99f69c8713400191ac66f55ffedc8b
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

feat(lib): add mmio_setbits_64 inline function

Add mmio 64 bits set utility function.

Change-Id: I35c54d1c0e981f6a68eb8fd1101947451dcf253f
Signed-off-by: Runyang Chen <runyang.chen@mediatek.com>

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaroun

Merge changes from topic "xl/c1nano-errata" into integration

* changes:
fix(cpus): workaround for C1-Nano erratum 3754876
fix(cpus): workaround for C1-Nano erratum 3419531
fix(cpus): workaround for C1-Nano erratum 3630925
fix(cpus): workaround for C1-Nano erratum 3616450
fix(cpus): workaround for C1-Nano erratum 3516455
fix(cpus): workaround for C1-Nano erratum 3437202
fix(cpus): workaround for C1-Nano erratum 3392149

show more ...

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workarou

Merge changes from topic "v3_errata" into integration

* changes:
fix(cpus): workaround for Neoverse-V3 erratum 3312417
fix(cpus): workaround for Neoverse V3 erratum 3878291
fix(cpus): workaround for Neoverse V3 erratum 3864536
fix(cpus): workaround for Neoverse V3 erratum 3782181
fix(cpus): workaround for Neoverse V3 erratum 3734562
fix(cpus): workaround for Neoverse V3 erratum 3696307

show more ...

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_

fix(cpus): workaround for C1-Nano erratum 3419531

C1-Nano erratum 3419531 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by setting IMP_CPUACTLR_EL1[27] to
1, which disable write streaming for MTE stores when MTE
feature is enabled.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Ib5103483163a1f93cbb2df8c3b3fcfb2c6d487c6
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full

fix(cpus): workaround for C1-Nano erratum 3630925

C1-Nano erratum 3630925 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata can be avoided by disable entering full
retention mode by setting both IMP_CPUPWRCTLR_EL1[9:7] and
IMP_CPUPWRCTLR_EL1[6:4] to 3'b000.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I61cdf21b50dfb534ce2a1e74c22b06bde9a7c0a7
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
str

fix(cpus): workaround for C1-Nano erratum 3516455

C1-Nano erratum 3516455 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

This errata might cause the core to deadlock in
streaming mode when Non-SME instruction abort.
Which can be avoided by restricts address generation
based on speculatively produced data for vector
load/stores accessing 4 vector registers in streaming SVE
mode. The workaround can have a minor impact on
performance in heavy streaming SVE workloads, depending
on the density of the affected instructions

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: Id97fbfd1d76e9dc1a3488ce33e353c032c41e0f1
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, wh

fix(cpus): workaround for C1-Nano erratum 3437202

C1-Nano erratum 3437202 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might might lead to data corruption, which
can be avoided by seting IMP_CPUACTLR_EL1[26] to 1.
The workaround is expected to have negligible performance
and power impact.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: If6c12a7a26ccd67496909481a9683151d30d4339
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an

fix(cpus): workaround for C1-Nano erratum 3392149

C1-Nano erratum 3392149 is a Cat B erratum that applies
to revision r0p0, and is fixed in r0p1.

The erratum might cause deadlock when receiving an I-cache
invalidation, which can be avoided by seting
IMP_CPUACTLR3_EL1[39] to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273788/0800/

Change-Id: I530c75acf25ee57efaf7ff58ef4a43508fb6d52a
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - t

feat(locks): make spin_trylock with exclusives spin until it knows the state of the lock

spin_trylock() is meant to be a non-blocking equivalent of spin_lock().
When we have atomics this is easy - the `cas` will directly return the
state of the lock (held or not held). However, when using exclusives,
there's a third state - failed to hold the lock. This happens when the
store exclusive couldn't complete the write and bailed. The current
implementation will pigeonhole this state into a "not held" state which
loses this subtlety and can return with no one holding the lock.

This patch makes it so that the operation is retried until the core is
certain it either holds the lock or someone else does. This keeps the
nonblocking nature of the trylock call but may incur a delay until the
state of the lock settles.

Change-Id: I1a57de22557e13c22f6a6afdef4c28f679dbe7f2
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code paths it has thus far largely been
written in assembly. However, assembly is extremely tedious to read and
modify. Similar to context management, it is desirable to have as much
as possible in C. C code is generally easier to follow and can enable
the compiler to do more optimisations on surrounding code.

Most exceptions that BL31 deals with are the synchronous exceptions and
those are processed within BL31. They already get prepared for EL3 entry
and after the initial dispatch end up in C. So the dispatch can also be
converted in C. Interrupt exceptions are very similar so are converted
too. Finally, asynchronous external aborts share some code with
synchronous external aborts and may end up being processed deeper in
BL31. So they can safely be prepared for EL3 entry too and converted to
C so that they can share code properly.

The IMP DEF exceptions are not part of this refactor as their speed may
be important. There is currently little that uses them, but they can be
converted to C too once their use expands and usage allows it.

This refactor allows to expand the responsibilities of
prepare_el3_entry(). Its role is already to prepare context for
executing within EL3 but with this patch EL3 execution is synonymous
with C runtime execution. So it's given the responsibility of saving
spsr and elr as well as putting the runtime stack in.

When a synchronous exception happens, the only possible paths are to
enter the C EL3 runtime, exiting via el3_exit(), or to panic. In the EL3
runtime case, we always need prepare_el3_entry() and the runtime stack,
whereas in the panic case, this doesn't matter as we will never return.
So hoist the prepare_el3_entry() call and the changing of the stacks as
early as possible and make the rest of the code agnostic of this.

This patch also gets rid of smc_prohibited. It is an optimisation by
skipping prepare_el3_entry() when a bad smc call happens. However, speed
doesn't matter in this case as this is an erroneous case.

Change-Id: I411af9d17ef4046a736b1f4f5f8fbc9c28e66106
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "fix(cpus): workaround for Neoverse-V2 erratum 3442699" into integration

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.
As workaround, Set CPUACTLR_EL1[36] before enabling icache.

SDEN: https://developer.arm.com/documentation/SDEN-2332927/latest

Change-Id: I38edc6ba445223091c3933cbca35b56db491c926
Signed-off-by: Jaiprakash Singh <jaiprakashs@marvell.com>
Signed-off-by: Chandrakala Chavva <cchavva@cavium.com>
Reviewed-by: Chandrakala Chavva <cchavva@marvell.com>
Tested-by: Chandrakala Chavva <cchavva@marvell.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUE

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUECTLR_EL1[57]
to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I59a5d9316bf66793eae5dac08102231d0e2640fb
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3300099

C1-Pro erratum 3300099 is a Cat B erratum that applies
to revisions r0p0, r1p0, and is fixed in r1p1.

This is workaround for accessing ICH_VMCR_EL2.

fix(cpus): workaround for C1-Pro erratum 3300099

C1-Pro erratum 3300099 is a Cat B erratum that applies
to revisions r0p0, r1p0, and is fixed in r1p1.

This is workaround for accessing ICH_VMCR_EL2.
When ICH_VMCR_EL2.VBPR1 is written in Secure state (SCR_EL3.NS==0)
and then subsequently read in Non-secure state (SCR_EL3.NS==1), a
wrong value might be returned. The same issue exists in the opposite way.

Adding workaround in EL3 software that performs context save/restore
on a change of Security state to use a value of SCR_EL3.NS when
accessing ICH_VMCR_EL2 that reflects the Security state that owns the
data being saved or restored. For example, EL3 software should set
SCR_EL3.NS to 1 when saving or restoring the value ICH_VMCR_EL2 for
Non-secure(or Realm) state. EL3 software should clear
SCR_EL3.NS to 0 when saving or restoring the value ICH_VMCR_EL2 for
Secure state.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: If24d3230c4b4e87fcb831d446cf0d0c68c95ea18
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3878291

Neoverse V3 erratum 3878291 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by

fix(cpus): workaround for Neoverse V3 erratum 3878291

Neoverse V3 erratum 3878291 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by setting CPUACTLR4_EL1[57]. Setting this
bit causes the PE to treat GPT invalidations as TLBI PAALL, thereby
invalidating all GPT entries. If the physical memory map does not use
addresses with bits 46 or 47 set, then no workaround is necessary.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: I0ebab877b6481a18bec963b95cf2f37c97d8de65
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3864536

Neoverse V3 erratum 3864536 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by

fix(cpus): workaround for Neoverse V3 erratum 3864536

Neoverse V3 erratum 3864536 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by setting CPUACTLR2[22] to 1'b1 which will
disable linking multiple Non-Cacheable or Device GRE loads to the same
read request for the cache-line. This might have a significant
performance impact to Non-cacheable and Device GRE read bandwidth for
streaming scenarios.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: If4b20d941d628b92748b14d027b8127f74005eff
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3782181

Neoverse V3 erratum 3782181 is a Cat B erratum that applies to
revision r0p1 and is fixed in r0p2.

If the erratum condition occurs, then the co

fix(cpus): workaround for Neoverse V3 erratum 3782181

Neoverse V3 erratum 3782181 is a Cat B erratum that applies to
revision r0p1 and is fixed in r0p2.

If the erratum condition occurs, then the core will not leave the
FULL_RET power mode, which will cause the system to deadlock. The
FULL_RET power mode should not be enabled. This can be done by setting
both IMP_CPUPWRCTLR_EL1.WFE_RET_CTL and IMP_CPUPWRCTLR_EL1.WFI_RET_CTL
to 0b000 which is the default value.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: Icfa463cf4888bd48f16a218e7ad399528feca55e
Signed-off-by: John Powell <john.powell@arm.com>

show more ...