feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code

feat(el3-runtime): translate EL3 handled exceptions to C and always call prepare_el3_entry

Exception handling in BL31 is tricky business and to satisfy the varying
requirements of the different code paths it has thus far largely been
written in assembly. However, assembly is extremely tedious to read and
modify. Similar to context management, it is desirable to have as much
as possible in C. C code is generally easier to follow and can enable
the compiler to do more optimisations on surrounding code.

Most exceptions that BL31 deals with are the synchronous exceptions and
those are processed within BL31. They already get prepared for EL3 entry
and after the initial dispatch end up in C. So the dispatch can also be
converted in C. Interrupt exceptions are very similar so are converted
too. Finally, asynchronous external aborts share some code with
synchronous external aborts and may end up being processed deeper in
BL31. So they can safely be prepared for EL3 entry too and converted to
C so that they can share code properly.

The IMP DEF exceptions are not part of this refactor as their speed may
be important. There is currently little that uses them, but they can be
converted to C too once their use expands and usage allows it.

This refactor allows to expand the responsibilities of
prepare_el3_entry(). Its role is already to prepare context for
executing within EL3 but with this patch EL3 execution is synonymous
with C runtime execution. So it's given the responsibility of saving
spsr and elr as well as putting the runtime stack in.

When a synchronous exception happens, the only possible paths are to
enter the C EL3 runtime, exiting via el3_exit(), or to panic. In the EL3
runtime case, we always need prepare_el3_entry() and the runtime stack,
whereas in the panic case, this doesn't matter as we will never return.
So hoist the prepare_el3_entry() call and the changing of the stacks as
early as possible and make the rest of the code agnostic of this.

This patch also gets rid of smc_prohibited. It is an optimisation by
skipping prepare_el3_entry() when a bad smc call happens. However, speed
doesn't matter in this case as this is an erroneous case.

Change-Id: I411af9d17ef4046a736b1f4f5f8fbc9c28e66106
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(el3-runtime): factor out handler fetching code

handle_runtime_svc() is AArch32 only, but the part that fetches the
handler is not. Factor it out into its own function so it can be used
for

refactor(el3-runtime): factor out handler fetching code

handle_runtime_svc() is AArch32 only, but the part that fetches the
handler is not. Factor it out into its own function so it can be used
for AArch64.

Change-Id: I89813759814817390065540329af98168fd04a88
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Neoverse-N3 erratum 3456111
fix(cpus): workaround for Neoverse-N2 erratum 3324339
fix(cpus)

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Neoverse-N3 erratum 3456111
fix(cpus): workaround for Neoverse-N2 erratum 3324339
fix(cpus): workaround for Neoverse-N1 erratum 3324349

show more ...

Merge "fix(cpus): workaround for Neoverse-V2 erratum 3442699" into integration

fix(cpus): workaround for Neoverse-N3 erratum 3456111

Neoverse-N3 erratum 3456111 is a Cat B erratum that applies
to revisions r0p0 and r0p1 and is still open.

This errata can be avoided by adding

fix(cpus): workaround for Neoverse-N3 erratum 3456111

Neoverse-N3 erratum 3456111 is a Cat B erratum that applies
to revisions r0p0 and r0p1 and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3050973

Change-Id: I1685c2cacbe64ddf70501e8cce94b4fbf03f0ba0
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

Merge "fix(cpus): workaround for C1-Pro erratum 3619847" into integration

Merge changes from topic "xl/c1pro-errata" into integration

* changes:
fix(cpus): workaround for C1-Pro erratum 3686597
fix(cpus): workaround for C1-Pro erratum 3300099
fix(cpus): workaround f

Merge changes from topic "xl/c1pro-errata" into integration

* changes:
fix(cpus): workaround for C1-Pro erratum 3686597
fix(cpus): workaround for C1-Pro erratum 3300099
fix(cpus): workaround for C1-Pro erratum 3338470
fix(cpus): workaround for C1-Pro erratum 3362007
fix(cpus): workaround for C1-Pro erratum 3684268
fix(cpus): workaround for C1-Pro erratum 3694158
fix(cpus): workaround for C1-Pro erratum 3706576

show more ...

fix(cpus): workaround for Neoverse-N2 erratum 3324339

Neoverse-N2 erratum 3324339 is a Cat B erratum that applies
to revisions r0p0, r0p1, r0p2 and r0p3 and is still open.

This errata can be avoide

fix(cpus): workaround for Neoverse-N2 erratum 3324339

Neoverse-N2 erratum 3324339 is a Cat B erratum that applies
to revisions r0p0, r0p1, r0p2 and r0p3 and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-1982442

Change-Id: I6b023279816005cfa459bc6947f60b1a3c0f2380
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse-N1 erratum 3324349

Neoverse-N1 erratum 3324349 is a Cat B erratum that applies
to all revisions <= r4p1, and is still open.

This errata can be avoided by adding a

fix(cpus): workaround for Neoverse-N1 erratum 3324349

Neoverse-N1 erratum 3324349 is a Cat B erratum that applies
to all revisions <= r4p1, and is still open.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-885747

Change-Id: I1f142027ed73135d78c368be926072c2f73eab46
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.

fix(cpus): workaround for Neoverse-V2 erratum 3442699

Neoverse-V2 erratum 3442699 applies to r0p0, r0p1, and r0p2
and it is still open.

PE may execute incorrect instructions when icache is enabled.
As workaround, Set CPUACTLR_EL1[36] before enabling icache.

SDEN: https://developer.arm.com/documentation/SDEN-2332927/latest

Change-Id: I38edc6ba445223091c3933cbca35b56db491c926
Signed-off-by: Jaiprakash Singh <jaiprakashs@marvell.com>
Signed-off-by: Chandrakala Chavva <cchavva@cavium.com>
Reviewed-by: Chandrakala Chavva <cchavva@marvell.com>
Tested-by: Chandrakala Chavva <cchavva@marvell.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3619847

C1-Pro erratum 3619847 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_E

fix(cpus): workaround for C1-Pro erratum 3619847

C1-Pro erratum 3619847 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_EL1[42] to 1.
Only a minor performance drop is expected when mixing SME and
non-SME store instructions.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: Id92e7180df20d973e4e2d112c4f187a561a4d924
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUE

fix(cpus): workaround for C1-Pro erratum 3686597

C1-Pro erratum 3686597 is a Cat B erratum that applies
to revisions r0p0, r1p0 and is fixed in r1p1.

This erratum can be avoided by setting IMP_CPUECTLR_EL1[57]
to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I59a5d9316bf66793eae5dac08102231d0e2640fb
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3300099

C1-Pro erratum 3300099 is a Cat B erratum that applies
to revisions r0p0, r1p0, and is fixed in r1p1.

This is workaround for accessing ICH_VMCR_EL2.

fix(cpus): workaround for C1-Pro erratum 3300099

C1-Pro erratum 3300099 is a Cat B erratum that applies
to revisions r0p0, r1p0, and is fixed in r1p1.

This is workaround for accessing ICH_VMCR_EL2.
When ICH_VMCR_EL2.VBPR1 is written in Secure state (SCR_EL3.NS==0)
and then subsequently read in Non-secure state (SCR_EL3.NS==1), a
wrong value might be returned. The same issue exists in the opposite way.

Adding workaround in EL3 software that performs context save/restore
on a change of Security state to use a value of SCR_EL3.NS when
accessing ICH_VMCR_EL2 that reflects the Security state that owns the
data being saved or restored. For example, EL3 software should set
SCR_EL3.NS to 1 when saving or restoring the value ICH_VMCR_EL2 for
Non-secure(or Realm) state. EL3 software should clear
SCR_EL3.NS to 0 when saving or restoring the value ICH_VMCR_EL2 for
Secure state.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: If24d3230c4b4e87fcb831d446cf0d0c68c95ea18
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for Neoverse-V3 erratum 3312417

Neoverse-V3 erratum 3312417 is a Cat B erratum that applies
to revisions r0p0 and r0p1, and is fixed in r0p2.

This errata can be avoided by add

fix(cpus): workaround for Neoverse-V3 erratum 3312417

Neoverse-V3 erratum 3312417 is a Cat B erratum that applies
to revisions r0p0 and r0p1, and is fixed in r0p2.

This errata can be avoided by adding a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: I78a7682cbdf3dbc4c31fcca8cbd892350b998cf4
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3878291

Neoverse V3 erratum 3878291 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by

fix(cpus): workaround for Neoverse V3 erratum 3878291

Neoverse V3 erratum 3878291 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by setting CPUACTLR4_EL1[57]. Setting this
bit causes the PE to treat GPT invalidations as TLBI PAALL, thereby
invalidating all GPT entries. If the physical memory map does not use
addresses with bits 46 or 47 set, then no workaround is necessary.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: I0ebab877b6481a18bec963b95cf2f37c97d8de65
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3864536

Neoverse V3 erratum 3864536 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by

fix(cpus): workaround for Neoverse V3 erratum 3864536

Neoverse V3 erratum 3864536 is a Cat B erratum that applies to
revisions r0p0, r0p1 and r0p2, and is still open.

The erratum can be avoided by setting CPUACTLR2[22] to 1'b1 which will
disable linking multiple Non-Cacheable or Device GRE loads to the same
read request for the cache-line. This might have a significant
performance impact to Non-cacheable and Device GRE read bandwidth for
streaming scenarios.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: If4b20d941d628b92748b14d027b8127f74005eff
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3782181

Neoverse V3 erratum 3782181 is a Cat B erratum that applies to
revision r0p1 and is fixed in r0p2.

If the erratum condition occurs, then the co

fix(cpus): workaround for Neoverse V3 erratum 3782181

Neoverse V3 erratum 3782181 is a Cat B erratum that applies to
revision r0p1 and is fixed in r0p2.

If the erratum condition occurs, then the core will not leave the
FULL_RET power mode, which will cause the system to deadlock. The
FULL_RET power mode should not be enabled. This can be done by setting
both IMP_CPUPWRCTLR_EL1.WFE_RET_CTL and IMP_CPUPWRCTLR_EL1.WFI_RET_CTL
to 0b000 which is the default value.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: Icfa463cf4888bd48f16a218e7ad399528feca55e
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3734562

Neoverse V3 erratum 3734562 is a Cat B erratum that applies to
revisions r0p0 and r0p1, and is fixed in r0p2.

This erratum can be avoided throu

fix(cpus): workaround for Neoverse V3 erratum 3734562

Neoverse V3 erratum 3734562 is a Cat B erratum that applies to
revisions r0p0 and r0p1, and is fixed in r0p2.

This erratum can be avoided through the following write sequence to
several IMPLEMENTATION DEFINED registers, which will execute a PSB
instruction following the TSB CSYNC instruction. The code sequence
should be applied early in the boot sequence prior to executing a TSB
CSYNC instruction.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: Ib3c35c7e619e6a836c974b7016bb6a4d66da48d6
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for Neoverse V3 erratum 3696307

Neoverse V3 erratum 3696307 is a Cat B erratum that applies to
revisions r0p0 and r0p1, and is fixed in r0p2.

The erratum can be avoided by dis

fix(cpus): workaround for Neoverse V3 erratum 3696307

Neoverse V3 erratum 3696307 is a Cat B erratum that applies to
revisions r0p0 and r0p1, and is fixed in r0p2.

The erratum can be avoided by disabling the affected prefetcher by
setting CPUACTLR6_EL1[41].

SDEN documentation:
https://developer.arm.com/documentation/SDEN-2891958

Change-Id: If274749621549356e41485d0bf09682281df3a9b
Signed-off-by: John Powell <john.powell@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3338470

C1-Pro erratum 3338470 is a Cat B erratum that applies
to revision r0p0, and is fixed in r1p0.

This errata can be avoid by having a speculation barr

fix(cpus): workaround for C1-Pro erratum 3338470

C1-Pro erratum 3338470 is a Cat B erratum that applies
to revision r0p0, and is fixed in r1p0.

This errata can be avoid by having a speculation barrier
instruction following writes to the SSBS register to
ensure the new value of PSTATE.SSBS affects the subsequent
instructions in the execution stream under speculation.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I86e2b8f70ceb468c75c0386a790641d51eeea9cb
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3362007

C1-Pro erratum 3362007 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_E

fix(cpus): workaround for C1-Pro erratum 3362007

C1-Pro erratum 3362007 is a Cat B erratum that applies to
CPU revision r0p0 and is fixed in r1p0.

This erratum can be avoided by setting CPUACTLR2_EL1[27] to 1.
Only a minor increase in power consumption is expected.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I529e9812bddffe927c986f9b5ee135f4866aa455
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3684268

C1-Pro erratum 3684268 is a Cat B erratum that applies
to revisions r0p0, r1p0 and it is fixed in r1p1.

The erratum is avoided by disabling the affe

fix(cpus): workaround for C1-Pro erratum 3684268

C1-Pro erratum 3684268 is a Cat B erratum that applies
to revisions r0p0, r1p0 and it is fixed in r1p1.

The erratum is avoided by disabling the affected prefetcher,
which is done by setting CPUECTLR2_EL1[49] to 1.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I7929e931572471370b1a899d412b11f1c4d206c8
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3694158

C1-Pro erratum 3694158 is a Cat B erratum that applies
to revisions r0p0, r1p0 and r1p1, it is fixed in r1p2.

This erratum can be avoided by inserti

fix(cpus): workaround for C1-Pro erratum 3694158

C1-Pro erratum 3694158 is a Cat B erratum that applies
to revisions r0p0, r1p0 and r1p1, it is fixed in r1p2.

This erratum can be avoided by inserting a DMB LD after
each DSB ST instruction with a CPU implementation specific
patch sequence.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: I38f0fb6565110c579ab16b76e0f4ca601fa1b912
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

fix(cpus): workaround for C1-Pro erratum 3706576

C1-Pro erratum 3706576 is a Cat B erratum that applies to
CPU revisions r0p0 and r1p0, and is fixed in r1p1.

This erratum might cause data corruptio

fix(cpus): workaround for C1-Pro erratum 3706576

C1-Pro erratum 3706576 is a Cat B erratum that applies to
CPU revisions r0p0 and r1p0, and is fixed in r1p1.

This erratum might cause data corruption when Memory read
effect crossing a 64B boundary, which can be avoided by
setting CPUACTLR2_EL1[37] to 1. Setting this bit is expected
to have a negligible performance impact.

SDEN documentation:
https://developer.arm.com/documentation/SDEN-3273080/1300/?lang=en

Change-Id: Ie427e56c682065bdf82da9b11e71da6383db4e73
Signed-off-by: Xialin Liu <xialin.liu@arm.com>

show more ...

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Cortex-A720AE erratum 3456103
fix(cpus): workaround for Cortex-A720 erratum 3456091
fix(cpu

Merge changes from topic "ssbs_errata_catchup" into integration

* changes:
fix(cpus): workaround for Cortex-A720AE erratum 3456103
fix(cpus): workaround for Cortex-A720 erratum 3456091
fix(cpus): workaround for Cortex-A715 erratum 3456084
fix(cpus): workaround for Cortex-X2 erratum 3324338
fix(cpus): workaround for Cortex-A710 erratum 3324338

show more ...