Merge changes from topic "NUMA_AWARE_PER_CPU" into integration

* changes:
docs(maintainers): add per-cpu framework into maintainers.rst
feat(per-cpu): add documentation for per-cpu framework
f

Merge changes from topic "NUMA_AWARE_PER_CPU" into integration

* changes:
docs(maintainers): add per-cpu framework into maintainers.rst
feat(per-cpu): add documentation for per-cpu framework
feat(rdv3): enable numa aware per-cpu for RD-V3-Cfg2
feat(per-cpu): migrate amu_ctx to per-cpu framework
feat(per-cpu): migrate spm_core_context to per-cpu framework
feat(per-cpu): migrate psci_ns_context to per-cpu framework
feat(per-cpu): migrate psci_cpu_pd_nodes to per-cpu framework
feat(per-cpu): migrate rmm_context to per-cpu framework
feat(per-cpu): integrate per-cpu framework into BL31/BL32
feat(per-cpu): introduce framework accessors/definers
feat(per-cpu): introduce linker changes for NUMA aware per-cpu framework
docs(changelog): add scope for per-cpu framework

show more ...

feat(per-cpu): migrate psci_cpu_pd_nodes to per-cpu framework

migrate psci_cpu_pd_nodes object to the NUMA-aware per-cpu
framework to optimize memory access and to efficiently
utilize memory.

Signe

feat(per-cpu): migrate psci_cpu_pd_nodes to per-cpu framework

migrate psci_cpu_pd_nodes object to the NUMA-aware per-cpu
framework to optimize memory access and to efficiently
utilize memory.

Signed-off-by: Sammit Joshi <sammit.joshi@arm.com>
Signed-off-by: Rohit Mathew <rohit.mathew@arm.com>
Change-Id: Idec3e3b74ecf03b420b339a183be2b9e00f8a78f

show more ...

Merge changes from topic "bk/pabandon_cleanup" into integration

* changes:
feat(cpus): add pabandon support to the Alto cpu
feat(psci): optimise clock init on a pabandon
feat(psci): check that

Merge changes from topic "bk/pabandon_cleanup" into integration

* changes:
feat(cpus): add pabandon support to the Alto cpu
feat(psci): optimise clock init on a pabandon
feat(psci): check that CPUs handled a pabandon
feat(psci): make pabandon support generic
refactor(psci): unify coherency exit between AArch64 and AArch32
refactor(psci): absorb psci_power_down_wfi() into common code
refactor(platforms): remove usage of psci_power_down_wfi
fix(cm): disable SPE/TRBE correctly

show more ...

feat(psci): make pabandon support generic

Support for aborted powerdowns does not require much dedicated code.
Rather, it is largely a matter of orchestrating things to happen in the
right order.

T

feat(psci): make pabandon support generic

Support for aborted powerdowns does not require much dedicated code.
Rather, it is largely a matter of orchestrating things to happen in the
right order.

The only exception to this are older secure world dispatchers, which
assume that a CPU_SUSPEND call will be terminal and therefore can
clobber context. This was patched over in common code and hidden behind
a flag. This patch moves this to the dispatchers themselves.

Dispatchers that don't register svc_suspend{_finish} are unaffected.
Those that do must save the NS context before clobbering it and
restoring in only in case of a pabandon. Due to this operation being
non-trivial, this patch makes the assumption that these dispatchers will
only be present on hardware that does not support pabandon and therefore
does not add any contexting for them. In case this assumption ever
changes, asserts are added that should alert us of this change.

Change-Id: I94a907515b782b4d2136c0d274246cfe1d567c0e
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(psci): unify coherency exit between AArch64 and AArch32

The procedure is fairly simple: if we have hardware assisted coherency,
call into the cpu driver and let it do its thing. If we don't

refactor(psci): unify coherency exit between AArch64 and AArch32

The procedure is fairly simple: if we have hardware assisted coherency,
call into the cpu driver and let it do its thing. If we don't, then we
must turn data caches off, handle the confusion that causes with the
stack, and call into the cpu driver which will flush the caches that
need flushing.

On AArch32 the above happens in common code. On AArch64, however, the
turning off of the caches happens in the cpu driver. Since we're dealing
with the stack, we must exercise control over it and implement this in
assembly. But as the two implementations are nominally different (in the
ordering of operations), the part that is in assembly is quite large as
jumping back to C to handle the difference might involve the stack.

Presumably, the AArch difference was introduced in order to cater for a
possible implementation where turning off the caches requires an IMP DEF
sequence. Well, Arm no longer makes cores without hardware assisted
coherency, so this eventually is not possible.

So take this part out of the cpu driver and put it into common code,
just like in AArch32. With this, there is no longer a need call
prepare_cpu_pwr_dwn() in a different order either - we can delay it a
bit to happen after the stack management. So the two AArch-s flows
become identical. We can convert prepare_cpu_pwr_dwn() to C and leave
psci_do_pwrdown_cache_maintenance() only to exercise control over stack.

Change-Id: Ie4759ebe20bb74b60533c6a47dbc2b101875900f
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "feat(psci): remove cpu context init by index" into integration

feat(psci): remove cpu context init by index

Currently, the calling core (meaning the core which received the call to
CPU_ON or the powerdown path of CPU_SUSPEND on the same core) is in
charge of in

feat(psci): remove cpu context init by index

Currently, the calling core (meaning the core which received the call to
CPU_ON or the powerdown path of CPU_SUSPEND on the same core) is in
charge of initialising the context for the waking core (the warmboot
entrypoint for both). This is convenient because the calling core can
write the context while in coherency and the waking core will only need
the context after its entered coherency. This avoids any cache
maintenance and makes communication simple.

However, this has 3 main problems:
a) asymmetric feature support is problematic - the calling core has no
way of knowing the feature set of the waking core. If the two
diverge, the architectural feature discovery via ID registers breaks
down. We've thus far "fixed" this on a case by case basis which
doesn't scale and introduces redundancy.

b) powerdown abandon (pabandon) introduces a contradiction - the calling
core has to initialise the context for when the core wakes up, but
should the core not powerdown it needs its old context intact. The only
way to work around this is by keeping two copies of context which
incurs a runtime and memory overhead.

c) cm_prepare_el3_exit[_ns]() doesn't have access to the entrypoint but needs
it to make initialisation decisions. We can infer some of this from
registers that have already been written but this is awkwardly
limiting for what we can do. This also necessitates the split from
the context initialisation.

We can solve all three by a making a core be in full ownership of its
own context. The calling core then only writes entrypoint information
and nothing else. The waking core then initialises its own context as it
sees fit with full knowledge of the whole picture.

The only tricky bit is cache coherency - the waking core has to be able
to coherently observe its new entrypoint. Calling cores will write to
the shared region with coherent caches on. If we make sure to read the
context only after the waking core has entered coherency, then we can
avoid cache operations and let hardware handle everything.

We can skip the spsr check for FEAT_TCR2 as it doesn't make a
difference. We can also skip enabling it twice from generic code.

Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I86e7fe8b698191fc3b469e5ced1fd010f8754b0e

show more ...

Merge changes I765a7fa0,Ic33f0b6d,I8d1a88c7,I381f96be,I698fa849, ... into integration

* changes:
fix(cpus): clear CPUPWRCTLR_EL1.CORE_PWRDN_EN_BIT on reset
chore(docs): drop the "wfi" from `pwr_

Merge changes I765a7fa0,Ic33f0b6d,I8d1a88c7,I381f96be,I698fa849, ... into integration

* changes:
fix(cpus): clear CPUPWRCTLR_EL1.CORE_PWRDN_EN_BIT on reset
chore(docs): drop the "wfi" from `pwr_domain_pwr_down_wfi`
chore(psci): drop skip_wfi variable
feat(arm): convert arm platforms to expect a wakeup
fix(cpus): avoid SME related loss of context on powerdown
feat(psci): allow cores to wake up from powerdown
refactor: panic after calling psci_power_down_wfi()
refactor(cpus): undo errata mitigations
feat(cpus): add sysreg_bit_toggle

show more ...

feat(psci): allow cores to wake up from powerdown

The simplistic view of a core's powerdown sequence is that power is
atomically cut upon calling `wfi`. However, it turns out that it has
lots to do

feat(psci): allow cores to wake up from powerdown

The simplistic view of a core's powerdown sequence is that power is
atomically cut upon calling `wfi`. However, it turns out that it has
lots to do - it has to talk to the interconnect to exit coherency, clean
caches, check for RAS errors, etc. These take significant amounts of
time and are certainly not atomic. As such there is a significant window
of opportunity for external events to happen. Many of these steps are
not destructive to context, so theoretically, the core can just "give
up" half way (or roll certain actions back) and carry on running. The
point in this sequence after which roll back is not possible is called
the point of no return.

One of these actions is the checking for RAS errors. It is possible for
one to happen during this lengthy sequence, or at least remain
undiscovered until that point. If the core were to continue powerdown
when that happens, there would be no (easy) way to inform anyone about
it. Rejecting the powerdown and letting software handle the error is the
best way to implement this.

Arm cores since at least the a510 have included this exact feature. So
far it hasn't been deemed necessary to account for it in firmware due to
the low likelihood of this happening. However, events like GIC wakeup
requests are much more probable. Older cores will powerdown and
immediately power back up when this happens. Travis and Gelas include a
feature similar to the RAS case above, called powerdown abandon. The
idea is that this will improve the latency to service the interrupt by
saving on work which the core and software need to do.

So far firmware has relied on the `wfi` being the point of no return and
if it doesn't explicitly detect a pending interrupt quite early on, it
will embark onto a sequence that it expects to end with shutdown. To
accommodate for it not being a point of no return, we must undo all of
the system management we did, just like in the warm boot entrypoint.

To achieve that, the pwr_domain_pwr_down_wfi hook must not be terminal.
Most recent platforms do some platform management and finish on the
standard `wfi`, followed by a panic or an endless loop as this is
expected to not return. To make this generic, any platform that wishes
to support wakeups must instead let common code call
`psci_power_down_wfi()` right after. Besides wakeups, this lets common
code handle powerdown errata better as well.

Then, the CPU_OFF case is simple - PSCI does not allow it to return. So
the best that can be done is to attempt the `wfi` a few times (the
choice of 32 is arbitrary) in the hope that the wakeup is transient. If
it isn't, the only choice is to panic, as the system is likely to be in
a bad state, eg. interrupts weren't routed away. The same applies for
SYSTEM_OFF, SYSTEM_RESET, and SYSTEM_RESET2. There the panic won't
matter as the system is going offline one way or another. The RAS case
will be considered in a separate patch.

Now, the CPU_SUSPEND case is more involved. First, to powerdown it must
wipe its context as it is not written on warm boot. But it cannot be
overwritten in case of a wakeup. To avoid the catch 22, save a copy that
will only be used if powerdown fails. That is about 500 bytes on the
stack so it hopefully doesn't tip anyone over any limits. In future that
can be avoided by having a core manage its own context.

Second, when the core wakes up, it must undo anything it did to prepare
for poweroff, which for the cores we care about, is writing
CPUPWRCTLR_EL1.CORE_PWRDN_EN. The least intrusive for the cpu library
way of doing this is to simply call the power off hook again and have
the hook toggle the bit. If in the future there need to be more complex
sequences, their direction can be advised on the value of this bit.

Third, do the actual "resume". Most of the logic is already there for
the retention suspend, so that only needs a small touch up to apply to
the powerdown case as well. The missing bit is the powerdown specific
state management. Luckily, the warmboot entrypoint does exactly that
already too, so steal that and we're done.

All of this is hidden behind a FEAT_PABANDON flag since it has a large
memory and runtime cost that we don't want to burden non pabandon cores
with.

Finally, do some function renaming to better reflect their purpose and
make names a little bit more consistent.

Change-Id: I2405b59300c2e24ce02e266f91b7c51474c1145f
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge changes I137f69be,Ia2e7168f,I0e569d12,I614272ec,Ib68293f2 into integration

* changes:
perf(psci): pass my_core_pos around instead of calling it repeatedly
refactor(psci): move timestamp co

Merge changes I137f69be,Ia2e7168f,I0e569d12,I614272ec,Ib68293f2 into integration

* changes:
perf(psci): pass my_core_pos around instead of calling it repeatedly
refactor(psci): move timestamp collection to psci_pwrdown_cpu
refactor(psci): factor common code out of the standby finisher
refactor(psci): don't use PSCI_INVALID_PWR_LVL to signal OFF state
docs(psci): drop outdated cache maintenance comment

show more ...

perf(psci): pass my_core_pos around instead of calling it repeatedly

On some platforms plat_my_core_pos is a nontrivial function that takes a
bit of time and the compiler really doesn't like to inli

perf(psci): pass my_core_pos around instead of calling it repeatedly

On some platforms plat_my_core_pos is a nontrivial function that takes a
bit of time and the compiler really doesn't like to inline. In the PSCI
library, at least, we have no need to keep repeatedly calling it and we
can instead pass it around as an argument. This saves on a lot of
redundant calls, speeding the library up a bit.

Change-Id: I137f69bea80d7cac90d7a20ffe98e1ba8d77246f
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

refactor(psci): factor common code out of the standby finisher

psci_suspend_to_standby_finisher and psci_cpu_suspend_finish do mostly
the same stuff, besides the system management associated with th

refactor(psci): factor common code out of the standby finisher

psci_suspend_to_standby_finisher and psci_cpu_suspend_finish do mostly
the same stuff, besides the system management associated with their
respective wakeup paths. So bring the wake from standby path in line
with the wake from reset path - caller acquires locks and manages
context. This way both behave in vaguely the same way. We can also bring
their names in line so it's more apparent how they are different.

This is in preparation for cores waking from sleep, coming in another
patch. No functional change is expected.

Change-Id: I0e569d12f65d231606080faa0149d22efddc386d
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>

show more ...

Merge "fix(smccc): ensure that mpidr passed through SMC is valid" into integration

fix(smccc): ensure that mpidr passed through SMC is valid

There are various SMC calls which pass mpidr as an argument which is
currently tested at random places in SMC call path.
To make the mpidr v

fix(smccc): ensure that mpidr passed through SMC is valid

There are various SMC calls which pass mpidr as an argument which is
currently tested at random places in SMC call path.
To make the mpidr validation check consistent across SMC calls, do
this check as part of SMC argument validation.

This patch introduce a helper function is_valid_mpidr() to validate
mpidr and call it as part of validating SMC arguments at starting of
SMC handlers (which expect mpidr as an argument).

Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I11ea50e22caf17896cf4b2059b87029b2ba136b1

show more ...

Merge changes from topic "psci-osi" into integration

* changes:
fix(psci): add optional pwr_domain_validate_suspend to plat_psci_ops_t
fix(sc7280): update pwr_domain_suspend
fix(fvp): update p

Merge changes from topic "psci-osi" into integration

* changes:
fix(psci): add optional pwr_domain_validate_suspend to plat_psci_ops_t
fix(sc7280): update pwr_domain_suspend
fix(fvp): update pwr_domain_suspend

show more ...

fix(psci): add optional pwr_domain_validate_suspend to plat_psci_ops_t

This patch adds a new optional member `pwr_domain_validate_suspend` to
the `plat_psci_ops_t` structure that allows a platform t

fix(psci): add optional pwr_domain_validate_suspend to plat_psci_ops_t

This patch adds a new optional member `pwr_domain_validate_suspend` to
the `plat_psci_ops_t` structure that allows a platform to optionally
perform platform specific validations in OS-initiated mode. This is
conditionally compiled into the build depending on the value of the
`PSCI_OS_INIT_MODE` build option.

In https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/17682,
the return type of the `pwr_domain_suspend` handler was updated from
`void` to `int` to allow a platform to optionally perform platform
specific validations in OS-initiated mode. However, when an error code
other than `PSCI_E_SUCCESS` is returned, the current exit path does not
undo the operations in `psci_suspend_to_pwrdown_start`, and as a result,
the system ends up in an unexpected state.

The fix in this patch prevents the need to undo the operations in
`psci_suspend_to_pwrdown_start`, by allowing the platform to first
perform any necessary platform specific validations before the PSCI
generic code proceeds to the point of no return where the CPU_SUSPEND
request is expected to complete successfully.

Change-Id: I05d92c7ea3f5364da09af630d44d78252185db20
Signed-off-by: Wing Li <wingers@google.com>

show more ...

Merge changes from topic "psci-osi" into integration

* changes:
feat(sc7280): add support for PSCI_OS_INIT_MODE
feat(fvp): enable support for PSCI OS-initiated mode
feat(psci): update PSCI_FEA

Merge changes from topic "psci-osi" into integration

* changes:
feat(sc7280): add support for PSCI_OS_INIT_MODE
feat(fvp): enable support for PSCI OS-initiated mode
feat(psci): update PSCI_FEATURES
feat(psci): add support for OS-initiated mode
feat(psci): add support for PSCI_SET_SUSPEND_MODE
build(psci): add build option for OS-initiated mode
docs(psci): add design proposal for OS-initiated mode

show more ...

feat(psci): add support for OS-initiated mode

This patch adds a `psci_validate_state_coordination` function that is
called by `psci_cpu_suspend_start` in OS-initiated mode.

This function validates

feat(psci): add support for OS-initiated mode

This patch adds a `psci_validate_state_coordination` function that is
called by `psci_cpu_suspend_start` in OS-initiated mode.

This function validates the request per sections 4.2.3.2, 5.4.5, and 6.3
of the PSCI spec (DEN0022D.b):
- The requested power states are consistent with the system's state
- The calling core is the last running core at the requested power level

This function differs from `psci_do_state_coordination` in that:
- The `psci_req_local_pwr_states` map is not modified if the request
were to be denied
- The `state_info` argument is never modified since it contains the
power states requested by the calling OS

This is conditionally compiled into the build depending on the value of
the `PSCI_OS_INIT_MODE` build option.

Change-Id: I667041c842d2856e9d128c98db4d5ae4e4552df3
Signed-off-by: Wing Li <wingers@google.com>

show more ...

feat(psci): add support for PSCI_SET_SUSPEND_MODE

This patch adds a PSCI_SET_SUSPEND_MODE handler that validates the
request per section 5.20.2 of the PSCI spec (DEN0022D.b), and updates
the suspend

feat(psci): add support for PSCI_SET_SUSPEND_MODE

This patch adds a PSCI_SET_SUSPEND_MODE handler that validates the
request per section 5.20.2 of the PSCI spec (DEN0022D.b), and updates
the suspend mode to the requested mode.

This is conditionally compiled into the build depending on the value of
the `PSCI_OS_INIT_MODE` build option.

Change-Id: Iebf65f5f7846aef6b8643ad6082db99b4dcc4bef
Signed-off-by: Wing Li <wingers@google.com>

show more ...

Merge "fix(cpus): workaround for Cortex-A510 erratum 2684597" into integration

fix(cpus): workaround for Cortex-A510 erratum 2684597

Cortex-A510 erratum 2684597 is a Cat B erratum that applies to revisions
r0p0, r0p1, r0p2, r0p3, r1p0, r1p1 and r1p2. It is fixed in r1p3. The
w

fix(cpus): workaround for Cortex-A510 erratum 2684597

Cortex-A510 erratum 2684597 is a Cat B erratum that applies to revisions
r0p0, r0p1, r0p2, r0p3, r1p0, r1p1 and r1p2. It is fixed in r1p3. The
workaround is to execute a TSB CSYNC and DSB before executing WFI for
power down.

SDEN can be found here:
https://developer.arm.com/documentation/SDEN1873361/latest
https://developer.arm.com/documentation/SDEN1873351/latest

Change-Id: Ic0b24b600bc013eb59c797401fbdc9bda8058d6d
Signed-off-by: Harrison Mutai <harrison.mutai@arm.com>

show more ...

Merge "refactor(psci): unify psci_is_last_on_cpu and psci_is_last_on_cpu_safe" into integration

refactor(psci): unify psci_is_last_on_cpu and psci_is_last_on_cpu_safe

"psci_is_last_on_cpu" and "psci_is_last_on_cpu_safe" modules perform
mostly similar functionalities, verifying whether the curr

refactor(psci): unify psci_is_last_on_cpu and psci_is_last_on_cpu_safe

"psci_is_last_on_cpu" and "psci_is_last_on_cpu_safe" modules perform
mostly similar functionalities, verifying whether the current CPU
is the only active core and other cores have been turned off.

However, psci_is_last_on_cpu_safe function differs from the other with:
1. Safe API locks the power domain

This patch removes the section duplicating the functionality
and ensures that "psci_is_last_on_cpu api",is reused in
"psci_is_last_on_cpu_safe" procedure.

Signed-off-by: Jayanth Dodderi Chidanand <jayanthdodderi.chidanand@arm.com>
Change-Id: Ie372519e423898d7afa5427cdd77a7f9d3369587

show more ...

Merge changes from topic "RDN2_WARM_REBOOT_WITH_SGI" into integration

* changes:
feat(sgi): enable css implementation of warm reset
feat(scmi): send powerdown request to online secondary cpus

Merge changes from topic "RDN2_WARM_REBOOT_WITH_SGI" into integration

* changes:
feat(sgi): enable css implementation of warm reset
feat(scmi): send powerdown request to online secondary cpus
feat(plat/arm/css): add interrupt handler for reboot request
refactor(psci): move psci_do_pwrdown_sequence() out of private header
feat(plat/arm/css): add per-cpu power down support for warm reset
feat(scmi): set warm reboot entry point
fix(gicv3): update the affinity mask to 8 bit

show more ...

refactor(psci): move psci_do_pwrdown_sequence() out of private header

Move the psci_do_pwrdown_sequence() function declaration from PSCI
private header to common header. The psci_do_pwrdown_sequence

refactor(psci): move psci_do_pwrdown_sequence() out of private header

Move the psci_do_pwrdown_sequence() function declaration from PSCI
private header to common header. The psci_do_pwrdown_sequence is
required to support warm reset, where each CPU need to execute the
powerdown sequence.

Change-Id: I298e7a120be814941fa91c0b001002a080e56263
Signed-off-by: Pranav Madhu <pranav.madhu@arm.com>

show more ...