core: add thread_count to thread_init_threads()

Add a thread_count parameter to thread_init_threads(). This must currently
always be equal to CFG_NUM_THREADS, but may become a dynamic configuration

core: add thread_count to thread_init_threads()

Add a thread_count parameter to thread_init_threads(). This must currently
always be equal to CFG_NUM_THREADS, but may become a dynamic configuration
parameter with CFG_DYN_CONFIG=y in later patches.

The array threads[] is changed into a pointer to allow dynamic
allocation in later patches. The assembly code is updated accordingly to
handle a pointer instead of an array.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Jerome Forissier <jerome.forissier@linaro.org>
Acked-by: Alvin Chang <alvinga@andestech.com>
Tested-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu-Chien Peter Lin <peter.lin@sifive.com>
Tested-by: Yu-Chien Peter Lin <peter.lin@sifive.com>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>

show more ...

riscv: kernel: support booting non-contiguous non-zero-based hart IDs

Currently, OP-TEE assumes 0 <= hartid < CFG_TEE_CORE_NB_CORE,
and must be contiguous, which fails to accommodate different
CPU t

riscv: kernel: support booting non-contiguous non-zero-based hart IDs

Currently, OP-TEE assumes 0 <= hartid < CFG_TEE_CORE_NB_CORE,
and must be contiguous, which fails to accommodate different
CPU topologies. For example, some RISC-V platforms, such as
the HiFive Unmatched board, do not run Linux and OP-TEE on
hart0, as it is a monitor core without supervisor mode support.

To address this, introduce hart_index, which is used to index
per-hart structures, such as thread_core_local and root_pgt.
The hart_index will range from 0 to (CFG_TEE_CORE_NB_CORE - 1),
and the primary hart will have an index of 0.

Additionally, a new function, boot_primary_init_core_ids(),
is added to initialize secondary hart IDs for booting via
sbi_hsm_hart_start().

Signed-off-by: Yu-Chien Peter Lin <peter.lin@sifive.com>
Reviewed-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Zong Li <zong.li@sifive.com>
Reviewed-by: Alvin Chang <alvinga@andestech.com>

show more ...

core: riscv: Improve thread user mode record

Make the asm definitions be more human-readable.

Besides, it's unnecessary to save and restore kernel SP and GP into
thread_user_mode_rec, since they wi

core: riscv: Improve thread user mode record

Make the asm definitions be more human-readable.

Besides, it's unnecessary to save and restore kernel SP and GP into
thread_user_mode_rec, since they will be setup by system call trap
handler before executing thread_unwind_user_mode().

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Ensure XSTATUS is restored before XIE

In previous implementation, we found some accidental interrupts during
entering user mode and resuming of thread. We fixed it by clearing
XSTATUS.X

core: riscv: Ensure XSTATUS is restored before XIE

In previous implementation, we found some accidental interrupts during
entering user mode and resuming of thread. We fixed it by clearing
XSTATUS.XIE first, which is global interrupt enable bit, to ensure there
are no interrupts during those operations.

Now we found the better solution: restore XSTATUS before restoring XIE.
This can ensure the global interrupt bit in XSTATUS is cleared before we
restore the individual interrupt bits in XIE.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Remove thread_exit_user_mode()

Currently, the user mode abort and some system calls return to kernel
mode by thread_exit_user_mode(). Although this function creates a
shorter path to re

core: riscv: Remove thread_exit_user_mode()

Currently, the user mode abort and some system calls return to kernel
mode by thread_exit_user_mode(). Although this function creates a
shorter path to return to kernel mode, it leads to some problems because
the function does not update the core local flags. Especially when
CFG_CORE_DEBUG_CHECK_STACKS=y, some checks will fail due to wrong type
of stack recorded in the core local flags.

Fix it by removing thread_exit_user_mode(). So that the core local flags
can be correctly updated in the common trap handler.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Use sp as base register of load instructions

Use sp as base register of load instructions can reduce code size if RVC
extension is enabled to generate 16-bit instructions. The following

core: riscv: Use sp as base register of load instructions

Use sp as base register of load instructions can reduce code size if RVC
extension is enabled to generate 16-bit instructions. The following code
shows the difference after applying this commit.

Before:
f10009da: 0d053d83 ld s11,208(a0)
f10009de: 0c853d03 ld s10,200(a0)
f10009e2: 0c053c83 ld s9,192(a0)
f10009e6: 0b853c03 ld s8,184(a0)
f10009ea: 0b053b83 ld s7,176(a0)
f10009ee: 0a853b03 ld s6,168(a0)
f10009f2: 0a053a83 ld s5,160(a0)
f10009f6: 09853a03 ld s4,152(a0)
f10009fa: 09053983 ld s3,144(a0)
f10009fe: 08853903 ld s2,136(a0)

After:
f10009a6: 6dce ld s11,208(sp)
f10009a8: 6d2e ld s10,200(sp)
f10009aa: 6c8e ld s9,192(sp)
f10009ac: 7c6a ld s8,184(sp)
f10009ae: 7bca ld s7,176(sp)
f10009b0: 7b2a ld s6,168(sp)
f10009b2: 7a8a ld s5,160(sp)
f10009b4: 6a6a ld s4,152(sp)
f10009b6: 69ca ld s3,144(sp)
f10009b8: 692a ld s2,136(sp)

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Fix misconfiguration of XSCRATCH when XRET to kernel mode

When the program wants to XRET to kernel mode, the value of XSCRATCH
must be cleared to zero.

Signed-off-by: Alvin Chang <alvi

core: riscv: Fix misconfiguration of XSCRATCH when XRET to kernel mode

When the program wants to XRET to kernel mode, the value of XSCRATCH
must be cleared to zero.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Set exception return PC into XEPC for entering user mode

Instead of setting exception return PC into "ra" register and assign it
to XEPC, we should directly set exception return PC into

core: riscv: Set exception return PC into XEPC for entering user mode

Instead of setting exception return PC into "ra" register and assign it
to XEPC, we should directly set exception return PC into "XEPC" CSR to
improve code redability.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

show more ...

core: riscv: Update thread core local flags for native interrupts

Record the type of stack and the type of interrupt into thread core
local flags before handling the native interrupts. Once we finis

core: riscv: Update thread core local flags for native interrupts

Record the type of stack and the type of interrupt into thread core
local flags before handling the native interrupts. Once we finish the
handling of the native interrupts, we clear the previous flags.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: riscv: Determine foreign interrupts during interrupt handling

The foreign interrupts are defined in THREAD_EXCP_FOREIGN_INTR, whose
bit fields are corresponding to RISC-V MIP/SIP CSR. Thus, we

core: riscv: Determine foreign interrupts during interrupt handling

The foreign interrupts are defined in THREAD_EXCP_FOREIGN_INTR, whose
bit fields are corresponding to RISC-V MIP/SIP CSR. Thus, we can
determine whether the coming interrupt is foreign interrupt or not by
THREAD_EXCP_FOREIGN_INTR.

However, the trap information is encoded in XCAUSE CSR. Thus, we must
translate the value of XCAUSE to XIP first, and then compare the XIP
with THREAD_EXCP_FOREIGN_INTR. For example, if the coming interrupt is
supervisor timer interrupt, the value of SCAUSE will be
0x8000000000000005 on RV64 system. We need to translate it to be
sip.STIP, which is 0x20. Then, we can further compare 0x20 with
THREAD_EXCP_FOREIGN_INTR. If 0x20 is subset of THREAD_EXCP_FOREIGN_INTR,
it is determined as a foreign interrupt. Finally, we call
thread_foreign_interrupt_handler() to further handle it.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: riscv: Implement foreign interrupt handler

In SMP based OP-TEE system, the foreign interrupts, which are handled
outside OP-TEE, might be injected during the execution of OP-TEE. When
OP-TEE r

core: riscv: Implement foreign interrupt handler

In SMP based OP-TEE system, the foreign interrupts, which are handled
outside OP-TEE, might be injected during the execution of OP-TEE. When
OP-TEE receives such foreign interrupts, it should suspend current
thread and yield the execution to the foreign domains to handle those
foreign interrupts.

This commit implements thread_foreign_interrupt_handler(), which is
expected to be called during trap handling. The job of this function are
suspending the current thread, and returning to the other domain to let
them handle the foreign interrupts.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: riscv: Rename and re-order parameters of the interrupt handlers

Rename thread_interrupt_handler() to thread_native_interrupt_handler(),
since we are going to support the foreign interrupt hand

core: riscv: Rename and re-order parameters of the interrupt handlers

Rename thread_interrupt_handler() to thread_native_interrupt_handler(),
since we are going to support the foreign interrupt handler.

The native interrupts are handled in OP-TEE side, while the foreign
interrupts are handled outside OP-TEE.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: riscv: Improve the trap handler

In current RISC-V trap handler, we always save the context into struct
thread_trap_regs first, and copy the saved context into another specific
structure to han

core: riscv: Improve the trap handler

In current RISC-V trap handler, we always save the context into struct
thread_trap_regs first, and copy the saved context into another specific
structure to handle that trap. For example, thread_user_ecall_handler()
copies the struct thread_trap_regs to be struct thread_scall_regs before
handling this system call. In fact, such copies may drop the performance
and they should be eliminated.

In this commit, the specific data structure used to save the context is
determined early in the trap handler by checking the type of the trap
from CSR XCAUSE. Thus, the copies between context structures are
eliminated.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Tested-by: Yu Chien Peter Lin <peterlin@andestech.com>
Tested-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: Refine thread trap handler

In order to support SMP, we made change on CSR SCRATCH from kernel stack
pointer to be kernel TP(thread_core_local). So that we can get TP from
SCRATCH easily

core: riscv: Refine thread trap handler

In order to support SMP, we made change on CSR SCRATCH from kernel stack
pointer to be kernel TP(thread_core_local). So that we can get TP from
SCRATCH easily in trap handler when the thread is in user mode. We also
save/restore CSR IE, kernel GP and SP so that we can handle task
migration to another hart.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Tested-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: Refine thread enter/exit user mode

Now when thread is in user mode, the kernel TP is saved into CSR SCRATCH
instead of into kernel stack. The IE is also considered since it
contains mas

core: riscv: Refine thread enter/exit user mode

Now when thread is in user mode, the kernel TP is saved into CSR SCRATCH
instead of into kernel stack. The IE is also considered since it
contains masks of different exceptions.

Apply exception return to thread_exit_user_mode() to let hart correctly
back to kernel mode from exception.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Tested-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: Apply exception return to resume thread

In current implementation, the thread is resumed by function return. It
is not suitable for all scenarios, especially when the thread should be
r

core: riscv: Apply exception return to resume thread

In current implementation, the thread is resumed by function return. It
is not suitable for all scenarios, especially when the thread should be
resumed to user mode. The kernel mode can not return to user mode by
pure function return.

This commit applies exception return to resume the thread. The EPC and
IE are added into thread context. The xstatus_for_xret() helper function
is added to prepare the value of CSR STATUS for exception return.
Currently we only consider PIE(previous interrupt-enable) and
PP(previous privilege mode) for exception return.

We clear thread context when the context is reinitialized, enable native
interrupt, and setup kernel GP/TP. The thread_resume() now takes care of
restoring CSR EPC, STATUS, IE, SCRATCH and all general-purpose
registers. Finally it executes exception return to target privilege mode
encoded in CSR STATUS. The registers GP and TP are also restored since
user mode may use them.

This commit also modify the usage of CSR SCRATCH. In current
implementation the SCRATCH is used to save kernel stack pointer when the
thread is in user mode. The value of TP, which stores thread_core_local
structure, is saved into kernel stack before entering user mode. The
trap handler can then get TP(thread_core_local) from kernel stack. This
is not suitable for SMP system, since the thread might be resumed to
another core, and that core gets wrong TP from kernel stack. Fix it by
directly storing TP into CSR SCRATCH.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Tested-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: move TEE ABI handlers from thread_rv.S to thread_optee_abi_rv.S

This commits moves the following functions from thread_rv.S to a separate
file thread_optee_abi_rv.S:

- thread_return_fr

core: riscv: move TEE ABI handlers from thread_rv.S to thread_optee_abi_rv.S

This commits moves the following functions from thread_rv.S to a separate
file thread_optee_abi_rv.S:

- thread_return_from_nsec_call()
- thread_std_smc_entry() -> renamed to thread_std_abi_entry()
- thread_rpc()

Signed-off-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Reviewed-by: Alvin Chang <alvinga@andestech.com>
Tested-by: Alvin Chang <alvinga@andestech.com>

show more ...

core: riscv: Load register TP from thread_user_mode_rec in trap handler

RISC-V kernel uses TP register to store thread_core_local structure.
When the thread enters user mode, the value of TP is used

core: riscv: Load register TP from thread_user_mode_rec in trap handler

RISC-V kernel uses TP register to store thread_core_local structure.
When the thread enters user mode, the value of TP is used by user mode.
Therefore, when CPU enters trap handler, it needs to restore TP to get
thread_core_local structure. In previous implementation, the value of TP
is saved under kernel SP before entering user mode, and the trap handler
restores TP from that stack location. However, the value of TP has
already been saved into the thread_user_mode_rec structure, which is
also upon kernel SP, before entering user mode. So the value of TP can
be restored just from thread_user_mode_rec, instead of saving into
another location which is under the kernel SP.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Tested-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: Fix thread_rpc() wrong stack usage and CSR value

Since there are four registers to be stored onto stack, we should
preserve up to 32 bytes space on the stack instead of only 16 bytes,
o

core: riscv: Fix thread_rpc() wrong stack usage and CSR value

Since there are four registers to be stored onto stack, we should
preserve up to 32 bytes space on the stack instead of only 16 bytes,
otherwise the stack overflow occurs. The s0 is regarded as frame
pointer. The value of CSR status is also restored before returning from
thread_rpc().

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Marouene Boubakri <marouene.boubakri@nxp.com>

show more ...

core: riscv: Save and restore register s0 when trap occurs

In RISC-V the frame pointer is stored into register s0. It is necessary
to save/restore s0 when we enable CFG_UNWIND to process stack unwin

core: riscv: Save and restore register s0 when trap occurs

In RISC-V the frame pointer is stored into register s0. It is necessary
to save/restore s0 when we enable CFG_UNWIND to process stack unwinding.

Signed-off-by: Alvin Chang <alvinga@andestech.com>
Reviewed-by: Jerome Forissier <jerome.forissier@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>

show more ...

core: riscv: add thread manager assembly code in thread_rv.S

This commit implements:
- An entry point of trap handler in non-vectored mode.
- thread_unwind_user_mode() and thread_exit_user_mode()
to

core: riscv: add thread manager assembly code in thread_rv.S

This commit implements:
- An entry point of trap handler in non-vectored mode.
- thread_unwind_user_mode() and thread_exit_user_mode()
to return from U-Mode.
- __thread_enter_user_mode() to jump to U-Mode from S-Mode or M-Mode.
- thread_std_smc_entry(), thread_resume() and thread_rpc().

Signed-off-by: Marouene Boubakri <marouene.boubakri@nxp.com>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...