core: optimize AArch32 AES-GCM routines

In AArch32 there are not enough SIMD registers to make a fused GHASH and
AES-CTR assembly function. But we can do better than using the default
implementation

core: optimize AArch32 AES-GCM routines

In AArch32 there are not enough SIMD registers to make a fused GHASH and
AES-CTR assembly function. But we can do better than using the default
implementation. By carefully using the GHASH and AES primitive assembly
functions there's some gain in performance.

Before this patch throughput was around 12MiB/s to now a bit more than
110MiB/s with blocks of 4096 bytes.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: add ce_aes_xor_block()

Adds ce_aes_xor_block() which xors two memory blocks of size
TEE_AES_BLOCK_SIZE and saves the result back into memory. The operations
are done with SIMD instructions so

core: add ce_aes_xor_block()

Adds ce_aes_xor_block() which xors two memory blocks of size
TEE_AES_BLOCK_SIZE and saves the result back into memory. The operations
are done with SIMD instructions so the memory blocks may be unaligned,
but VFP must be enabled with thread_kernel_enable_vfp().

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: crypto: remove internal_aes_gcm_expand_enc_key()

Removes internal_aes_gcm_expand_enc_key() which is replaced by
crypto_aes_expand_enc_key().

Reviewed-by: Etienne Carriere <etienne.carriere@li

core: crypto: remove internal_aes_gcm_expand_enc_key()

Removes internal_aes_gcm_expand_enc_key() which is replaced by
crypto_aes_expand_enc_key().

Reviewed-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: update AArch64 GHASH acceleration routines

Update AArch64 GHASH acceleration routines for improved performance.

The core parts of assembly and wrapper updates are written by
Ard Biesheuvel <a

core: update AArch64 GHASH acceleration routines

Update AArch64 GHASH acceleration routines for improved performance.

The core parts of assembly and wrapper updates are written by
Ard Biesheuvel <ard.biesheuvel@linaro.org>, see [1].

Link: [1] https://github.com/torvalds/linux/commit/22240df7ac6d76a271197571a7be45addef2ba15
Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: crypto: add internal_aes_gcm_{en,de}crypt_block()

Adds internal_aes_gcm_encrypt_block() and
internal_aes_gcm_decrypt_block() to encrypt or decrypt a well aligned
AES-GCM payload block.

Review

core: crypto: add internal_aes_gcm_{en,de}crypt_block()

Adds internal_aes_gcm_encrypt_block() and
internal_aes_gcm_decrypt_block() to encrypt or decrypt a well aligned
AES-GCM payload block.

Reviewed-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: crypto: remove internal_aes_gcm_encrypt_block()

Replaces calls to internal_aes_gcm_encrypt_block() with calls to
crypto_aes_enc_block(). Removes internal_aes_gcm_encrypt_block().

Reviewed-by:

core: crypto: remove internal_aes_gcm_encrypt_block()

Replaces calls to internal_aes_gcm_encrypt_block() with calls to
crypto_aes_enc_block(). Removes internal_aes_gcm_encrypt_block().

Reviewed-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: crypto: unaligned aes-gcm acceleration

The Arm CE code supports working with unaligned data. In order to make
full use of that is the generic __weak function
internal_aes_gcm_update_payload_bl

core: crypto: unaligned aes-gcm acceleration

The Arm CE code supports working with unaligned data. In order to make
full use of that is the generic __weak function
internal_aes_gcm_update_payload_block_aligned() replaced with
internal_aes_gcm_update_payload_blocks(). The latter now supports
working with unaligned buffers.

Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: arm: pmull_ghash_update_*() accepts unaligned payload

Updates the relevant ld1 and vld1 instructions for AArch64 and AArch32
respectively to allow unaligned src and head parameters.

Reviewed-

core: arm: pmull_ghash_update_*() accepts unaligned payload

Updates the relevant ld1 and vld1 instructions for AArch64 and AArch32
respectively to allow unaligned src and head parameters.

Reviewed-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: crypto: refactor aes-gcm implementation

Adds struct internal_ghash_key to represent the ghash key instead of
some lose fields inside struct internal_aes_gcm_state.

Software of CE configuratio

core: crypto: refactor aes-gcm implementation

Adds struct internal_ghash_key to represent the ghash key instead of
some lose fields inside struct internal_aes_gcm_state.

Software of CE configuration is done explicitly in
core/crypto/aes-gcm-sw.c, dropping the __weak attribute for all
functions but internal_aes_gcm_update_payload_block_aligned() which
is only overridden with CFG_CRYPTO_WITH_CE=y in AArch64.

Content of aes-gcm-private.h is moved into internal_aes-gcm.h.

internal_aes_gcm_gfmul() is made available for generic GF
multiplication.

The CE versions of internal_aes_gcm_expand_enc_key() and
internal_aes_gcm_encrypt_block() are now only wrappers around
crypto_accel_aes_expand_keys() and crypto_accel_aes_ecb_enc().

Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: fobj.c: use crypto_aes_expand_enc_key()

fobj_generate_authenc_key() uses crypto_aes_expand_enc_key() instead
to prepare the key used for paging.

Acked-by: Etienne Carriere <etienne.carriere@l

core: fobj.c: use crypto_aes_expand_enc_key()

fobj_generate_authenc_key() uses crypto_aes_expand_enc_key() instead
to prepare the key used for paging.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: use SHA-256 crypto accelerated function

Uses the recently provided accelerated SHA-256 function in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wikla

core: ltc: use SHA-256 crypto accelerated function

Uses the recently provided accelerated SHA-256 function in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: use SHA1 crypto accelerated function

Uses the recently provided accelerated SHA1 function in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <

core: ltc: use SHA1 crypto accelerated function

Uses the recently provided accelerated SHA1 function in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: use AES crypto accelerated routines

Uses the recently provided accelerated AES crypto routines in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklan

core: ltc: use AES crypto accelerated routines

Uses the recently provided accelerated AES crypto routines in LTC.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: add accelerated SHA-256 routines

Adds an Arm CE accelerated SHA-256 function to core/arch/arm/crypto. The
code originates from the previous implementation inside LTC library.
With this multipl

core: add accelerated SHA-256 routines

Adds an Arm CE accelerated SHA-256 function to core/arch/arm/crypto. The
code originates from the previous implementation inside LTC library.
With this multiple crypto libraries can share the function.

The old CFG_CRYPTO_SHA256_ARM64_CE and CFG_CRYPTO_SHA256_ARM32_CE are
replaced by CFG_CRYPTO_SHA256_ARM_CE.

CFG_CORE_CRYPTO_SHA256_ACCEL is introduced as to indicate that some kind of
SHA-256 acceleration is available, not necessarily based on Arm CE.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: add accelerated SHA1 routines

Adds an Arm CE accelerated SHA1 function to core/arch/arm/crypto. The code
originates from the previous implementation inside LTC library. With
this multiple cryp

core: add accelerated SHA1 routines

Adds an Arm CE accelerated SHA1 function to core/arch/arm/crypto. The code
originates from the previous implementation inside LTC library. With
this multiple crypto libraries can share the function.

The old CFG_CRYPTO_SHA1_ARM64_CE and CFG_CRYPTO_SHA1_ARM32_CE are
replaced by CFG_CRYPTO_SHA1_ARM_CE.

CFG_CORE_CRYPTO_SHA1_ACCEL is introduced as to indicate that some kind of
SHA-1 acceleration is available, not necessarily based on Arm CE.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: add accelerated AES routines

Adds Arm CE accelerated AES routines to core/arch/arm/crypto. The code
originates from the previous implementation inside LTC library. With
this multiple crypto li

core: add accelerated AES routines

Adds Arm CE accelerated AES routines to core/arch/arm/crypto. The code
originates from the previous implementation inside LTC library. With
this multiple crypto library can share these routines.

A new header file, <crypto/crypto_accel.h>, is added with primitive
functions implementing crypto accelerated ciphers.

The old CFG_CRYPTO_AES_ARM64_CE and CFG_CRYPTO_AES_ARM32_CE are
replaced by CFG_CRYPTO_AES_ARM_CE.

CFG_CORE_CRYPTO_AES_ACCEL is introduced as to indicate that some kind of
AES acceleration is available, not necessarily based on Arm CE.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: remove Arm SHA-256 CE routines

Removes the Arm CE routines accelerating SHA-256 in the LTC library.

This will later be added in common code to be shared with other crypto
libraries etc.

core: ltc: remove Arm SHA-256 CE routines

Removes the Arm CE routines accelerating SHA-256 in the LTC library.

This will later be added in common code to be shared with other crypto
libraries etc.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: remove Arm SHA1 CE routines

Removes the Arm CE routines accelerating SHA1 in the LTC library.

This will later be added in common code to be shared with other crypto
libraries etc.

Acked

core: ltc: remove Arm SHA1 CE routines

Removes the Arm CE routines accelerating SHA1 in the LTC library.

This will later be added in common code to be shared with other crypto
libraries etc.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: remove Arm AES CE routines

Removes the Arm AES CE routines from the library.

This will later be added in common code to be shared with other crypto
libraries etc.

Acked-by: Etienne Carr

core: ltc: remove Arm AES CE routines

Removes the Arm AES CE routines from the library.

This will later be added in common code to be shared with other crypto
libraries etc.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core: ltc: simplify _CFG_CORE_LTC_*_DESC logic

If a SHA-{256,384,512} or AES algorithm is needed in LTC a matching
_CFG_CORE_LTC_*_DESC variable will be set. So only check the
_CFG_CORE_LTC_*_DESC v

core: ltc: simplify _CFG_CORE_LTC_*_DESC logic

If a SHA-{256,384,512} or AES algorithm is needed in LTC a matching
_CFG_CORE_LTC_*_DESC variable will be set. So only check the
_CFG_CORE_LTC_*_DESC variable to see if a certain algorithm is needed.

Acked-by: Etienne Carriere <etienne.carriere@linaro.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

core/crypto.mk: simplify _CFG_CORE_LTC_*_DESC logic

Simplifies _CFG_CORE_LTC_*_DESC logic by always defining the
corresponding _CFG_CORE_LTC_*_DESC variable to the algorithm needed
inside LTC.

Acke

core/crypto.mk: simplify _CFG_CORE_LTC_*_DESC logic

Simplifies _CFG_CORE_LTC_*_DESC logic by always defining the
corresponding _CFG_CORE_LTC_*_DESC variable to the algorithm needed
inside LTC.

Acked-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

pta: invoke_test.pta: add aes performance test

Adds test PTA function to run performance test with xtest --aes-perf.

Reviewed-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wikland

pta: invoke_test.pta: add aes performance test

Adds test PTA function to run performance test with xtest --aes-perf.

Reviewed-by: Jerome Forissier <jerome@forissier.org>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

drivers/scmi-msg: smt entry points for incoming messages

This change implements SCMI channels for reading a SCMI message from a
shared memory and call the SCMI message drivers to route the message
t

drivers/scmi-msg: smt entry points for incoming messages

This change implements SCMI channels for reading a SCMI message from a
shared memory and call the SCMI message drivers to route the message
to the target platform services.

SMT refers to the shared memory management protocol which is used
to get/put message/response in shared memory. SMT is a 28byte header
stating shared memory state and exchanged protocol data.

The processing entry for a SCMI message can be a secure interrupt
(CFG_SCMI_MSG_SMT_INTERRUPT_ENTRY=y), and fastcall SMC
(CFG_SCMI_MSG_SMT_FASTCALL_ENTRY=y) or a threaded execution
context entry (CFG_SCMI_MSG_SMT_THREAD_ENTRY=y).

SMT description in this implementation is based on the SCP-firmware
implementation [1].

Link: [1] https://github.com/ARM-software/SCP-firmware.git

Signed-off-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

drivers/scmi-msg: support for reset domain protocol

Adds SCMI reset domain protocol support in the SCMI message drivers
as defined in SCMI specification v2.0 [1]. Not all the messages
defined in the

drivers/scmi-msg: support for reset domain protocol

Adds SCMI reset domain protocol support in the SCMI message drivers
as defined in SCMI specification v2.0 [1]. Not all the messages
defined in the specification are supported.

Embedded upon CFG_SCMI_MSG_RESET_DOMAIN=y.

scmi_msg_get_rd_handler() sanitizes the message_id value
against any speculative use of reset domain ID as a index since by
SCMI specification, IDs are indices.

SCMI resource in this implementation are dumped or inspired by the
SCP-firmware implementation [2] of the SCMI protocol, server side.

Link: [1] http://infocenter.arm.com/help/topic/com.arm.doc.den0056a/DEN0056A_System_Control_and_Management_Interface.pdf
Link: [2] https://github.com/ARM-software/SCP-firmware.git

Signed-off-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...

drivers/scmi-msg: support for clock protocol

Adds SCMI clock protocol support in the SCMI message drivers as
defined in SCMI specification v2.0 [1]. Not all the messages
defined in the specification

drivers/scmi-msg: support for clock protocol

Adds SCMI clock protocol support in the SCMI message drivers as
defined in SCMI specification v2.0 [1]. Not all the messages
defined in the specification are supported.

Embedded upon CFG_SCMI_MSG_CLOCK=y.

Platform can provide one of the plat_scmi_clock_*() handler for the
supported operations set/get state/rate and others.

scmi_msg_get_clock_handler() sanitizes the message_id value
against any speculative use of clock ID as a index since by
SCMI specification, IDs are indices.

SCMI resource in this implementation are dumped or inspired by the
SCP-firmware implementation [2] of the SCMI protocol, server side.

Link: [1] http://infocenter.arm.com/help/topic/com.arm.doc.den0056a/DEN0056A_System_Control_and_Management_Interface.pdf
Link: [2] https://github.com/ARM-software/SCP-firmware.git

Signed-off-by: Etienne Carriere <etienne.carriere@linaro.org>
Acked-by: Jens Wiklander <jens.wiklander@linaro.org>

show more ...