1Trusted Firmware-A for Raspberry Pi 3 2===================================== 3 4.. section-numbering:: 5 :suffix: . 6 7.. contents:: 8 9The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four 10Arm Cortex-A53 cores. 11 12The following instructions explain how to use this port of the TF-A with the 13default distribution of `Raspbian`_ because that's the distribution officially 14supported by the Raspberry Pi Foundation. At the moment of writing this, the 15officially supported kernel is a AArch32 kernel. This doesn't mean that this 16port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by 17the Foundation can be compiled for AArch64 by following the steps in 18`AArch64 kernel build instructions`_. 19 20**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM, 21which is available from both the Non-secure and Secure worlds. This port 22shouldn't be considered more than a prototype to play with and implement 23elements like PSCI to support the Linux kernel. 24 25Design 26------ 27 28The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a 29VideoCore IV that acts as primary processor (and loads everything from the SD 30card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi 313 documentation`_ for more information. 32 33This explains why it is possible to change the execution state (AArch64/AArch32) 34depending on a few files on the SD card. We only care about the cases in which 35the cores boot in AArch64 mode. 36 37The rules are simple: 38 39- If a file called ``kernel8.img`` is located on the ``boot`` partition of the 40 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes 41 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel. 42 43- If there is also a file called ``armstub8.bin``, it will load it at address 44 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All 45 the cores are powered on at the same time and start at address **0x0**. 46 47This means that we can use the default AArch32 kernel provided in the official 48`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and 49anything else we need is in ``armstub8.bin``. This way we can forget about the 50default bootstrap code. When using a AArch64 kernel, it is only needed to make 51sure that the name on the SD card is ``kernel8.img``. 52 53Ideally, we want to load the kernel and have all cores available, which means 54that we need to make the secondary cores work in the way the kernel expects, as 55explained in `Secondary cores`_. In practice, a small bootstrap is needed 56between TF-A and the kernel. 57 58To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode 59in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The 60architecture specifies that AArch32 Hypervisor mode isn't present when AArch64 61is used for EL2. When using a AArch64 kernel, it should simply start in EL2. 62 63Placement of images 64~~~~~~~~~~~~~~~~~~~ 65 66The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding 67between them so that the addresses they are loaded to match the ones specified 68when compiling TF-A. This is done automatically by the build system. 69 70The device tree block is loaded by the VideoCore loader from an appropriate 71file, but we can specify the address it is loaded to in ``config.txt``. 72 73The file ``kernel8.img`` contains a kernel image that is loaded to the address 74specified in ``config.txt``. The `Linux kernel tree`_ has information about how 75a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``: 76 77:: 78 79 The zImage may also be placed in system RAM and called there. The 80 kernel should be placed in the first 128MiB of RAM. It is recommended 81 that it is loaded above 32MiB in order to avoid the need to relocate 82 prior to decompression, which will make the boot process slightly 83 faster. 84 85There are no similar restrictions for AArch64 kernels, as specified in the file 86``Documentation/arm64/booting.txt``. 87 88This means that we need to avoid the first 128 MiB of RAM when placing the 89TF-A images (and specially the first 32 MiB, as they are directly used to 90place the uncompressed AArch32 kernel image. This way, both AArch32 and 91AArch64 kernels can be placed at the same address. 92 93In the end, the images look like the following diagram when placed in memory. 94All addresses are Physical Addresses from the point of view of the Arm cores. 95Again, note that this is all just part of the same DRAM that goes from 96**0x00000000** to **0x3F000000**, it just has different names to simulate a real 97secure platform! 98 99:: 100 101 0x00000000 +-----------------+ 102 | ROM | BL1 103 0x00020000 +-----------------+ 104 | FIP | 105 0x00200000 +-----------------+ 106 | | 107 | ... | 108 | | 109 0x01000000 +-----------------+ 110 | DTB | (Loaded by the VideoCore) 111 +-----------------+ 112 | | 113 | ... | 114 | | 115 0x02000000 +-----------------+ 116 | Kernel | (Loaded by the VideoCore) 117 +-----------------+ 118 | | 119 | ... | 120 | | 121 0x10000000 +-----------------+ 122 | Secure SRAM | BL2, BL31 123 0x10100000 +-----------------+ 124 | Secure DRAM | BL32 (Secure payload) 125 0x11000000 +-----------------+ 126 | Non-secure DRAM | BL33 127 +-----------------+ 128 | | 129 | ... | 130 | | 131 0x3F000000 +-----------------+ 132 | I/O | 133 0x40000000 +-----------------+ 134 135The area between **0x10000000** and **0x11000000** has to be manually protected 136so that the kernel doesn't use it. The current port tries to modify the live DTB 137to add a memreserve region that reserves the previously mentioned area. 138 139If this is not possible, the user may manually add ``memmap=16M$256M`` to the 140command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_ 141instructions to see how to do it. This system is strongly discouraged. 142 143The last 16 MiB of DRAM can only be accessed by the VideoCore, that has 144different mappings than the Arm cores in which the I/O addresses don't overlap 145the DRAM. The memory reserved to be used by the VideoCore is always placed at 146the end of the DRAM, so this space isn't wasted. 147 148Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for 149TF-A, there are 880 MiB available for Linux. 150 151Boot sequence 152~~~~~~~~~~~~~ 153 154The boot sequence of TF-A is the usual one except when booting an AArch32 155kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it 156can jump to the kernel in the same mode and let it take over that privilege 157level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of 158TF-A) it could only jump to the kernel in AArch32 in Supervisor mode. 159 160The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel 161in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The 162bootstrap should take care of this. 163 164This port support a direct boot of the Linux kernel from the firmware (as a BL33 165image). Alternatively, U-Boot or other bootloaders may be used. 166 167Secondary cores 168~~~~~~~~~~~~~~~ 169 170This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``, 171`PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn 172the system off, it simply reboots it and asks the VideoCore firmware to keep it 173in a low power mode permanently. 174 175The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to 176use mailboxes to trap the secondary cores until they are ready to jump to the 177kernel. This mailbox is located at a different address in the AArch32 default 178kernel than in the AArch64 kernel. 179 180Kernels with PSCI support can use the PSCI calls instead for a cleaner boot. 181 182Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During 183cold boot, all secondary cores wait in a loop until they are given given an 184address to jump to in this Mailbox (``bl31_warm_entrypoint``). 185 186Once BL31 has finished and the primary core has jumped to the BL33 payload, it 187has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop. 188The payload then makes them wait in another waitloop listening from messages 189from the kernel. When the primary CPU jumps into the kernel, it will send an 190address to the mailbox so that the secondary CPUs jump to it and are recognised 191by the kernel. 192 193Build Instructions 194------------------ 195 196To boot a AArch64 kernel, only the AArch64 toolchain is required. 197 198To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The 199AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit 200kernel. 201 202The build system concatenates BL1 and the FIP so that the addresses match the 203ones in the memory map. The resulting file is ``armstub8.bin``, located in the 204build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this 205file, follow the instructions in `Setup SD card`_. 206 207The following build options are supported: 208 209- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image. 210 By default this option is 0, which means that TF-A will jump to BL33 in EL2 211 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32 212 mode. 213 214- ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has 215 been preloaded by any other system than using the firmware. ``BL33`` isn't 216 needed in the build command line if this option is used. Specially useful 217 because the file ``kernel8.img`` can be loaded anywhere by modifying the file 218 ``config.txt``. It doesn't have to contain a kernel, it could have any 219 arbitrary payload. 220 221- ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct 222 boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE`` 223 is mandatory when the direct Linux kernel boot is used. Options 224 ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is 225 unlikely that the kernel image will fit in the space reserved for BL33 images. 226 This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a 227 32-bit kernel. The only thing this option does is to set the arguments in 228 registers x0-x3 or r0-r2 as expected by the kernel. 229 230- ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using 231 ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a 232 DTB in memory. 233 234- ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime 235 or disabled. ``-1`` (default) disables the runtime UART. Any other value 236 enables the default UART (currently UART1) for runtime messages. 237 238- ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory 239 mapping required for an UEFI firmware payload. These changes are needed 240 to be able to run Windows on ARM64. This option, which is disabled by 241 default, results in the following memory mappings: 242 243:: 244 245 0x00000000 +-----------------+ 246 | ROM | BL1 247 0x00010000 +-----------------+ 248 | DTB | (Loaded by the VideoCore) 249 0x00020000 +-----------------+ 250 | FIP | 251 0x00030000 +-----------------+ 252 | | 253 | UEFI PAYLOAD | 254 | | 255 0x00200000 +-----------------+ 256 | Secure SRAM | BL2, BL31 257 0x00300000 +-----------------+ 258 | Secure DRAM | BL32 (Secure payload) 259 0x00400000 +-----------------+ 260 | | 261 | | 262 | Non-secure DRAM | BL33 263 | | 264 | | 265 0x01000000 +-----------------+ 266 | | 267 | ... | 268 | | 269 0x3F000000 +-----------------+ 270 | I/O | 271 272 273- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional. 274 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__. 275 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin 276 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin`` 277 to put the binaries into the FIP. 278 279 Note: If OP-TEE is used it may be needed to add the following options to the 280 Linux command line so that the USB driver doesn't use FIQs: 281 ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``. 282 This will unfortunately reduce the performance of the USB driver. It is needed 283 when using Raspbian, for example. 284 285- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable 286 it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the 287 contents of the FIP automatically signed by the build process. The ROT key 288 will be generated and output to ``rot_key.pem`` in the build directory. It is 289 able to set ROT_KEY to your own key in PEM format. Also in order to build, 290 you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__. 291 ``MBEDTLS_DIR`` must point at the mbed TLS source directory. 292 293- ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of 294 the board. 295 296The following is not currently supported: 297 298- AArch32 for TF-A itself. 299 300- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any 301 address by changing the file ``armstub8.bin``, so there's no point in using 302 TF-A in this case. 303 304- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the 305 crash console uses the internal 16550 driver functions directly in order to be 306 able to print error messages during early crashes before setting up the 307 multi console API. 308 309Building the firmware for kernels that don't support PSCI 310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 311 312This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels 313always support PSCI, but they may not know that the system understands PSCI due 314to an incorrect DTB file. 315 316First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A 317bootstrap`_. Choose the one needed for the architecture of your kernel. 318 319Then compile TF-A. For a 32-bit kernel, use the following command line: 320 321.. code:: shell 322 323 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 324 RPI3_BL33_IN_AARCH32=1 \ 325 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin 326 327For a 64-bit kernel, use this other command line: 328 329.. code:: shell 330 331 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 332 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin 333 334However, enabling PSCI support in a 64-bit kernel is really easy. In the 335repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied 336to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes 337the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the 338instructions in `AArch64 kernel build instructions`_ to get a working 64-bit 339kernel image and supporting files. 340 341Building the firmware for kernels that support PSCI 342~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 343 344For a 64-bit kernel: 345 346.. code:: shell 347 348 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 349 PRELOADED_BL33_BASE=0x02000000 \ 350 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 351 RPI3_DIRECT_LINUX_BOOT=1 352 353For a 32-bit kernel: 354 355.. code:: shell 356 357 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 358 PRELOADED_BL33_BASE=0x02000000 \ 359 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 360 RPI3_DIRECT_LINUX_BOOT=1 \ 361 RPI3_BL33_IN_AARCH32=1 362 363AArch64 kernel build instructions 364--------------------------------- 365 366The following instructions show how to install and run a AArch64 kernel by 367using a SD card with the default `Raspbian`_ install as base. Skip them if you 368want to use the default 32-bit kernel. 369 370Note that this system won't be fully 64-bit because all the tools in the 371filesystem are 32-bit binaries, but it's a quick way to get it working, and it 372allows the user to run 64-bit binaries in addition to 32-bit binaries. 373 3741. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To 375 speed things up, do a shallow clone of the desired branch. 376 377.. code:: shell 378 379 git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux 380 cd linux 381 3822. Configure and compile the kernel. Adapt the number after ``-j`` so that it is 383 1.5 times the number of CPUs in your computer. This may take some time to 384 finish. 385 386.. code:: shell 387 388 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig 389 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 390 3913. Copy the kernel image and the device tree to the SD card. Replace the path 392 by the corresponding path in your computers to the ``boot`` partition of the 393 SD card. 394 395.. code:: shell 396 397 cp arch/arm64/boot/Image /path/to/boot/kernel8.img 398 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/ 399 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/ 400 4014. Install the kernel modules. Replace the path by the corresponding path to the 402 filesystem partition of the SD card on your computer. 403 404.. code:: shell 405 406 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ 407 INSTALL_MOD_PATH=/path/to/filesystem modules_install 408 4095. Follow the instructions in `Setup SD card`_ except for the step of renaming 410 the existing ``kernel7.img`` (we have already copied a AArch64 kernel). 411 412Setup SD card 413------------- 414 415The instructions assume that you have an SD card with a fresh install of 416`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly 417untouched). They have been tested with the image available in 2018-03-13. 418 4191. Insert the SD card and open the ``boot`` partition. 420 4212. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore 422 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs, 423 even though the kernel is not compiled for AArch64. 424 4253. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore 426 bootloader will look for a file called ``armstub8.bin`` and load it at 427 address **0x0** instead of a predefined one. 428 4294. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add 430 ``console=serial0,115200 console=tty1``. 431 4325. Open ``config.txt`` and add the following lines at the end (``enable_uart=1`` 433 is only needed to enable debugging through the Mini UART): 434 435:: 436 437 enable_uart=1 438 kernel_address=0x02000000 439 device_tree_address=0x01000000 440 441If you connect a serial cable to the Mini UART and your computer, and connect 442to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some 443text. In the case of an AArch32 kernel, you should see something like this: 444 445:: 446 447 NOTICE: Booting Trusted Firmware 448 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty 449 NOTICE: BL1: Built : 00:09:25, Nov 6 2017 450 NOTICE: BL1: Booting BL2 451 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty 452 NOTICE: BL2: Built : 00:09:25, Nov 6 2017 453 NOTICE: BL1: Booting BL31 454 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty 455 NOTICE: BL31: Built : 00:09:25, Nov 6 2017 456 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517 457 458 Raspbian GNU/Linux 9 raspberrypi ttyS0 459 raspberrypi login: 460 461Just enter your credentials, everything should work as expected. Note that the 462HDMI output won't show any text during boot. 463 464.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S 465.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S 466.. _Linux kernel tree: https://github.com/torvalds/linux 467.. _Linux tree fork: https://github.com/raspberrypi/linux 468.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ 469.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap 470.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/ 471.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/ 472