1Trusted Firmware-A for Raspberry Pi 3 2===================================== 3 4 5 6.. contents:: 7 8The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four 9Arm Cortex-A53 cores. 10 11The following instructions explain how to use this port of the TF-A with the 12default distribution of `Raspbian`_ because that's the distribution officially 13supported by the Raspberry Pi Foundation. At the moment of writing this, the 14officially supported kernel is a AArch32 kernel. This doesn't mean that this 15port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by 16the Foundation can be compiled for AArch64 by following the steps in 17`AArch64 kernel build instructions`_. 18 19**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM, 20which is available from both the Non-secure and Secure worlds. This port 21shouldn't be considered more than a prototype to play with and implement 22elements like PSCI to support the Linux kernel. 23 24Design 25------ 26 27The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a 28VideoCore IV that acts as primary processor (and loads everything from the SD 29card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi 303 documentation`_ for more information. 31 32This explains why it is possible to change the execution state (AArch64/AArch32) 33depending on a few files on the SD card. We only care about the cases in which 34the cores boot in AArch64 mode. 35 36The rules are simple: 37 38- If a file called ``kernel8.img`` is located on the ``boot`` partition of the 39 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes 40 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel. 41 42- If there is also a file called ``armstub8.bin``, it will load it at address 43 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All 44 the cores are powered on at the same time and start at address **0x0**. 45 46This means that we can use the default AArch32 kernel provided in the official 47`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and 48anything else we need is in ``armstub8.bin``. This way we can forget about the 49default bootstrap code. When using a AArch64 kernel, it is only needed to make 50sure that the name on the SD card is ``kernel8.img``. 51 52Ideally, we want to load the kernel and have all cores available, which means 53that we need to make the secondary cores work in the way the kernel expects, as 54explained in `Secondary cores`_. In practice, a small bootstrap is needed 55between TF-A and the kernel. 56 57To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode 58in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The 59architecture specifies that AArch32 Hypervisor mode isn't present when AArch64 60is used for EL2. When using a AArch64 kernel, it should simply start in EL2. 61 62Placement of images 63~~~~~~~~~~~~~~~~~~~ 64 65The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding 66between them so that the addresses they are loaded to match the ones specified 67when compiling TF-A. This is done automatically by the build system. 68 69The device tree block is loaded by the VideoCore loader from an appropriate 70file, but we can specify the address it is loaded to in ``config.txt``. 71 72The file ``kernel8.img`` contains a kernel image that is loaded to the address 73specified in ``config.txt``. The `Linux kernel tree`_ has information about how 74a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``: 75 76:: 77 78 The zImage may also be placed in system RAM and called there. The 79 kernel should be placed in the first 128MiB of RAM. It is recommended 80 that it is loaded above 32MiB in order to avoid the need to relocate 81 prior to decompression, which will make the boot process slightly 82 faster. 83 84There are no similar restrictions for AArch64 kernels, as specified in the file 85``Documentation/arm64/booting.txt``. 86 87This means that we need to avoid the first 128 MiB of RAM when placing the 88TF-A images (and specially the first 32 MiB, as they are directly used to 89place the uncompressed AArch32 kernel image. This way, both AArch32 and 90AArch64 kernels can be placed at the same address. 91 92In the end, the images look like the following diagram when placed in memory. 93All addresses are Physical Addresses from the point of view of the Arm cores. 94Again, note that this is all just part of the same DRAM that goes from 95**0x00000000** to **0x3F000000**, it just has different names to simulate a real 96secure platform! 97 98:: 99 100 0x00000000 +-----------------+ 101 | ROM | BL1 102 0x00020000 +-----------------+ 103 | FIP | 104 0x00200000 +-----------------+ 105 | | 106 | ... | 107 | | 108 0x01000000 +-----------------+ 109 | DTB | (Loaded by the VideoCore) 110 +-----------------+ 111 | | 112 | ... | 113 | | 114 0x02000000 +-----------------+ 115 | Kernel | (Loaded by the VideoCore) 116 +-----------------+ 117 | | 118 | ... | 119 | | 120 0x10000000 +-----------------+ 121 | Secure SRAM | BL2, BL31 122 0x10100000 +-----------------+ 123 | Secure DRAM | BL32 (Secure payload) 124 0x11000000 +-----------------+ 125 | Non-secure DRAM | BL33 126 +-----------------+ 127 | | 128 | ... | 129 | | 130 0x3F000000 +-----------------+ 131 | I/O | 132 0x40000000 +-----------------+ 133 134The area between **0x10000000** and **0x11000000** has to be manually protected 135so that the kernel doesn't use it. The current port tries to modify the live DTB 136to add a memreserve region that reserves the previously mentioned area. 137 138If this is not possible, the user may manually add ``memmap=16M$256M`` to the 139command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_ 140instructions to see how to do it. This system is strongly discouraged. 141 142The last 16 MiB of DRAM can only be accessed by the VideoCore, that has 143different mappings than the Arm cores in which the I/O addresses don't overlap 144the DRAM. The memory reserved to be used by the VideoCore is always placed at 145the end of the DRAM, so this space isn't wasted. 146 147Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for 148TF-A, there are 880 MiB available for Linux. 149 150Boot sequence 151~~~~~~~~~~~~~ 152 153The boot sequence of TF-A is the usual one except when booting an AArch32 154kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it 155can jump to the kernel in the same mode and let it take over that privilege 156level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of 157TF-A) it could only jump to the kernel in AArch32 in Supervisor mode. 158 159The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel 160in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The 161bootstrap should take care of this. 162 163This port support a direct boot of the Linux kernel from the firmware (as a BL33 164image). Alternatively, U-Boot or other bootloaders may be used. 165 166Secondary cores 167~~~~~~~~~~~~~~~ 168 169This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``, 170`PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn 171the system off, it simply reboots it and asks the VideoCore firmware to keep it 172in a low power mode permanently. 173 174The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to 175use mailboxes to trap the secondary cores until they are ready to jump to the 176kernel. This mailbox is located at a different address in the AArch32 default 177kernel than in the AArch64 kernel. 178 179Kernels with PSCI support can use the PSCI calls instead for a cleaner boot. 180 181Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During 182cold boot, all secondary cores wait in a loop until they are given given an 183address to jump to in this Mailbox (``bl31_warm_entrypoint``). 184 185Once BL31 has finished and the primary core has jumped to the BL33 payload, it 186has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop. 187The payload then makes them wait in another waitloop listening from messages 188from the kernel. When the primary CPU jumps into the kernel, it will send an 189address to the mailbox so that the secondary CPUs jump to it and are recognised 190by the kernel. 191 192Build Instructions 193------------------ 194 195To boot a AArch64 kernel, only the AArch64 toolchain is required. 196 197To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The 198AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit 199kernel. 200 201The build system concatenates BL1 and the FIP so that the addresses match the 202ones in the memory map. The resulting file is ``armstub8.bin``, located in the 203build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this 204file, follow the instructions in `Setup SD card`_. 205 206The following build options are supported: 207 208- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image. 209 By default this option is 0, which means that TF-A will jump to BL33 in EL2 210 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32 211 mode. 212 213- ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has 214 been preloaded by any other system than using the firmware. ``BL33`` isn't 215 needed in the build command line if this option is used. Specially useful 216 because the file ``kernel8.img`` can be loaded anywhere by modifying the file 217 ``config.txt``. It doesn't have to contain a kernel, it could have any 218 arbitrary payload. 219 220- ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct 221 boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE`` 222 is mandatory when the direct Linux kernel boot is used. Options 223 ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is 224 unlikely that the kernel image will fit in the space reserved for BL33 images. 225 This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a 226 32-bit kernel. The only thing this option does is to set the arguments in 227 registers x0-x3 or r0-r2 as expected by the kernel. 228 229- ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using 230 ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a 231 DTB in memory. 232 233- ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime 234 or disabled. ``-1`` (default) disables the runtime UART. Any other value 235 enables the default UART (currently UART1) for runtime messages. 236 237- ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory 238 mapping required for an UEFI firmware payload. These changes are needed 239 to be able to run Windows on ARM64. This option, which is disabled by 240 default, results in the following memory mappings: 241 242:: 243 244 0x00000000 +-----------------+ 245 | ROM | BL1 246 0x00010000 +-----------------+ 247 | DTB | (Loaded by the VideoCore) 248 0x00020000 +-----------------+ 249 | FIP | 250 0x00030000 +-----------------+ 251 | | 252 | UEFI PAYLOAD | 253 | | 254 0x00200000 +-----------------+ 255 | Secure SRAM | BL2, BL31 256 0x00300000 +-----------------+ 257 | Secure DRAM | BL32 (Secure payload) 258 0x00400000 +-----------------+ 259 | | 260 | | 261 | Non-secure DRAM | BL33 262 | | 263 | | 264 0x01000000 +-----------------+ 265 | | 266 | ... | 267 | | 268 0x3F000000 +-----------------+ 269 | I/O | 270 271- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional. 272 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__. 273 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin 274 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin`` 275 to put the binaries into the FIP. 276 277 Note: If OP-TEE is used it may be needed to add the following options to the 278 Linux command line so that the USB driver doesn't use FIQs: 279 ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``. 280 This will unfortunately reduce the performance of the USB driver. It is needed 281 when using Raspbian, for example. 282 283- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable 284 it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the 285 contents of the FIP automatically signed by the build process. The ROT key 286 will be generated and output to ``rot_key.pem`` in the build directory. It is 287 able to set ROT_KEY to your own key in PEM format. Also in order to build, 288 you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__. 289 ``MBEDTLS_DIR`` must point at the mbed TLS source directory. 290 291- ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of 292 the board. 293 294The following is not currently supported: 295 296- AArch32 for TF-A itself. 297 298- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any 299 address by changing the file ``armstub8.bin``, so there's no point in using 300 TF-A in this case. 301 302- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the 303 crash console uses the internal 16550 driver functions directly in order to be 304 able to print error messages during early crashes before setting up the 305 multi console API. 306 307Building the firmware for kernels that don't support PSCI 308~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 309 310This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels 311always support PSCI, but they may not know that the system understands PSCI due 312to an incorrect DTB file. 313 314First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A 315bootstrap`_. Choose the one needed for the architecture of your kernel. 316 317Then compile TF-A. For a 32-bit kernel, use the following command line: 318 319.. code:: shell 320 321 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 322 RPI3_BL33_IN_AARCH32=1 \ 323 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin 324 325For a 64-bit kernel, use this other command line: 326 327.. code:: shell 328 329 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 330 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin 331 332However, enabling PSCI support in a 64-bit kernel is really easy. In the 333repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied 334to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes 335the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the 336instructions in `AArch64 kernel build instructions`_ to get a working 64-bit 337kernel image and supporting files. 338 339Building the firmware for kernels that support PSCI 340~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 341 342For a 64-bit kernel: 343 344.. code:: shell 345 346 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 347 PRELOADED_BL33_BASE=0x02000000 \ 348 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 349 RPI3_DIRECT_LINUX_BOOT=1 350 351For a 32-bit kernel: 352 353.. code:: shell 354 355 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 356 PRELOADED_BL33_BASE=0x02000000 \ 357 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 358 RPI3_DIRECT_LINUX_BOOT=1 \ 359 RPI3_BL33_IN_AARCH32=1 360 361AArch64 kernel build instructions 362--------------------------------- 363 364The following instructions show how to install and run a AArch64 kernel by 365using a SD card with the default `Raspbian`_ install as base. Skip them if you 366want to use the default 32-bit kernel. 367 368Note that this system won't be fully 64-bit because all the tools in the 369filesystem are 32-bit binaries, but it's a quick way to get it working, and it 370allows the user to run 64-bit binaries in addition to 32-bit binaries. 371 3721. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To 373 speed things up, do a shallow clone of the desired branch. 374 375.. code:: shell 376 377 git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux 378 cd linux 379 3802. Configure and compile the kernel. Adapt the number after ``-j`` so that it is 381 1.5 times the number of CPUs in your computer. This may take some time to 382 finish. 383 384.. code:: shell 385 386 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig 387 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 388 3893. Copy the kernel image and the device tree to the SD card. Replace the path 390 by the corresponding path in your computers to the ``boot`` partition of the 391 SD card. 392 393.. code:: shell 394 395 cp arch/arm64/boot/Image /path/to/boot/kernel8.img 396 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/ 397 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/ 398 3994. Install the kernel modules. Replace the path by the corresponding path to the 400 filesystem partition of the SD card on your computer. 401 402.. code:: shell 403 404 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ 405 INSTALL_MOD_PATH=/path/to/filesystem modules_install 406 4075. Follow the instructions in `Setup SD card`_ except for the step of renaming 408 the existing ``kernel7.img`` (we have already copied a AArch64 kernel). 409 410Setup SD card 411------------- 412 413The instructions assume that you have an SD card with a fresh install of 414`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly 415untouched). They have been tested with the image available in 2018-03-13. 416 4171. Insert the SD card and open the ``boot`` partition. 418 4192. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore 420 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs, 421 even though the kernel is not compiled for AArch64. 422 4233. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore 424 bootloader will look for a file called ``armstub8.bin`` and load it at 425 address **0x0** instead of a predefined one. 426 4274. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add 428 ``console=serial0,115200 console=tty1``. 429 4305. Open ``config.txt`` and add the following lines at the end (``enable_uart=1`` 431 is only needed to enable debugging through the Mini UART): 432 433:: 434 435 enable_uart=1 436 kernel_address=0x02000000 437 device_tree_address=0x01000000 438 439If you connect a serial cable to the Mini UART and your computer, and connect 440to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some 441text. In the case of an AArch32 kernel, you should see something like this: 442 443:: 444 445 NOTICE: Booting Trusted Firmware 446 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty 447 NOTICE: BL1: Built : 00:09:25, Nov 6 2017 448 NOTICE: BL1: Booting BL2 449 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty 450 NOTICE: BL2: Built : 00:09:25, Nov 6 2017 451 NOTICE: BL1: Booting BL31 452 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty 453 NOTICE: BL31: Built : 00:09:25, Nov 6 2017 454 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517 455 456 Raspbian GNU/Linux 9 raspberrypi ttyS0 457 raspberrypi login: 458 459Just enter your credentials, everything should work as expected. Note that the 460HDMI output won't show any text during boot. 461 462.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S 463.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S 464.. _Linux kernel tree: https://github.com/torvalds/linux 465.. _Linux tree fork: https://github.com/raspberrypi/linux 466.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ 467.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap 468.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/ 469.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/ 470