1Raspberry Pi 3 2============== 3 4.. contents:: 5 6The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four 7Arm Cortex-A53 cores. 8 9The following instructions explain how to use this port of the TF-A with the 10default distribution of `Raspbian`_ because that's the distribution officially 11supported by the Raspberry Pi Foundation. At the moment of writing this, the 12officially supported kernel is a AArch32 kernel. This doesn't mean that this 13port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by 14the Foundation can be compiled for AArch64 by following the steps in 15`AArch64 kernel build instructions`_. 16 17**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM, 18which is available from both the Non-secure and Secure worlds. This port 19shouldn't be considered more than a prototype to play with and implement 20elements like PSCI to support the Linux kernel. 21 22Design 23------ 24 25The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a 26VideoCore IV that acts as primary processor (and loads everything from the SD 27card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi 283 documentation`_ for more information. 29 30This explains why it is possible to change the execution state (AArch64/AArch32) 31depending on a few files on the SD card. We only care about the cases in which 32the cores boot in AArch64 mode. 33 34The rules are simple: 35 36- If a file called ``kernel8.img`` is located on the ``boot`` partition of the 37 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes 38 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel. 39 40- If there is also a file called ``armstub8.bin``, it will load it at address 41 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All 42 the cores are powered on at the same time and start at address **0x0**. 43 44This means that we can use the default AArch32 kernel provided in the official 45`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and 46anything else we need is in ``armstub8.bin``. This way we can forget about the 47default bootstrap code. When using a AArch64 kernel, it is only needed to make 48sure that the name on the SD card is ``kernel8.img``. 49 50Ideally, we want to load the kernel and have all cores available, which means 51that we need to make the secondary cores work in the way the kernel expects, as 52explained in `Secondary cores`_. In practice, a small bootstrap is needed 53between TF-A and the kernel. 54 55To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode 56in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The 57architecture specifies that AArch32 Hypervisor mode isn't present when AArch64 58is used for EL2. When using a AArch64 kernel, it should simply start in EL2. 59 60Placement of images 61~~~~~~~~~~~~~~~~~~~ 62 63The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding 64between them so that the addresses they are loaded to match the ones specified 65when compiling TF-A. This is done automatically by the build system. 66 67The device tree block is loaded by the VideoCore loader from an appropriate 68file, but we can specify the address it is loaded to in ``config.txt``. 69 70The file ``kernel8.img`` contains a kernel image that is loaded to the address 71specified in ``config.txt``. The `Linux kernel tree`_ has information about how 72a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``: 73 74:: 75 76 The zImage may also be placed in system RAM and called there. The 77 kernel should be placed in the first 128MiB of RAM. It is recommended 78 that it is loaded above 32MiB in order to avoid the need to relocate 79 prior to decompression, which will make the boot process slightly 80 faster. 81 82There are no similar restrictions for AArch64 kernels, as specified in the file 83``Documentation/arm64/booting.txt``. 84 85This means that we need to avoid the first 128 MiB of RAM when placing the 86TF-A images (and specially the first 32 MiB, as they are directly used to 87place the uncompressed AArch32 kernel image. This way, both AArch32 and 88AArch64 kernels can be placed at the same address. 89 90In the end, the images look like the following diagram when placed in memory. 91All addresses are Physical Addresses from the point of view of the Arm cores. 92Again, note that this is all just part of the same DRAM that goes from 93**0x00000000** to **0x3F000000**, it just has different names to simulate a real 94secure platform! 95 96:: 97 98 0x00000000 +-----------------+ 99 | ROM | BL1 100 0x00020000 +-----------------+ 101 | FIP | 102 0x00200000 +-----------------+ 103 | | 104 | ... | 105 | | 106 0x01000000 +-----------------+ 107 | DTB | (Loaded by the VideoCore) 108 +-----------------+ 109 | | 110 | ... | 111 | | 112 0x02000000 +-----------------+ 113 | Kernel | (Loaded by the VideoCore) 114 +-----------------+ 115 | | 116 | ... | 117 | | 118 0x10000000 +-----------------+ 119 | Secure SRAM | BL2, BL31 120 0x10100000 +-----------------+ 121 | Secure DRAM | BL32 (Secure payload) 122 0x11000000 +-----------------+ 123 | Non-secure DRAM | BL33 124 +-----------------+ 125 | | 126 | ... | 127 | | 128 0x3F000000 +-----------------+ 129 | I/O | 130 0x40000000 +-----------------+ 131 132The area between **0x10000000** and **0x11000000** has to be manually protected 133so that the kernel doesn't use it. The current port tries to modify the live DTB 134to add a memreserve region that reserves the previously mentioned area. 135 136If this is not possible, the user may manually add ``memmap=16M$256M`` to the 137command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_ 138instructions to see how to do it. This system is strongly discouraged. 139 140The last 16 MiB of DRAM can only be accessed by the VideoCore, that has 141different mappings than the Arm cores in which the I/O addresses don't overlap 142the DRAM. The memory reserved to be used by the VideoCore is always placed at 143the end of the DRAM, so this space isn't wasted. 144 145Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for 146TF-A, there are 880 MiB available for Linux. 147 148Boot sequence 149~~~~~~~~~~~~~ 150 151The boot sequence of TF-A is the usual one except when booting an AArch32 152kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it 153can jump to the kernel in the same mode and let it take over that privilege 154level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of 155TF-A) it could only jump to the kernel in AArch32 in Supervisor mode. 156 157The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel 158in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The 159bootstrap should take care of this. 160 161This port support a direct boot of the Linux kernel from the firmware (as a BL33 162image). Alternatively, U-Boot or other bootloaders may be used. 163 164Secondary cores 165~~~~~~~~~~~~~~~ 166 167This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``, 168``PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn 169the system off, it simply reboots it and asks the VideoCore firmware to keep it 170in a low power mode permanently. 171 172The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to 173use mailboxes to trap the secondary cores until they are ready to jump to the 174kernel. This mailbox is located at a different address in the AArch32 default 175kernel than in the AArch64 kernel. 176 177Kernels with PSCI support can use the PSCI calls instead for a cleaner boot. 178 179Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During 180cold boot, all secondary cores wait in a loop until they are given given an 181address to jump to in this Mailbox (``bl31_warm_entrypoint``). 182 183Once BL31 has finished and the primary core has jumped to the BL33 payload, it 184has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop. 185The payload then makes them wait in another waitloop listening from messages 186from the kernel. When the primary CPU jumps into the kernel, it will send an 187address to the mailbox so that the secondary CPUs jump to it and are recognised 188by the kernel. 189 190Build Instructions 191------------------ 192 193To boot a AArch64 kernel, only the AArch64 toolchain is required. 194 195To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The 196AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit 197kernel. 198 199The build system concatenates BL1 and the FIP so that the addresses match the 200ones in the memory map. The resulting file is ``armstub8.bin``, located in the 201build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this 202file, follow the instructions in `Setup SD card`_. 203 204The following build options are supported: 205 206- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image. 207 By default this option is 0, which means that TF-A will jump to BL33 in EL2 208 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32 209 mode. 210 211- ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has 212 been preloaded by any other system than using the firmware. ``BL33`` isn't 213 needed in the build command line if this option is used. Specially useful 214 because the file ``kernel8.img`` can be loaded anywhere by modifying the file 215 ``config.txt``. It doesn't have to contain a kernel, it could have any 216 arbitrary payload. 217 218- ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct 219 boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE`` 220 is mandatory when the direct Linux kernel boot is used. Options 221 ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is 222 unlikely that the kernel image will fit in the space reserved for BL33 images. 223 This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a 224 32-bit kernel. The only thing this option does is to set the arguments in 225 registers x0-x3 or r0-r2 as expected by the kernel. 226 227- ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using 228 ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a 229 DTB in memory. 230 231- ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime 232 or disabled. ``-1`` (default) disables the runtime UART. Any other value 233 enables the default UART (currently UART1) for runtime messages. 234 235- ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory 236 mapping required for an UEFI firmware payload. These changes are needed 237 to be able to run Windows on ARM64. This option, which is disabled by 238 default, results in the following memory mappings: 239 240:: 241 242 0x00000000 +-----------------+ 243 | ROM | BL1 244 0x00010000 +-----------------+ 245 | DTB | (Loaded by the VideoCore) 246 0x00020000 +-----------------+ 247 | FIP | 248 0x00030000 +-----------------+ 249 | | 250 | UEFI PAYLOAD | 251 | | 252 0x00200000 +-----------------+ 253 | Secure SRAM | BL2, BL31 254 0x00300000 +-----------------+ 255 | Secure DRAM | BL32 (Secure payload) 256 0x00400000 +-----------------+ 257 | | 258 | | 259 | Non-secure DRAM | BL33 260 | | 261 | | 262 0x01000000 +-----------------+ 263 | | 264 | ... | 265 | | 266 0x3F000000 +-----------------+ 267 | I/O | 268 269- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional. 270 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__. 271 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin 272 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin`` 273 to put the binaries into the FIP. 274 275 Note: If OP-TEE is used it may be needed to add the following options to the 276 Linux command line so that the USB driver doesn't use FIQs: 277 ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``. 278 This will unfortunately reduce the performance of the USB driver. It is needed 279 when using Raspbian, for example. 280 281- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable 282 it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the 283 contents of the FIP automatically signed by the build process. The ROT key 284 will be generated and output to ``rot_key.pem`` in the build directory. It is 285 able to set ROT_KEY to your own key in PEM format. Also in order to build, 286 you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__. 287 ``MBEDTLS_DIR`` must point at the mbed TLS source directory. 288 289- ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of 290 the board. 291 292The following is not currently supported: 293 294- AArch32 for TF-A itself. 295 296- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any 297 address by changing the file ``armstub8.bin``, so there's no point in using 298 TF-A in this case. 299 300- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the 301 crash console uses the internal 16550 driver functions directly in order to be 302 able to print error messages during early crashes before setting up the 303 multi console API. 304 305Building the firmware for kernels that don't support PSCI 306~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 307 308This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels 309always support PSCI, but they may not know that the system understands PSCI due 310to an incorrect DTB file. 311 312First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A 313bootstrap`_. Choose the one needed for the architecture of your kernel. 314 315Then compile TF-A. For a 32-bit kernel, use the following command line: 316 317.. code:: shell 318 319 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 320 RPI3_BL33_IN_AARCH32=1 \ 321 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin 322 323For a 64-bit kernel, use this other command line: 324 325.. code:: shell 326 327 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 328 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin 329 330However, enabling PSCI support in a 64-bit kernel is really easy. In the 331repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied 332to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes 333the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the 334instructions in `AArch64 kernel build instructions`_ to get a working 64-bit 335kernel image and supporting files. 336 337Building the firmware for kernels that support PSCI 338~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 339 340For a 64-bit kernel: 341 342.. code:: shell 343 344 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 345 PRELOADED_BL33_BASE=0x02000000 \ 346 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 347 RPI3_DIRECT_LINUX_BOOT=1 348 349For a 32-bit kernel: 350 351.. code:: shell 352 353 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 354 PRELOADED_BL33_BASE=0x02000000 \ 355 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 356 RPI3_DIRECT_LINUX_BOOT=1 \ 357 RPI3_BL33_IN_AARCH32=1 358 359AArch64 kernel build instructions 360--------------------------------- 361 362The following instructions show how to install and run a AArch64 kernel by 363using a SD card with the default `Raspbian`_ install as base. Skip them if you 364want to use the default 32-bit kernel. 365 366Note that this system won't be fully 64-bit because all the tools in the 367filesystem are 32-bit binaries, but it's a quick way to get it working, and it 368allows the user to run 64-bit binaries in addition to 32-bit binaries. 369 3701. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To 371 speed things up, do a shallow clone of the desired branch. 372 373.. code:: shell 374 375 git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux 376 cd linux 377 3782. Configure and compile the kernel. Adapt the number after ``-j`` so that it is 379 1.5 times the number of CPUs in your computer. This may take some time to 380 finish. 381 382.. code:: shell 383 384 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig 385 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 386 3873. Copy the kernel image and the device tree to the SD card. Replace the path 388 by the corresponding path in your computers to the ``boot`` partition of the 389 SD card. 390 391.. code:: shell 392 393 cp arch/arm64/boot/Image /path/to/boot/kernel8.img 394 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/ 395 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/ 396 3974. Install the kernel modules. Replace the path by the corresponding path to the 398 filesystem partition of the SD card on your computer. 399 400.. code:: shell 401 402 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ 403 INSTALL_MOD_PATH=/path/to/filesystem modules_install 404 4055. Follow the instructions in `Setup SD card`_ except for the step of renaming 406 the existing ``kernel7.img`` (we have already copied a AArch64 kernel). 407 408Setup SD card 409------------- 410 411The instructions assume that you have an SD card with a fresh install of 412`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly 413untouched). They have been tested with the image available in 2018-03-13. 414 4151. Insert the SD card and open the ``boot`` partition. 416 4172. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore 418 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs, 419 even though the kernel is not compiled for AArch64. 420 4213. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore 422 bootloader will look for a file called ``armstub8.bin`` and load it at 423 address **0x0** instead of a predefined one. 424 4254. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add 426 ``console=serial0,115200 console=tty1``. 427 4285. Open ``config.txt`` and add the following lines at the end (``enable_uart=1`` 429 is only needed to enable debugging through the Mini UART): 430 431:: 432 433 enable_uart=1 434 kernel_address=0x02000000 435 device_tree_address=0x01000000 436 437If you connect a serial cable to the Mini UART and your computer, and connect 438to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some 439text. In the case of an AArch32 kernel, you should see something like this: 440 441:: 442 443 NOTICE: Booting Trusted Firmware 444 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty 445 NOTICE: BL1: Built : 00:09:25, Nov 6 2017 446 NOTICE: BL1: Booting BL2 447 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty 448 NOTICE: BL2: Built : 00:09:25, Nov 6 2017 449 NOTICE: BL1: Booting BL31 450 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty 451 NOTICE: BL31: Built : 00:09:25, Nov 6 2017 452 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517 453 454 Raspbian GNU/Linux 9 raspberrypi ttyS0 455 raspberrypi login: 456 457Just enter your credentials, everything should work as expected. Note that the 458HDMI output won't show any text during boot. 459 460.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S 461.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S 462.. _Linux kernel tree: https://github.com/torvalds/linux 463.. _Linux tree fork: https://github.com/raspberrypi/linux 464.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ 465.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap 466.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/ 467.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/ 468