1Trusted Firmware-A for Raspberry Pi 3 2===================================== 3 4.. section-numbering:: 5 :suffix: . 6 7.. contents:: 8 9The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four 10Arm Cortex-A53 cores, which makes it possible to have a port of Trusted 11Firmware-A (TF-A). 12 13The following instructions explain how to use this port of the TF-A with the 14default distribution of `Raspbian`_ because that's the distribution officially 15supported by the Raspberry Pi Foundation. At the moment of writing this, the 16officially supported kernel is a AArch32 kernel. This doesn't mean that this 17port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by 18the Foundation can be compiled for AArch64 by following the steps in 19`AArch64 kernel build instructions`_. 20 21**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM, 22which is available from both the Non-secure and Secure worlds. This port 23shouldn't be considered more than a prototype to play with and implement 24elements like PSCI to support the Linux kernel. 25 26Design 27------ 28 29The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a 30VideoCore IV that acts as primary processor (and loads everything from the SD 31card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi 323 documentation`_ for more information. 33 34This explains why it is possible to change the execution state (AArch64/AArch32) 35depending on a few files on the SD card. We only care about the cases in which 36the cores boot in AArch64 mode. 37 38The rules are simple: 39 40- If a file called ``kernel8.img`` is located on the ``boot`` partition of the 41 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes 42 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel. 43 44- If there is also a file called ``armstub8.bin``, it will load it at address 45 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All 46 the cores are powered on at the same time and start at address **0x0**. 47 48This means that we can use the default AArch32 kernel provided in the official 49`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and 50anything else we need is in ``armstub8.bin``. This way we can forget about the 51default bootstrap code. When using a AArch64 kernel, it is only needed to make 52sure that the name on the SD card is ``kernel8.img``. 53 54Ideally, we want to load the kernel and have all cores available, which means 55that we need to make the secondary cores work in the way the kernel expects, as 56explained in `Secondary cores`_. In practice, a small bootstrap is needed 57between TF-A and the kernel. 58 59To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode 60in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The 61architecture specifies that AArch32 Hypervisor mode isn't present when AArch64 62is used for EL2. When using a AArch64 kernel, it should simply start in EL2. 63 64Placement of images 65~~~~~~~~~~~~~~~~~~~ 66 67The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding 68between them so that the addresses they are loaded to match the ones specified 69when compiling TF-A. 70 71The device tree block is loaded by the VideoCore loader from an appropriate 72file, but we can specify the address it is loaded to in ``config.txt``. 73 74The file ``kernel8.img`` contains a kernel image that is loaded to the address 75specified in ``config.txt``. The `Linux kernel tree`_ has information about how 76a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``: 77 78:: 79 80 The zImage may also be placed in system RAM and called there. The 81 kernel should be placed in the first 128MiB of RAM. It is recommended 82 that it is loaded above 32MiB in order to avoid the need to relocate 83 prior to decompression, which will make the boot process slightly 84 faster. 85 86There are no similar restrictions for AArch64 kernels, as specified in the file 87``Documentation/arm64/booting.txt``. 88 89This means that we need to avoid the first 128 MiB of RAM when placing the 90TF-A images (and specially the first 32 MiB, as they are directly used to 91place the uncompressed AArch32 kernel image. This way, both AArch32 and 92AArch64 kernels can be placed at the same address. 93 94In the end, the images look like the following diagram when placed in memory. 95All addresses are Physical Addresses from the point of view of the Arm cores. 96Again, note that this is all just part of the same DRAM that goes from 97**0x00000000** to **0x3F000000**, it just has different names to simulate a real 98secure platform! 99 100:: 101 102 0x00000000 +-----------------+ 103 | ROM | BL1 104 0x00020000 +-----------------+ 105 | FIP | 106 0x00200000 +-----------------+ 107 | | 108 | ... | 109 | | 110 0x01000000 +-----------------+ 111 | Kernel | 112 +-----------------+ 113 | | 114 | ... | 115 | | 116 0x02000000 +-----------------+ 117 | DTB | 118 +-----------------+ 119 | | 120 | ... | 121 | | 122 0x10000000 +-----------------+ 123 | Secure SRAM | BL2, BL31 124 0x10100000 +-----------------+ 125 | Secure DRAM | BL32 (Secure payload) 126 0x10C00000 +-----------------+ 127 | Non-secure DRAM | BL33 128 0x11000000 +-----------------+ 129 | | 130 | ... | 131 | | 132 0x3F000000 +-----------------+ 133 | I/O | 134 0x40000000 +-----------------+ 135 136The area between **0x10000000** and **0x11000000** has to be protected so that 137the kernel doesn't use it. That is done by adding ``memmap=16M$256M`` to the 138command line passed to the kernel. See the `Setup SD card`_ instructions to see 139how to do it. 140 141The last 16 MiB of DRAM can only be accessed by the VideoCore, that has 142different mappings than the Arm cores in which the I/O addresses don't overlap 143the DRAM. The memory reserved to be used by the VideoCore is always placed at 144the end of the DRAM, so this space isn't wasted. 145 146Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for 147TF-A, there are 880 MiB available for Linux. 148 149Boot sequence 150~~~~~~~~~~~~~ 151 152The boot sequence of TF-A is the usual one except when booting an AArch32 153kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it 154can jump to the kernel in the same mode and let it take over that privilege 155level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of 156TF-A) it could only jump to the kernel in AArch32 in Supervisor mode. 157 158The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel 159in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The 160bootstrap should take care of this. 161 162Secondary cores 163~~~~~~~~~~~~~~~ 164 165The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to 166use mailboxes to trap the secondary cores until they are ready to jump to the 167kernel. This mailbox is located at a different address in the AArch32 default 168kernel than in the AArch64 kernel. 169 170Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During 171cold boot, all secondary cores wait in a loop until they are given given an 172address to jump to in this Mailbox (``bl31_warm_entrypoint``). 173 174Once BL31 has finished and the primary core has jumped to the BL33 payload, it 175has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop. 176The payload then makes them wait in another waitloop listening from messages 177from the kernel. When the primary CPU jumps into the kernel, it will send an 178address to the mailbox so that the secondary CPUs jump to it and are recognised 179by the kernel. 180 181Build Instructions 182------------------ 183 184To boot a AArch64 kernel, only the AArch64 toolchain is required. 185 186To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The 187AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit 188kernel. 189 190First, clone and compile `Raspberry Pi 3 TF-A bootstrap`_. Choose the one 191needed for the architecture of your kernel. 192 193Then compile TF-A. For a AArch32 kernel, use the following command line: 194 195.. code:: shell 196 197 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 198 RPI3_BL33_IN_AARCH32=1 \ 199 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin \ 200 all fip 201 202For a AArch64 kernel, use this other command line: 203 204.. code:: shell 205 206 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 207 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin \ 208 all fip 209 210Then, join BL1 and the FIP with the following instructions (replace ``release`` 211by ``debug`` if you set the build option ``DEBUG=1``): 212 213.. code:: shell 214 215 cp build/rpi3/release/bl1.bin bl1.pad.bin 216 truncate --size=131072 bl1.pad.bin 217 cat bl1.pad.bin build/rpi3/release/fip.bin > armstub8.bin 218 219The resulting file, ``armstub8.bin``, contains BL1 and the FIP in the place they 220need to be for TF-A to boot correctly. Now, follow the instructions in 221`Setup SD card`_. 222 223The following build options are supported: 224 225- ``PRELOADED_BL33_BASE``: Specially useful because the file ``kernel8.img`` can 226 be loaded anywhere by modifying the file ``config.txt``. It doesn't have to 227 contain a kernel, it could have any arbitrary payload. 228 229- ``RESET_TO_BL31``: Set to 1 by default. If using a 32-bit kernel like 230 `Raspbian`_, the space used by BL1 can overwritten by the kernel when it is 231 being loaded. Even when using a AArch64 kernel the region used by 232 BL1 isn't protected and the kernel could overwrite it. The space used by BL31 233 is reserved by the command line passed to the kernel. 234 235- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image. 236 By default this option is 0, which means that TF-A will jump to BL33 in EL2 237 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32 238 mode. 239 240- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional. 241 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__. 242 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin 243 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin`` 244 to put the binaries into the FIP. 245 246 Note: If OP-TEE is used it may be needed to add the following options to the 247 Linux command line so that the USB driver doesn't use FIQs: 248 ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``. 249 This will unfortunately reduce the performance of the USB driver. It is needed 250 when using Raspbian, for example. 251 252- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option 253 ``TRUSTED_BOARD_BOOT=1`` to enable it. In order to use TBB, you might 254 want to set ``GENERATE_COT=1`` to let the contents of the FIP automatically 255 signed by the build process. The ROT key will be generated and output to 256 ``rot_key.pem`` in the build directory. It is able to set ROT_KEY to 257 your own key in PEM format. 258 Also in order to build, you need to clone mbedtls from 259 `here <https://github.com/ARMmbed/mbedtls>`__. 260 And set MBEDTLS_DIR to mbedtls source directory. 261 262The following is not currently supported: 263 264- AArch32 for TF-A itself. 265 266- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any 267 address by changing the file ``armstub8.bin``, so there's no point in using 268 TF-A in this case. 269 270- ``LOAD_IMAGE_V2=0``: Only version 2 is supported. 271 272- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the 273 crash console uses the internal 16550 driver functions directly in order to be 274 able to print error messages during early crashes before setting up the 275 multi console API. 276 277AArch64 kernel build instructions 278--------------------------------- 279 280The following instructions show how to install and run a AArch64 kernel by 281using a SD card with the default `Raspbian`_ install as base. Skip them if you 282want to use the default 32-bit kernel. 283 284Note that this system won't be fully 64-bit because all the tools in the 285filesystem are 32-bit binaries, but it's a quick way to get it working, and it 286allows the user to run 64-bit binaries in addition to 32-bit binaries. 287 2881. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To 289 speed things up, do a shallow clone of the desired branch. 290 291.. code:: shell 292 293 git clone --depth=1 -b rpi-4.14.y https://github.com/raspberrypi/linux 294 cd linux 295 2962. Configure and compile the kernel. Adapt the number after ``-j`` so that it is 297 1.5 times the number of CPUs in your computer. This may take some time to 298 finish. 299 300.. code:: shell 301 302 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig 303 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 304 3053. Copy the kernel image and the device tree to the SD card. Replace the path 306 by the corresponding path in your computers to the ``boot`` partition of the 307 SD card. 308 309.. code:: shell 310 311 cp arch/arm64/boot/Image /path/to/boot/kernel8.img 312 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/ 313 3144. Install the kernel modules. Replace the path by the corresponding path to the 315 filesystem partition of the SD card on your computer. 316 317.. code:: shell 318 319 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ 320 INSTALL_MOD_PATH=/path/to/filesystem modules_install 321 3225. Follow the instructions in `Setup SD card`_ except for the step of renaming 323 the existing ``kernel7.img`` (we have already copied a AArch64 kernel). 324 325Setup SD card 326------------- 327 328The instructions assume that you have an SD card with a fresh install of 329`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly 330untouched). They have been tested with the image available in 2018-03-13. 331 3321. Insert the SD card and open the ``boot`` partition. 333 3342. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore 335 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs, 336 even though the kernel is not compiled for AArch64. 337 3383. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore 339 bootloader will look for a file called ``armstub8.bin`` and load it at 340 address **0x0** instead of a predefined one. 341 3424. Open ``cmdline.txt`` and add ``memmap=16M$256M`` to prevent the kernel from 343 using the memory needed by TF-A. If you want to enable the serial port 344 "Mini UART", make sure that this file also contains 345 ``console=serial0,115200 console=tty1``. 346 347 Note that the 16 MiB reserved this way won't be available for Linux, the same 348 way as the memory reserved in DRAM for the GPU isn't available. 349 3505. Open ``config.txt`` and add the following lines at the end (``enable_uart=1`` 351 is only needed to enable debugging through the Mini UART): 352 353:: 354 355 enable_uart=1 356 kernel_address=0x01000000 357 device_tree_address=0x02000000 358 359If you connect a serial cable to the Mini UART and your computer, and connect 360to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some 361text. In the case of an AArch32 kernel, you should see something like this: 362 363:: 364 365 NOTICE: Booting Trusted Firmware 366 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty 367 NOTICE: BL1: Built : 00:09:25, Nov 6 2017 368 NOTICE: BL1: Booting BL2 369 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty 370 NOTICE: BL2: Built : 00:09:25, Nov 6 2017 371 NOTICE: BL1: Booting BL31 372 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty 373 NOTICE: BL31: Built : 00:09:25, Nov 6 2017 374 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517 375 376 Raspbian GNU/Linux 9 raspberrypi ttyS0 377 raspberrypi login: 378 379Just enter your credentials, everything should work as expected. Note that the 380HDMI output won't show any text during boot. 381 382.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S 383.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S 384.. _Linux kernel tree: https://github.com/torvalds/linux 385.. _Linux tree fork: https://github.com/raspberrypi/linux 386.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ 387.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap 388.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/ 389.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/ 390