1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3************ 4Common Tasks 5************ 6 7This chapter describes fundamental procedures such as creating layers, 8adding new software packages, extending or customizing images, porting 9work to new hardware (adding a new machine), and so forth. You will find 10that the procedures documented here occur often in the development cycle 11using the Yocto Project. 12 13Understanding and Creating Layers 14================================= 15 16The OpenEmbedded build system supports organizing 17:term:`Metadata` into multiple layers. 18Layers allow you to isolate different types of customizations from each 19other. For introductory information on the Yocto Project Layer Model, 20see the 21":ref:`overview-manual/yp-intro:the yocto project layer model`" 22section in the Yocto Project Overview and Concepts Manual. 23 24Creating Your Own Layer 25----------------------- 26 27.. note:: 28 29 It is very easy to create your own layers to use with the OpenEmbedded 30 build system, as the Yocto Project ships with tools that speed up creating 31 layers. This section describes the steps you perform by hand to create 32 layers so that you can better understand them. For information about the 33 layer-creation tools, see the 34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`" 35 section in the Yocto Project Board Support Package (BSP) Developer's 36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 37 section further down in this manual. 38 39Follow these general steps to create your layer without using tools: 40 411. *Check Existing Layers:* Before creating a new layer, you should be 42 sure someone has not already created a layer containing the Metadata 43 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>` 44 for a list of layers from the OpenEmbedded community that can be used in 45 the Yocto Project. You could find a layer that is identical or close 46 to what you need. 47 482. *Create a Directory:* Create the directory for your layer. When you 49 create the layer, be sure to create the directory in an area not 50 associated with the Yocto Project :term:`Source Directory` 51 (e.g. the cloned ``poky`` repository). 52 53 While not strictly required, prepend the name of the directory with 54 the string "meta-". For example:: 55 56 meta-mylayer 57 meta-GUI_xyz 58 meta-mymachine 59 60 With rare exceptions, a layer's name follows this form:: 61 62 meta-root_name 63 64 Following this layer naming convention can save 65 you trouble later when tools, components, or variables "assume" your 66 layer name begins with "meta-". A notable example is in configuration 67 files as shown in the following step where layer names without the 68 "meta-" string are appended to several variables used in the 69 configuration. 70 713. *Create a Layer Configuration File:* Inside your new layer folder, 72 you need to create a ``conf/layer.conf`` file. It is easiest to take 73 an existing layer configuration file and copy that to your layer's 74 ``conf`` directory and then modify the file as needed. 75 76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project 77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>` 78 demonstrates the required syntax. For your layer, you need to replace 79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz" 80 for a layer named "meta-machinexyz"):: 81 82 # We have a conf and classes directory, add to BBPATH 83 BBPATH .= ":${LAYERDIR}" 84 85 # We have recipes-* directories, add to BBFILES 86 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ 87 ${LAYERDIR}/recipes-*/*/*.bbappend" 88 89 BBFILE_COLLECTIONS += "yoctobsp" 90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/" 91 BBFILE_PRIORITY_yoctobsp = "5" 92 LAYERVERSION_yoctobsp = "4" 93 LAYERSERIES_COMPAT_yoctobsp = "dunfell" 94 95 Following is an explanation of the layer configuration file: 96 97 - :term:`BBPATH`: Adds the layer's 98 root directory to BitBake's search path. Through the use of the 99 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``), 100 configuration files, and files that are included with ``include`` 101 and ``require`` statements. For these cases, BitBake uses the 102 first file that matches the name found in :term:`BBPATH`. This is 103 similar to the way the ``PATH`` variable is used for binaries. It 104 is recommended, therefore, that you use unique class and 105 configuration filenames in your custom layer. 106 107 - :term:`BBFILES`: Defines the 108 location for all recipes in the layer. 109 110 - :term:`BBFILE_COLLECTIONS`: 111 Establishes the current layer through a unique identifier that is 112 used throughout the OpenEmbedded build system to refer to the 113 layer. In this example, the identifier "yoctobsp" is the 114 representation for the container layer named "meta-yocto-bsp". 115 116 - :term:`BBFILE_PATTERN`: 117 Expands immediately during parsing to provide the directory of the 118 layer. 119 120 - :term:`BBFILE_PRIORITY`: 121 Establishes a priority to use for recipes in the layer when the 122 OpenEmbedded build finds recipes of the same name in different 123 layers. 124 125 - :term:`LAYERVERSION`: 126 Establishes a version number for the layer. You can use this 127 version number to specify this exact version of the layer as a 128 dependency when using the 129 :term:`LAYERDEPENDS` 130 variable. 131 132 - :term:`LAYERDEPENDS`: 133 Lists all layers on which this layer depends (if any). 134 135 - :term:`LAYERSERIES_COMPAT`: 136 Lists the :yocto_wiki:`Yocto Project </Releases>` 137 releases for which the current version is compatible. This 138 variable is a good way to indicate if your particular layer is 139 current. 140 1414. *Add Content:* Depending on the type of layer, add the content. If 142 the layer adds support for a machine, add the machine configuration 143 in a ``conf/machine/`` file within the layer. If the layer adds 144 distro policy, add the distro configuration in a ``conf/distro/`` 145 file within the layer. If the layer introduces new recipes, put the 146 recipes you need in ``recipes-*`` subdirectories within the layer. 147 148 .. note:: 149 150 For an explanation of layer hierarchy that is compliant with the 151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`" 152 section in the Yocto Project Board Support Package (BSP) Developer's Guide. 153 1545. *Optionally Test for Compatibility:* If you want permission to use 155 the Yocto Project Compatibility logo with your layer or application 156 that uses your layer, perform the steps to apply for compatibility. 157 See the 158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`" 159 section for more information. 160 161Following Best Practices When Creating Layers 162--------------------------------------------- 163 164To create layers that are easier to maintain and that will not impact 165builds for other machines, you should consider the information in the 166following list: 167 168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your 169 Configuration:* In other words, do not copy an entire recipe into 170 your layer and then modify it. Rather, use an append file 171 (``.bbappend``) to override only those parts of the original recipe 172 you need to modify. 173 174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``) 175 for each recipe that uses an include file. Or, if you are introducing 176 a new recipe that requires the included file, use the path relative 177 to the original layer directory to refer to the file. For example, 178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead 179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay 180 the include file, it could indicate a deficiency in the include file 181 in the layer to which it originally belongs. If this is the case, you 182 should try to address that deficiency instead of overlaying the 183 include file. For example, you could address this by getting the 184 maintainer of the include file to add a variable or variables to make 185 it easy to override the parts needing to be overridden. 186 187- *Structure Your Layers:* Proper use of overrides within append files 188 and placement of machine-specific files within your layer can ensure 189 that a build is not using the wrong Metadata and negatively impacting 190 a build for a different machine. Following are some examples: 191 192 - *Modify Variables to Support a Different Machine:* Suppose you 193 have a layer named ``meta-one`` that adds support for building 194 machine "one". To do so, you use an append file named 195 ``base-files.bbappend`` and create a dependency on "foo" by 196 altering the :term:`DEPENDS` 197 variable:: 198 199 DEPENDS = "foo" 200 201 The dependency is created during any 202 build that includes the layer ``meta-one``. However, you might not 203 want this dependency for all machines. For example, suppose you 204 are building for machine "two" but your ``bblayers.conf`` file has 205 the ``meta-one`` layer included. During the build, the 206 ``base-files`` for machine "two" will also have the dependency on 207 ``foo``. 208 209 To make sure your changes apply only when building machine "one", 210 use a machine override with the :term:`DEPENDS` statement:: 211 212 DEPENDS:one = "foo" 213 214 You should follow the same strategy when using ``:append`` 215 and ``:prepend`` operations:: 216 217 DEPENDS:append:one = " foo" 218 DEPENDS:prepend:one = "foo " 219 220 As an actual example, here's a 221 snippet from the generic kernel include file ``linux-yocto.inc``, 222 wherein the kernel compile and link options are adjusted in the 223 case of a subset of the supported architectures:: 224 225 DEPENDS:append:aarch64 = " libgcc" 226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}" 227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}" 228 229 DEPENDS:append:nios2 = " libgcc" 230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}" 231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}" 232 233 DEPENDS:append:arc = " libgcc" 234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}" 235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}" 236 237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc" 238 239 - *Place Machine-Specific Files in Machine-Specific Locations:* When 240 you have a base recipe, such as ``base-files.bb``, that contains a 241 :term:`SRC_URI` statement to a 242 file, you can use an append file to cause the build to use your 243 own version of the file. For example, an append file in your layer 244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could 245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows:: 246 247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:" 248 249 The build for machine "one" will pick up your machine-specific file as 250 long as you have the file in 251 ``meta-one/recipes-core/base-files/base-files/``. However, if you 252 are building for a different machine and the ``bblayers.conf`` 253 file includes the ``meta-one`` layer and the location of your 254 machine-specific file is the first location where that file is 255 found according to :term:`FILESPATH`, builds for all machines will 256 also use that machine-specific file. 257 258 You can make sure that a machine-specific file is used for a 259 particular machine by putting the file in a subdirectory specific 260 to the machine. For example, rather than placing the file in 261 ``meta-one/recipes-core/base-files/base-files/`` as shown above, 262 put it in ``meta-one/recipes-core/base-files/base-files/one/``. 263 Not only does this make sure the file is used only when building 264 for machine "one", but the build process locates the file more 265 quickly. 266 267 In summary, you need to place all files referenced from 268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in 269 order to restrict those files to machine-specific builds. 270 271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want 272 permission to use the Yocto Project Compatibility logo with your 273 layer or application that uses your layer, perform the steps to apply 274 for compatibility. See the 275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`" 276 section for more information. 277 278- *Follow the Layer Naming Convention:* Store custom layers in a Git 279 repository that use the ``meta-layer_name`` format. 280 281- *Group Your Layers Locally:* Clone your repository alongside other 282 cloned ``meta`` directories from the :term:`Source Directory`. 283 284Making Sure Your Layer is Compatible With Yocto Project 285------------------------------------------------------- 286 287When you create a layer used with the Yocto Project, it is advantageous 288to make sure that the layer interacts well with existing Yocto Project 289layers (i.e. the layer is compatible with the Yocto Project). Ensuring 290compatibility makes the layer easy to be consumed by others in the Yocto 291Project community and could allow you permission to use the Yocto 292Project Compatible Logo. 293 294.. note:: 295 296 Only Yocto Project member organizations are permitted to use the 297 Yocto Project Compatible Logo. The logo is not available for general 298 use. For information on how to become a Yocto Project member 299 organization, see the :yocto_home:`Yocto Project Website <>`. 300 301The Yocto Project Compatibility Program consists of a layer application 302process that requests permission to use the Yocto Project Compatibility 303Logo for your layer and application. The process consists of two parts: 304 3051. Successfully passing a script (``yocto-check-layer``) that when run 306 against your layer, tests it against constraints based on experiences 307 of how layers have worked in the real world and where pitfalls have 308 been found. Getting a "PASS" result from the script is required for 309 successful compatibility registration. 310 3112. Completion of an application acceptance form, which you can find at 312 :yocto_home:`/webform/yocto-project-compatible-registration`. 313 314To be granted permission to use the logo, you need to satisfy the 315following: 316 317- Be able to check the box indicating that you got a "PASS" when 318 running the script against your layer. 319 320- Answer "Yes" to the questions on the form or have an acceptable 321 explanation for any questions answered "No". 322 323- Be a Yocto Project Member Organization. 324 325The remainder of this section presents information on the registration 326form and on the ``yocto-check-layer`` script. 327 328Yocto Project Compatible Program Application 329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 330 331Use the form to apply for your layer's approval. Upon successful 332application, you can use the Yocto Project Compatibility Logo with your 333layer and the application that uses your layer. 334 335To access the form, use this link: 336:yocto_home:`/webform/yocto-project-compatible-registration`. 337Follow the instructions on the form to complete your application. 338 339The application consists of the following sections: 340 341- *Contact Information:* Provide your contact information as the fields 342 require. Along with your information, provide the released versions 343 of the Yocto Project for which your layer is compatible. 344 345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the 346 items in the checklist. There is space at the bottom of the form for 347 any explanations for items for which you answered "No". 348 349- *Recommendations:* Provide answers for the questions regarding Linux 350 kernel use and build success. 351 352``yocto-check-layer`` Script 353~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 354 355The ``yocto-check-layer`` script provides you a way to assess how 356compatible your layer is with the Yocto Project. You should run this 357script prior to using the form to apply for compatibility as described 358in the previous section. You need to achieve a "PASS" result in order to 359have your application form successfully processed. 360 361The script divides tests into three areas: COMMON, BSP, and DISTRO. For 362example, given a distribution layer (DISTRO), the layer must pass both 363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP 364layer, the layer must pass the COMMON and BSP set of tests. 365 366To execute the script, enter the following commands from your build 367directory:: 368 369 $ source oe-init-build-env 370 $ yocto-check-layer your_layer_directory 371 372Be sure to provide the actual directory for your 373layer as part of the command. 374 375Entering the command causes the script to determine the type of layer 376and then to execute a set of specific tests against the layer. The 377following list overviews the test: 378 379- ``common.test_readme``: Tests if a ``README`` file exists in the 380 layer and the file is not empty. 381 382- ``common.test_parse``: Tests to make sure that BitBake can parse the 383 files without error (i.e. ``bitbake -p``). 384 385- ``common.test_show_environment``: Tests that the global or per-recipe 386 environment is in order without errors (i.e. ``bitbake -e``). 387 388- ``common.test_world``: Verifies that ``bitbake world`` works. 389 390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO 391 layers do not come with recipes that change signatures. 392 393- ``common.test_layerseries_compat``: Verifies layer compatibility is 394 set properly. 395 396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine 397 configurations. 398 399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not 400 set the machine when the layer is added. 401 402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works 403 regardless of which machine is selected. 404 405- ``bsp.test_machine_signatures``: Verifies that building for a 406 particular machine affects only the signature of tasks specific to 407 that machine. 408 409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has 410 distro configurations. 411 412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer 413 does not set the distribution when the layer is added. 414 415Enabling Your Layer 416------------------- 417 418Before the OpenEmbedded build system can use your new layer, you need to 419enable it. To enable your layer, simply add your layer's path to the 420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is 421found in the :term:`Build Directory`. 422The following example shows how to enable your new 423``meta-mylayer`` layer (note how your new layer exists outside of 424the official ``poky`` repository which you would have checked out earlier):: 425 426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf 427 # changes incompatibly 428 POKY_BBLAYERS_CONF_VERSION = "2" 429 BBPATH = "${TOPDIR}" 430 BBFILES ?= "" 431 BBLAYERS ?= " \ 432 /home/user/poky/meta \ 433 /home/user/poky/meta-poky \ 434 /home/user/poky/meta-yocto-bsp \ 435 /home/user/mystuff/meta-mylayer \ 436 " 437 438BitBake parses each ``conf/layer.conf`` file from the top down as 439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf`` 440file. During the processing of each ``conf/layer.conf`` file, BitBake 441adds the recipes, classes and configurations contained within the 442particular layer to the source directory. 443 444Appending Other Layers Metadata With Your Layer 445----------------------------------------------- 446 447A recipe that appends Metadata to another recipe is called a BitBake 448append file. A BitBake append file uses the ``.bbappend`` file type 449suffix, while the corresponding recipe to which Metadata is being 450appended uses the ``.bb`` file type suffix. 451 452You can use a ``.bbappend`` file in your layer to make additions or 453changes to the content of another layer's recipe without having to copy 454the other layer's recipe into your layer. Your ``.bbappend`` file 455resides in your layer, while the main ``.bb`` recipe file to which you 456are appending Metadata resides in a different layer. 457 458Being able to append information to an existing recipe not only avoids 459duplication, but also automatically applies recipe changes from a 460different layer into your layer. If you were copying recipes, you would 461have to manually merge changes as they occur. 462 463When you create an append file, you must use the same root name as the 464corresponding recipe file. For example, the append file 465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This 466means the original recipe and append filenames are version 467number-specific. If the corresponding recipe is renamed to update to a 468newer version, you must also rename and possibly update the 469corresponding ``.bbappend`` as well. During the build process, BitBake 470displays an error on starting if it detects a ``.bbappend`` file that 471does not have a corresponding recipe with a matching name. See the 472:term:`BB_DANGLINGAPPENDS_WARNONLY` 473variable for information on how to handle this error. 474 475Overlaying a File Using Your Layer 476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 477 478As an example, consider the main formfactor recipe and a corresponding 479formfactor append file both from the :term:`Source Directory`. 480Here is the main 481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in 482the "meta" layer at ``meta/recipes-bsp/formfactor``:: 483 484 SUMMARY = "Device formfactor information" 485 DESCRIPTION = "A formfactor configuration file provides information about the \ 486 target hardware for which the image is being built and information that the \ 487 build system cannot obtain from other sources such as the kernel." 488 SECTION = "base" 489 LICENSE = "MIT" 490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" 491 PR = "r45" 492 493 SRC_URI = "file://config file://machconfig" 494 S = "${WORKDIR}" 495 496 PACKAGE_ARCH = "${MACHINE_ARCH}" 497 INHIBIT_DEFAULT_DEPS = "1" 498 499 do_install() { 500 # Install file only if it has contents 501 install -d ${D}${sysconfdir}/formfactor/ 502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/ 503 if [ -s "${S}/machconfig" ]; then 504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/ 505 fi 506 } 507 508In the main recipe, note the :term:`SRC_URI` 509variable, which tells the OpenEmbedded build system where to find files 510during the build. 511 512Following is the append file, which is named ``formfactor_0.0.bbappend`` 513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The 514file is in the layer at ``recipes-bsp/formfactor``:: 515 516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:" 517 518By default, the build system uses the 519:term:`FILESPATH` variable to 520locate files. This append file extends the locations by setting the 521:term:`FILESEXTRAPATHS` 522variable. Setting this variable in the ``.bbappend`` file is the most 523reliable and recommended method for adding directories to the search 524path used by the build system to find files. 525 526The statement in this example extends the directories to include 527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``, 528which resolves to a directory named ``formfactor`` in the same directory 529in which the append file resides (i.e. 530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must 531have the supporting directory structure set up that will contain any 532files or patches you will be including from the layer. 533 534Using the immediate expansion assignment operator ``:=`` is important 535because of the reference to :term:`THISDIR`. The trailing colon character is 536important as it ensures that items in the list remain colon-separated. 537 538.. note:: 539 540 BitBake automatically defines the :term:`THISDIR` variable. You should 541 never set this variable yourself. Using ":prepend" as part of the 542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other 543 paths in the final list. 544 545 Also, not all append files add extra files. Many append files simply 546 allow to add build options (e.g. ``systemd``). For these cases, your 547 append file would not even use the :term:`FILESEXTRAPATHS` statement. 548 549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where 550``rpi`` will exist in the list of :term:`OVERRIDES`, the file 551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be 552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in 553:ref:`ref-tasks-install` will return true, and the file will be installed. 554 555Installing Additional Files Using Your Layer 556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 557 558As another example, consider the main ``xserver-xf86-config`` recipe and a 559corresponding ``xserver-xf86-config`` append file both from the :term:`Source 560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named 561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at 562``meta/recipes-graphics/xorg-xserver``:: 563 564 SUMMARY = "X.Org X server configuration file" 565 HOMEPAGE = "http://www.x.org" 566 SECTION = "x11/base" 567 LICENSE = "MIT" 568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" 569 PR = "r33" 570 571 SRC_URI = "file://xorg.conf" 572 573 S = "${WORKDIR}" 574 575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf" 576 577 PACKAGE_ARCH = "${MACHINE_ARCH}" 578 ALLOW_EMPTY:${PN} = "1" 579 580 do_install () { 581 if test -s ${WORKDIR}/xorg.conf; then 582 install -d ${D}/${sysconfdir}/X11 583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/ 584 fi 585 } 586 587Following is the append file, which is named ``xserver-xf86-config_%.bbappend`` 588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The 589file is in the layer at ``recipes-graphics/xorg-xserver``:: 590 591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:" 592 593 SRC_URI:append:rpi = " \ 594 file://xorg.conf.d/98-pitft.conf \ 595 file://xorg.conf.d/99-calibration.conf \ 596 " 597 do_install:append:rpi () { 598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}" 599 if [ "${PITFT}" = "1" ]; then 600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/ 601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/ 602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/ 603 fi 604 } 605 606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*" 607 608Building off of the previous example, we once again are setting the 609:term:`FILESEXTRAPATHS` variable. In this case we are also using 610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in 611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a 612check for an additional :term:`MACHINE_FEATURES` that if set will cause these 613additional files to be installed. These additional files are listed in 614:term:`FILES` so that they will be packaged. 615 616Prioritizing Your Layer 617----------------------- 618 619Each layer is assigned a priority value. Priority values control which 620layer takes precedence if there are recipe files with the same name in 621multiple layers. For these cases, the recipe file from the layer with a 622higher priority number takes precedence. Priority values also affect the 623order in which multiple ``.bbappend`` files for the same recipe are 624applied. You can either specify the priority manually, or allow the 625build system to calculate it based on the layer's dependencies. 626 627To specify the layer's priority manually, use the 628:term:`BBFILE_PRIORITY` 629variable and append the layer's root name:: 630 631 BBFILE_PRIORITY_mylayer = "1" 632 633.. note:: 634 635 It is possible for a recipe with a lower version number 636 :term:`PV` in a layer that has a higher 637 priority to take precedence. 638 639 Also, the layer priority does not currently affect the precedence 640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake 641 might address this. 642 643Managing Layers 644--------------- 645 646You can use the BitBake layer management tool ``bitbake-layers`` to 647provide a view into the structure of recipes across a multi-layer 648project. Being able to generate output that reports on configured layers 649with their paths and priorities and on ``.bbappend`` files and their 650applicable recipes can help to reveal potential problems. 651 652For help on the BitBake layer management tool, use the following 653command:: 654 655 $ bitbake-layers --help 656 NOTE: Starting bitbake server... 657 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ... 658 659 BitBake layers utility 660 661 optional arguments: 662 -d, --debug Enable debug output 663 -q, --quiet Print only errors 664 -F, --force Force add without recipe parse verification 665 --color COLOR Colorize output (where COLOR is auto, always, never) 666 -h, --help show this help message and exit 667 668 subcommands: 669 <subcommand> 670 layerindex-fetch Fetches a layer from a layer index along with its 671 dependent layers, and adds them to conf/bblayers.conf. 672 layerindex-show-depends 673 Find layer dependencies from layer index. 674 add-layer Add one or more layers to bblayers.conf. 675 remove-layer Remove one or more layers from bblayers.conf. 676 flatten flatten layer configuration into a separate output 677 directory. 678 show-layers show current configured layers. 679 show-overlayed list overlayed recipes (where the same recipe exists 680 in another layer) 681 show-recipes list available recipes, showing the layer they are 682 provided by 683 show-appends list bbappend files and recipe files they apply to 684 show-cross-depends Show dependencies between recipes that cross layer 685 boundaries. 686 create-layer Create a basic layer 687 688 Use bitbake-layers <subcommand> --help to get help on a specific command 689 690The following list describes the available commands: 691 692- ``help:`` Displays general help or help on a specified command. 693 694- ``show-layers:`` Shows the current configured layers. 695 696- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed 697 when a recipe with the same name exists in another layer that has a 698 higher layer priority. 699 700- ``show-recipes:`` Lists available recipes and the layers that 701 provide them. 702 703- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to 704 which they apply. 705 706- ``show-cross-depends:`` Lists dependency relationships between 707 recipes that cross layer boundaries. 708 709- ``add-layer:`` Adds a layer to ``bblayers.conf``. 710 711- ``remove-layer:`` Removes a layer from ``bblayers.conf`` 712 713- ``flatten:`` Flattens the layer configuration into a separate 714 output directory. Flattening your layer configuration builds a 715 "flattened" directory that contains the contents of all layers, with 716 any overlayed recipes removed and any ``.bbappend`` files appended to 717 the corresponding recipes. You might have to perform some manual 718 cleanup of the flattened layer as follows: 719 720 - Non-recipe files (such as patches) are overwritten. The flatten 721 command shows a warning for these files. 722 723 - Anything beyond the normal layer setup has been added to the 724 ``layer.conf`` file. Only the lowest priority layer's 725 ``layer.conf`` is used. 726 727 - Overridden and appended items from ``.bbappend`` files need to be 728 cleaned up. The contents of each ``.bbappend`` end up in the 729 flattened recipe. However, if there are appended or changed 730 variable values, you need to tidy these up yourself. Consider the 731 following example. Here, the ``bitbake-layers`` command adds the 732 line ``#### bbappended ...`` so that you know where the following 733 lines originate:: 734 735 ... 736 DESCRIPTION = "A useful utility" 737 ... 738 EXTRA_OECONF = "--enable-something" 739 ... 740 741 #### bbappended from meta-anotherlayer #### 742 743 DESCRIPTION = "Customized utility" 744 EXTRA_OECONF += "--enable-somethingelse" 745 746 747 Ideally, you would tidy up these utilities as follows:: 748 749 ... 750 DESCRIPTION = "Customized utility" 751 ... 752 EXTRA_OECONF = "--enable-something --enable-somethingelse" 753 ... 754 755- ``layerindex-fetch``: Fetches a layer from a layer index, along 756 with its dependent layers, and adds the layers to the 757 ``conf/bblayers.conf`` file. 758 759- ``layerindex-show-depends``: Finds layer dependencies from the 760 layer index. 761 762- ``create-layer``: Creates a basic layer. 763 764Creating a General Layer Using the ``bitbake-layers`` Script 765------------------------------------------------------------ 766 767The ``bitbake-layers`` script with the ``create-layer`` subcommand 768simplifies creating a new general layer. 769 770.. note:: 771 772 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`" 773 section in the Yocto 774 Project Board Specific (BSP) Developer's Guide. 775 776 - In order to use a layer with the OpenEmbedded build system, you 777 need to add the layer to your ``bblayers.conf`` configuration 778 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`" 779 section for more information. 780 781The default mode of the script's operation with this subcommand is to 782create a layer with the following: 783 784- A layer priority of 6. 785 786- A ``conf`` subdirectory that contains a ``layer.conf`` file. 787 788- A ``recipes-example`` subdirectory that contains a further 789 subdirectory named ``example``, which contains an ``example.bb`` 790 recipe file. 791 792- A ``COPYING.MIT``, which is the license statement for the layer. The 793 script assumes you want to use the MIT license, which is typical for 794 most layers, for the contents of the layer itself. 795 796- A ``README`` file, which is a file describing the contents of your 797 new layer. 798 799In its simplest form, you can use the following command form to create a 800layer. The command creates a layer whose name corresponds to 801"your_layer_name" in the current directory:: 802 803 $ bitbake-layers create-layer your_layer_name 804 805As an example, the following command creates a layer named ``meta-scottrif`` 806in your home directory:: 807 808 $ cd /usr/home 809 $ bitbake-layers create-layer meta-scottrif 810 NOTE: Starting bitbake server... 811 Add your new layer with 'bitbake-layers add-layer meta-scottrif' 812 813If you want to set the priority of the layer to other than the default 814value of "6", you can either use the ``--priority`` option or you 815can edit the 816:term:`BBFILE_PRIORITY` value 817in the ``conf/layer.conf`` after the script creates it. Furthermore, if 818you want to give the example recipe file some name other than the 819default, you can use the ``--example-recipe-name`` option. 820 821The easiest way to see how the ``bitbake-layers create-layer`` command 822works is to experiment with the script. You can also read the usage 823information by entering the following:: 824 825 $ bitbake-layers create-layer --help 826 NOTE: Starting bitbake server... 827 usage: bitbake-layers create-layer [-h] [--priority PRIORITY] 828 [--example-recipe-name EXAMPLERECIPE] 829 layerdir 830 831 Create a basic layer 832 833 positional arguments: 834 layerdir Layer directory to create 835 836 optional arguments: 837 -h, --help show this help message and exit 838 --priority PRIORITY, -p PRIORITY 839 Layer directory to create 840 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE 841 Filename of the example recipe 842 843Adding a Layer Using the ``bitbake-layers`` Script 844-------------------------------------------------- 845 846Once you create your general layer, you must add it to your 847``bblayers.conf`` file. Adding the layer to this configuration file 848makes the OpenEmbedded build system aware of your layer so that it can 849search it for metadata. 850 851Add your layer by using the ``bitbake-layers add-layer`` command:: 852 853 $ bitbake-layers add-layer your_layer_name 854 855Here is an example that adds a 856layer named ``meta-scottrif`` to the configuration file. Following the 857command that adds the layer is another ``bitbake-layers`` command that 858shows the layers that are in your ``bblayers.conf`` file:: 859 860 $ bitbake-layers add-layer meta-scottrif 861 NOTE: Starting bitbake server... 862 Parsing recipes: 100% |##########################################################| Time: 0:00:49 863 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors. 864 $ bitbake-layers show-layers 865 NOTE: Starting bitbake server... 866 layer path priority 867 ========================================================================== 868 meta /home/scottrif/poky/meta 5 869 meta-poky /home/scottrif/poky/meta-poky 5 870 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5 871 workspace /home/scottrif/poky/build/workspace 99 872 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6 873 874 875Adding the layer to this file 876enables the build system to locate the layer during the build. 877 878.. note:: 879 880 During a build, the OpenEmbedded build system looks in the layers 881 from the top of the list down to the bottom in that order. 882 883Customizing Images 884================== 885 886You can customize images to satisfy particular requirements. This 887section describes several methods and provides guidelines for each. 888 889Customizing Images Using ``local.conf`` 890--------------------------------------- 891 892Probably the easiest way to customize an image is to add a package by 893way of the ``local.conf`` configuration file. Because it is limited to 894local use, this method generally only allows you to add packages and is 895not as flexible as creating your own customized image. When you add 896packages using local variables this way, you need to realize that these 897variable changes are in effect for every build and consequently affect 898all images, which might not be what you require. 899 900To add a package to your image using the local configuration file, use 901the :term:`IMAGE_INSTALL` variable with the ``:append`` operator:: 902 903 IMAGE_INSTALL:append = " strace" 904 905Use of the syntax is important; specifically, the leading space 906after the opening quote and before the package name, which is 907``strace`` in this example. This space is required since the ``:append`` 908operator does not add the space. 909 910Furthermore, you must use ``:append`` instead of the ``+=`` operator if 911you want to avoid ordering issues. The reason for this is because doing 912so unconditionally appends to the variable and avoids ordering problems 913due to the variable being set in image recipes and ``.bbclass`` files 914with operators like ``?=``. Using ``:append`` ensures the operation 915takes effect. 916 917As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all 918images. It is possible to extend the syntax so that the variable applies 919to a specific image only. Here is an example:: 920 921 IMAGE_INSTALL:append:pn-core-image-minimal = " strace" 922 923This example adds ``strace`` to the ``core-image-minimal`` image only. 924 925You can add packages using a similar approach through the 926:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only 927``core-image-*`` images are affected. 928 929Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES`` 930------------------------------------------------------------------------------- 931 932Another method for customizing your image is to enable or disable 933high-level image features by using the 934:term:`IMAGE_FEATURES` and 935:term:`EXTRA_IMAGE_FEATURES` 936variables. Although the functions for both variables are nearly 937equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within 938a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your 939``local.conf`` file, which is found in the 940:term:`Build Directory`. 941 942To understand how these features work, the best reference is 943:ref:`meta/classes/image.bbclass <ref-classes-image>`. 944This class lists out the available 945:term:`IMAGE_FEATURES` of which most map to package groups while some, such 946as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general 947configuration settings. 948 949In summary, the file looks at the contents of the :term:`IMAGE_FEATURES` 950variable and then maps or configures the feature accordingly. Based on 951this information, the build system automatically adds the appropriate 952packages or configurations to the 953:term:`IMAGE_INSTALL` variable. 954Effectively, you are enabling extra features by extending the class or 955creating a custom class for use with specialized image ``.bb`` files. 956 957Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local 958configuration file. Using a separate area from which to enable features 959with this variable helps you avoid overwriting the features in the image 960recipe that are enabled with :term:`IMAGE_FEATURES`. The value of 961:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within 962``meta/conf/bitbake.conf``. 963 964To illustrate how you can use these variables to modify your image, 965consider an example that selects the SSH server. The Yocto Project ships 966with two SSH servers you can use with your images: Dropbear and OpenSSH. 967Dropbear is a minimal SSH server appropriate for resource-constrained 968environments, while OpenSSH is a well-known standard SSH server 969implementation. By default, the ``core-image-sato`` image is configured 970to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb`` 971images both include OpenSSH. The ``core-image-minimal`` image does not 972contain an SSH server. 973 974You can customize your image and change these defaults. Edit the 975:term:`IMAGE_FEATURES` variable in your recipe or use the 976:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it 977configures the image you are working with to include 978``ssh-server-dropbear`` or ``ssh-server-openssh``. 979 980.. note:: 981 982 See the ":ref:`ref-manual/features:image features`" section in the Yocto 983 Project Reference Manual for a complete list of image features that ship 984 with the Yocto Project. 985 986Customizing Images Using Custom .bb Files 987----------------------------------------- 988 989You can also customize an image by creating a custom recipe that defines 990additional software as part of the image. The following example shows 991the form for the two lines you need:: 992 993 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2" 994 inherit core-image 995 996Defining the software using a custom recipe gives you total control over 997the contents of the image. It is important to use the correct names of 998packages in the :term:`IMAGE_INSTALL` variable. You must use the 999OpenEmbedded notation and not the Debian notation for the names (e.g. 1000``glibc-dev`` instead of ``libc6-dev``). 1001 1002The other method for creating a custom image is to base it on an 1003existing image. For example, if you want to create an image based on 1004``core-image-sato`` but add the additional package ``strace`` to the 1005image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new 1006``.bb`` and add the following line to the end of the copy:: 1007 1008 IMAGE_INSTALL += "strace" 1009 1010Customizing Images Using Custom Package Groups 1011---------------------------------------------- 1012 1013For complex custom images, the best approach for customizing an image is 1014to create a custom package group recipe that is used to build the image 1015or images. A good example of a package group recipe is 1016``meta/recipes-core/packagegroups/packagegroup-base.bb``. 1017 1018If you examine that recipe, you see that the :term:`PACKAGES` variable lists 1019the package group packages to produce. The ``inherit packagegroup`` 1020statement sets appropriate default values and automatically adds 1021``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each 1022package specified in the :term:`PACKAGES` statement. 1023 1024.. note:: 1025 1026 The ``inherit packagegroup`` line should be located near the top of the 1027 recipe, certainly before the :term:`PACKAGES` statement. 1028 1029For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS` 1030and :term:`RRECOMMENDS` entries to provide a list of packages the parent 1031task package should contain. You can see examples of these further down 1032in the ``packagegroup-base.bb`` recipe. 1033 1034Here is a short, fabricated example showing the same basic pieces for a 1035hypothetical packagegroup defined in ``packagegroup-custom.bb``, where 1036the variable :term:`PN` is the standard way to abbreviate the reference to 1037the full packagegroup name ``packagegroup-custom``:: 1038 1039 DESCRIPTION = "My Custom Package Groups" 1040 1041 inherit packagegroup 1042 1043 PACKAGES = "\ 1044 ${PN}-apps \ 1045 ${PN}-tools \ 1046 " 1047 1048 RDEPENDS:${PN}-apps = "\ 1049 dropbear \ 1050 portmap \ 1051 psplash" 1052 1053 RDEPENDS:${PN}-tools = "\ 1054 oprofile \ 1055 oprofileui-server \ 1056 lttng-tools" 1057 1058 RRECOMMENDS:${PN}-tools = "\ 1059 kernel-module-oprofile" 1060 1061In the previous example, two package group packages are created with 1062their dependencies and their recommended package dependencies listed: 1063``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To 1064build an image using these package group packages, you need to add 1065``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to 1066:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other 1067areas of this section. 1068 1069Customizing an Image Hostname 1070----------------------------- 1071 1072By default, the configured hostname (i.e. ``/etc/hostname``) in an image 1073is the same as the machine name. For example, if 1074:term:`MACHINE` equals "qemux86", the 1075configured hostname written to ``/etc/hostname`` is "qemux86". 1076 1077You can customize this name by altering the value of the "hostname" 1078variable in the ``base-files`` recipe using either an append file or a 1079configuration file. Use the following in an append file:: 1080 1081 hostname = "myhostname" 1082 1083Use the following in a configuration file:: 1084 1085 hostname:pn-base-files = "myhostname" 1086 1087Changing the default value of the variable "hostname" can be useful in 1088certain situations. For example, suppose you need to do extensive 1089testing on an image and you would like to easily identify the image 1090under test from existing images with typical default hostnames. In this 1091situation, you could change the default hostname to "testme", which 1092results in all the images using the name "testme". Once testing is 1093complete and you do not need to rebuild the image for test any longer, 1094you can easily reset the default hostname. 1095 1096Another point of interest is that if you unset the variable, the image 1097will have no default hostname in the filesystem. Here is an example that 1098unsets the variable in a configuration file:: 1099 1100 hostname:pn-base-files = "" 1101 1102Having no default hostname in the filesystem is suitable for 1103environments that use dynamic hostnames such as virtual machines. 1104 1105Writing a New Recipe 1106==================== 1107 1108Recipes (``.bb`` files) are fundamental components in the Yocto Project 1109environment. Each software component built by the OpenEmbedded build 1110system requires a recipe to define the component. This section describes 1111how to create, write, and test a new recipe. 1112 1113.. note:: 1114 1115 For information on variables that are useful for recipes and for 1116 information about recipe naming issues, see the 1117 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project 1118 Reference Manual. 1119 1120Overview 1121-------- 1122 1123The following figure shows the basic process for creating a new recipe. 1124The remainder of the section provides details for the steps. 1125 1126.. image:: figures/recipe-workflow.png 1127 :align: center 1128 1129Locate or Automatically Create a Base Recipe 1130-------------------------------------------- 1131 1132You can always write a recipe from scratch. However, there are three choices 1133that can help you quickly get started with a new recipe: 1134 1135- ``devtool add``: A command that assists in creating a recipe and an 1136 environment conducive to development. 1137 1138- ``recipetool create``: A command provided by the Yocto Project that 1139 automates creation of a base recipe based on the source files. 1140 1141- *Existing Recipes:* Location and modification of an existing recipe 1142 that is similar in function to the recipe you need. 1143 1144.. note:: 1145 1146 For information on recipe syntax, see the 1147 ":ref:`dev-manual/common-tasks:recipe syntax`" section. 1148 1149Creating the Base Recipe Using ``devtool add`` 1150~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1151 1152The ``devtool add`` command uses the same logic for auto-creating the 1153recipe as ``recipetool create``, which is listed below. Additionally, 1154however, ``devtool add`` sets up an environment that makes it easy for 1155you to patch the source and to make changes to the recipe as is often 1156necessary when adding a recipe to build a new piece of software to be 1157included in a build. 1158 1159You can find a complete description of the ``devtool add`` command in 1160the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section 1161in the Yocto Project Application Development and the Extensible Software 1162Development Kit (eSDK) manual. 1163 1164Creating the Base Recipe Using ``recipetool create`` 1165~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1166 1167``recipetool create`` automates creation of a base recipe given a set of 1168source code files. As long as you can extract or point to the source 1169files, the tool will construct a recipe and automatically configure all 1170pre-build information into the recipe. For example, suppose you have an 1171application that builds using Autotools. Creating the base recipe using 1172``recipetool`` results in a recipe that has the pre-build dependencies, 1173license requirements, and checksums configured. 1174 1175To run the tool, you just need to be in your 1176:term:`Build Directory` and have sourced the 1177build environment setup script (i.e. 1178:ref:`structure-core-script`). 1179To get help on the tool, use the following command:: 1180 1181 $ recipetool -h 1182 NOTE: Starting bitbake server... 1183 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ... 1184 1185 OpenEmbedded recipe tool 1186 1187 options: 1188 -d, --debug Enable debug output 1189 -q, --quiet Print only errors 1190 --color COLOR Colorize output (where COLOR is auto, always, never) 1191 -h, --help show this help message and exit 1192 1193 subcommands: 1194 create Create a new recipe 1195 newappend Create a bbappend for the specified target in the specified 1196 layer 1197 setvar Set a variable within a recipe 1198 appendfile Create/update a bbappend to replace a target file 1199 appendsrcfiles Create/update a bbappend to add or replace source files 1200 appendsrcfile Create/update a bbappend to add or replace a source file 1201 Use recipetool <subcommand> --help to get help on a specific command 1202 1203Running ``recipetool create -o OUTFILE`` creates the base recipe and 1204locates it properly in the layer that contains your source files. 1205Following are some syntax examples: 1206 1207 - Use this syntax to generate a recipe based on source. Once generated, 1208 the recipe resides in the existing source code layer:: 1209 1210 recipetool create -o OUTFILE source 1211 1212 - Use this syntax to generate a recipe using code that 1213 you extract from source. The extracted code is placed in its own layer 1214 defined by :term:`EXTERNALSRC`. 1215 :: 1216 1217 recipetool create -o OUTFILE -x EXTERNALSRC source 1218 1219 - Use this syntax to generate a recipe based on source. The options 1220 direct ``recipetool`` to generate debugging information. Once generated, 1221 the recipe resides in the existing source code layer:: 1222 1223 recipetool create -d -o OUTFILE source 1224 1225Locating and Using a Similar Recipe 1226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1227 1228Before writing a recipe from scratch, it is often useful to discover 1229whether someone else has already written one that meets (or comes close 1230to meeting) your needs. The Yocto Project and OpenEmbedded communities 1231maintain many recipes that might be candidates for what you are doing. 1232You can find a good central index of these recipes in the 1233:oe_layerindex:`OpenEmbedded Layer Index <>`. 1234 1235Working from an existing recipe or a skeleton recipe is the best way to 1236get started. Here are some points on both methods: 1237 1238- *Locate and modify a recipe that is close to what you want to do:* 1239 This method works when you are familiar with the current recipe 1240 space. The method does not work so well for those new to the Yocto 1241 Project or writing recipes. 1242 1243 Some risks associated with this method are using a recipe that has 1244 areas totally unrelated to what you are trying to accomplish with 1245 your recipe, not recognizing areas of the recipe that you might have 1246 to add from scratch, and so forth. All these risks stem from 1247 unfamiliarity with the existing recipe space. 1248 1249- *Use and modify the following skeleton recipe:* If for some reason 1250 you do not want to use ``recipetool`` and you cannot find an existing 1251 recipe that is close to meeting your needs, you can use the following 1252 structure to provide the fundamental areas of a new recipe. 1253 :: 1254 1255 DESCRIPTION = "" 1256 HOMEPAGE = "" 1257 LICENSE = "" 1258 SECTION = "" 1259 DEPENDS = "" 1260 LIC_FILES_CHKSUM = "" 1261 1262 SRC_URI = "" 1263 1264Storing and Naming the Recipe 1265----------------------------- 1266 1267Once you have your base recipe, you should put it in your own layer and 1268name it appropriately. Locating it correctly ensures that the 1269OpenEmbedded build system can find it when you use BitBake to process 1270the recipe. 1271 1272- *Storing Your Recipe:* The OpenEmbedded build system locates your 1273 recipe through the layer's ``conf/layer.conf`` file and the 1274 :term:`BBFILES` variable. This 1275 variable sets up a path from which the build system can locate 1276 recipes. Here is the typical use:: 1277 1278 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ 1279 ${LAYERDIR}/recipes-*/*/*.bbappend" 1280 1281 Consequently, you need to be sure you locate your new recipe inside 1282 your layer such that it can be found. 1283 1284 You can find more information on how layers are structured in the 1285 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section. 1286 1287- *Naming Your Recipe:* When you name your recipe, you need to follow 1288 this naming convention:: 1289 1290 basename_version.bb 1291 1292 Use lower-cased characters and do not include the reserved suffixes 1293 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use 1294 them as part of your recipe name unless the string applies). Here are some 1295 examples: 1296 1297 .. code-block:: none 1298 1299 cups_1.7.0.bb 1300 gawk_4.0.2.bb 1301 irssi_0.8.16-rc1.bb 1302 1303Running a Build on the Recipe 1304----------------------------- 1305 1306Creating a new recipe is usually an iterative process that requires 1307using BitBake to process the recipe multiple times in order to 1308progressively discover and add information to the recipe file. 1309 1310Assuming you have sourced the build environment setup script (i.e. 1311:ref:`structure-core-script`) and you are in 1312the :term:`Build Directory`, use 1313BitBake to process your recipe. All you need to provide is the 1314``basename`` of the recipe as described in the previous section:: 1315 1316 $ bitbake basename 1317 1318During the build, the OpenEmbedded build system creates a temporary work 1319directory for each recipe 1320(``${``\ :term:`WORKDIR`\ ``}``) 1321where it keeps extracted source files, log files, intermediate 1322compilation and packaging files, and so forth. 1323 1324The path to the per-recipe temporary work directory depends on the 1325context in which it is being built. The quickest way to find this path 1326is to have BitBake return it by running the following:: 1327 1328 $ bitbake -e basename | grep ^WORKDIR= 1329 1330As an example, assume a Source Directory 1331top-level folder named ``poky``, a default Build Directory at 1332``poky/build``, and a ``qemux86-poky-linux`` machine target system. 1333Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this 1334case, the work directory the build system uses to build the package 1335would be as follows:: 1336 1337 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0 1338 1339Inside this directory you can find sub-directories such as ``image``, 1340``packages-split``, and ``temp``. After the build, you can examine these 1341to determine how well the build went. 1342 1343.. note:: 1344 1345 You can find log files for each task in the recipe's ``temp`` 1346 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``). 1347 Log files are named ``log.taskname`` (e.g. ``log.do_configure``, 1348 ``log.do_fetch``, and ``log.do_compile``). 1349 1350You can find more information about the build process in 1351":doc:`/overview-manual/development-environment`" 1352chapter of the Yocto Project Overview and Concepts Manual. 1353 1354Fetching Code 1355------------- 1356 1357The first thing your recipe must do is specify how to fetch the source 1358files. Fetching is controlled mainly through the 1359:term:`SRC_URI` variable. Your recipe 1360must have a :term:`SRC_URI` variable that points to where the source is 1361located. For a graphical representation of source locations, see the 1362":ref:`overview-manual/concepts:sources`" section in 1363the Yocto Project Overview and Concepts Manual. 1364 1365The :ref:`ref-tasks-fetch` task uses 1366the prefix of each entry in the :term:`SRC_URI` variable value to determine 1367which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your 1368source files. It is the :term:`SRC_URI` variable that triggers the fetcher. 1369The :ref:`ref-tasks-patch` task uses 1370the variable after source is fetched to apply patches. The OpenEmbedded 1371build system uses 1372:term:`FILESOVERRIDES` for 1373scanning directory locations for local files in :term:`SRC_URI`. 1374 1375The :term:`SRC_URI` variable in your recipe must define each unique location 1376for your source files. It is good practice to not hard-code version 1377numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these 1378values, use ``${``\ :term:`PV`\ ``}``, 1379which causes the fetch process to use the version specified in the 1380recipe filename. Specifying the version in this manner means that 1381upgrading the recipe to a future version is as simple as renaming the 1382recipe to match the new version. 1383 1384Here is a simple example from the 1385``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source 1386comes from a single tarball. Notice the use of the 1387:term:`PV` variable:: 1388 1389 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \ 1390 1391Files mentioned in :term:`SRC_URI` whose names end in a typical archive 1392extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so 1393forth), are automatically extracted during the 1394:ref:`ref-tasks-unpack` task. For 1395another example that specifies these types of files, see the 1396":ref:`dev-manual/common-tasks:autotooled package`" section. 1397 1398Another way of specifying source is from an SCM. For Git repositories, 1399you must specify :term:`SRCREV` and you should specify :term:`PV` to include 1400the revision with :term:`SRCPV`. Here is an example from the recipe 1401``meta/recipes-core/musl/gcompat_git.bb``:: 1402 1403 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current" 1404 1405 PV = "1.0.0+1.1+git${SRCPV}" 1406 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793" 1407 1408If your :term:`SRC_URI` statement includes URLs pointing to individual files 1409fetched from a remote server other than a version control system, 1410BitBake attempts to verify the files against checksums defined in your 1411recipe to ensure they have not been tampered with or otherwise modified 1412since the recipe was written. Two checksums are used: 1413``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``. 1414 1415If your :term:`SRC_URI` variable points to more than a single URL (excluding 1416SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for 1417each URL. For these cases, you provide a name for each URL as part of 1418the :term:`SRC_URI` and then reference that name in the subsequent checksum 1419statements. Here is an example combining lines from the files 1420``git.inc`` and ``git_2.24.1.bb``:: 1421 1422 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \ 1423 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages" 1424 1425 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b" 1426 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02" 1427 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d" 1428 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230" 1429 1430Proper values for ``md5`` and ``sha256`` checksums might be available 1431with other signatures on the download page for the upstream source (e.g. 1432``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the 1433OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``, 1434you should verify all the signatures you find by hand. 1435 1436If no :term:`SRC_URI` checksums are specified when you attempt to build the 1437recipe, or you provide an incorrect checksum, the build will produce an 1438error for each missing or incorrect checksum. As part of the error 1439message, the build system provides the checksum string corresponding to 1440the fetched file. Once you have the correct checksums, you can copy and 1441paste them into your recipe and then run the build again to continue. 1442 1443.. note:: 1444 1445 As mentioned, if the upstream source provides signatures for 1446 verifying the downloaded source code, you should verify those 1447 manually before setting the checksum values in the recipe and 1448 continuing with the build. 1449 1450This final example is a bit more complicated and is from the 1451``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The 1452example's :term:`SRC_URI` statement identifies multiple files as the source 1453files for the recipe: a tarball, a patch file, a desktop file, and an 1454icon. 1455:: 1456 1457 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \ 1458 file://xwc.patch \ 1459 file://rxvt.desktop \ 1460 file://rxvt.png" 1461 1462When you specify local files using the ``file://`` URI protocol, the 1463build system fetches files from the local machine. The path is relative 1464to the :term:`FILESPATH` variable 1465and searches specific directories in a certain order: 1466``${``\ :term:`BP`\ ``}``, 1467``${``\ :term:`BPN`\ ``}``, and 1468``files``. The directories are assumed to be subdirectories of the 1469directory in which the recipe or append file resides. For another 1470example that specifies these types of files, see the 1471":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section. 1472 1473The previous example also specifies a patch file. Patch files are files 1474whose names usually end in ``.patch`` or ``.diff`` but can end with 1475compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example. 1476The build system automatically applies patches as described in the 1477":ref:`dev-manual/common-tasks:patching code`" section. 1478 1479Fetching Code Through Firewalls 1480~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1481 1482Some users are behind firewalls and need to fetch code through a proxy. 1483See the ":doc:`/ref-manual/faq`" chapter for advice. 1484 1485Limiting the Number of Parallel Connections 1486~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1487 1488Some users are behind firewalls or use servers where the number of parallel 1489connections is limited. In such cases, you can limit the number of fetch 1490tasks being run in parallel by adding the following to your ``local.conf`` 1491file:: 1492 1493 do_fetch[number_threads] = "4" 1494 1495Unpacking Code 1496-------------- 1497 1498During the build, the 1499:ref:`ref-tasks-unpack` task unpacks 1500the source with ``${``\ :term:`S`\ ``}`` 1501pointing to where it is unpacked. 1502 1503If you are fetching your source files from an upstream source archived 1504tarball and the tarball's internal structure matches the common 1505convention of a top-level subdirectory named 1506``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``, 1507then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to 1508fetch source from an archive that does not use this convention, or from 1509an SCM like Git or Subversion, your recipe needs to define :term:`S`. 1510 1511If processing your recipe using BitBake successfully unpacks the source 1512files, you need to be sure that the directory pointed to by ``${S}`` 1513matches the structure of the source. 1514 1515Patching Code 1516------------- 1517 1518Sometimes it is necessary to patch code after it has been fetched. Any 1519files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or 1520``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are 1521treated as patches. The 1522:ref:`ref-tasks-patch` task 1523automatically applies these patches. 1524 1525The build system should be able to apply patches with the "-p1" option 1526(i.e. one directory level in the path will be stripped off). If your 1527patch needs to have more directory levels stripped off, specify the 1528number of levels using the "striplevel" option in the :term:`SRC_URI` entry 1529for the patch. Alternatively, if your patch needs to be applied in a 1530specific subdirectory that is not specified in the patch file, use the 1531"patchdir" option in the entry. 1532 1533As with all local files referenced in 1534:term:`SRC_URI` using ``file://``, 1535you should place patch files in a directory next to the recipe either 1536named the same as the base name of the recipe 1537(:term:`BP` and 1538:term:`BPN`) or "files". 1539 1540Licensing 1541--------- 1542 1543Your recipe needs to have both the 1544:term:`LICENSE` and 1545:term:`LIC_FILES_CHKSUM` 1546variables: 1547 1548- :term:`LICENSE`: This variable specifies the license for the software. 1549 If you do not know the license under which the software you are 1550 building is distributed, you should go to the source code and look 1551 for that information. Typical files containing this information 1552 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could 1553 also find the information near the top of a source file. For example, 1554 given a piece of software licensed under the GNU General Public 1555 License version 2, you would set :term:`LICENSE` as follows:: 1556 1557 LICENSE = "GPL-2.0-only" 1558 1559 The licenses you specify within :term:`LICENSE` can have any name as long 1560 as you do not use spaces, since spaces are used as separators between 1561 license names. For standard licenses, use the names of the files in 1562 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names 1563 defined in ``meta/conf/licenses.conf``. 1564 1565- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this 1566 variable to make sure the license text has not changed. If it has, 1567 the build produces an error and it affords you the chance to figure 1568 it out and correct the problem. 1569 1570 You need to specify all applicable licensing files for the software. 1571 At the end of the configuration step, the build process will compare 1572 the checksums of the files to be sure the text has not changed. Any 1573 differences result in an error with the message containing the 1574 current checksum. For more explanation and examples of how to set the 1575 :term:`LIC_FILES_CHKSUM` variable, see the 1576 ":ref:`dev-manual/common-tasks:tracking license changes`" section. 1577 1578 To determine the correct checksum string, you can list the 1579 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect 1580 md5 strings, attempt to build the software, and then note the 1581 resulting error messages that will report the correct md5 strings. 1582 See the ":ref:`dev-manual/common-tasks:fetching code`" section for 1583 additional information. 1584 1585 Here is an example that assumes the software has a ``COPYING`` file:: 1586 1587 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx" 1588 1589 When you try to build the 1590 software, the build system will produce an error and give you the 1591 correct string that you can substitute into the recipe file for a 1592 subsequent build. 1593 1594Dependencies 1595------------ 1596 1597Most software packages have a short list of other packages that they 1598require, which are called dependencies. These dependencies fall into two 1599main categories: build-time dependencies, which are required when the 1600software is built; and runtime dependencies, which are required to be 1601installed on the target in order for the software to run. 1602 1603Within a recipe, you specify build-time dependencies using the 1604:term:`DEPENDS` variable. Although there are nuances, 1605items specified in :term:`DEPENDS` should be names of other 1606recipes. It is important that you specify all build-time dependencies 1607explicitly. 1608 1609Another consideration is that configure scripts might automatically 1610check for optional dependencies and enable corresponding functionality 1611if those dependencies are found. If you wish to make a recipe that is 1612more generally useful (e.g. publish the recipe in a layer for others to 1613use), instead of hard-disabling the functionality, you can use the 1614:term:`PACKAGECONFIG` variable to allow functionality and the 1615corresponding dependencies to be enabled and disabled easily by other 1616users of the recipe. 1617 1618Similar to build-time dependencies, you specify runtime dependencies 1619through a variable - 1620:term:`RDEPENDS`, which is 1621package-specific. All variables that are package-specific need to have 1622the name of the package added to the end as an override. Since the main 1623package for a recipe has the same name as the recipe, and the recipe's 1624name can be found through the 1625``${``\ :term:`PN`\ ``}`` variable, then 1626you specify the dependencies for the main package by setting 1627``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you 1628would set ``RDEPENDS:${PN}-tools``, and so forth. 1629 1630Some runtime dependencies will be set automatically at packaging time. 1631These dependencies include any shared library dependencies (i.e. if a 1632package "example" contains "libexample" and another package "mypackage" 1633contains a binary that links to "libexample" then the OpenEmbedded build 1634system will automatically add a runtime dependency to "mypackage" on 1635"example"). See the 1636":ref:`overview-manual/concepts:automatically added runtime dependencies`" 1637section in the Yocto Project Overview and Concepts Manual for further 1638details. 1639 1640Configuring the Recipe 1641---------------------- 1642 1643Most software provides some means of setting build-time configuration 1644options before compilation. Typically, setting these options is 1645accomplished by running a configure script with options, or by modifying 1646a build configuration file. 1647 1648.. note:: 1649 1650 As of Yocto Project Release 1.7, some of the core recipes that 1651 package binary configuration scripts now disable the scripts due to 1652 the scripts previously requiring error-prone path substitution. The 1653 OpenEmbedded build system uses ``pkg-config`` now, which is much more 1654 robust. You can find a list of the ``*-config`` scripts that are disabled 1655 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section 1656 in the Yocto Project Reference Manual. 1657 1658A major part of build-time configuration is about checking for 1659build-time dependencies and possibly enabling optional functionality as 1660a result. You need to specify any build-time dependencies for the 1661software you are building in your recipe's 1662:term:`DEPENDS` value, in terms of 1663other recipes that satisfy those dependencies. You can often find 1664build-time or runtime dependencies described in the software's 1665documentation. 1666 1667The following list provides configuration items of note based on how 1668your software is built: 1669 1670- *Autotools:* If your source files have a ``configure.ac`` file, then 1671 your software is built using Autotools. If this is the case, you just 1672 need to modify the configuration. 1673 1674 When using Autotools, your recipe needs to inherit the 1675 :ref:`autotools <ref-classes-autotools>` class 1676 and your recipe does not have to contain a 1677 :ref:`ref-tasks-configure` task. 1678 However, you might still want to make some adjustments. For example, 1679 you can set 1680 :term:`EXTRA_OECONF` or 1681 :term:`PACKAGECONFIG_CONFARGS` 1682 to pass any needed configure options that are specific to the recipe. 1683 1684- *CMake:* If your source files have a ``CMakeLists.txt`` file, then 1685 your software is built using CMake. If this is the case, you just 1686 need to modify the configuration. 1687 1688 When you use CMake, your recipe needs to inherit the 1689 :ref:`cmake <ref-classes-cmake>` class and your 1690 recipe does not have to contain a 1691 :ref:`ref-tasks-configure` task. 1692 You can make some adjustments by setting 1693 :term:`EXTRA_OECMAKE` to 1694 pass any needed configure options that are specific to the recipe. 1695 1696 .. note:: 1697 1698 If you need to install one or more custom CMake toolchain files 1699 that are supplied by the application you are building, install the 1700 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``. 1701 1702- *Other:* If your source files do not have a ``configure.ac`` or 1703 ``CMakeLists.txt`` file, then your software is built using some 1704 method other than Autotools or CMake. If this is the case, you 1705 normally need to provide a 1706 :ref:`ref-tasks-configure` task 1707 in your recipe unless, of course, there is nothing to configure. 1708 1709 Even if your software is not being built by Autotools or CMake, you 1710 still might not need to deal with any configuration issues. You need 1711 to determine if configuration is even a required step. You might need 1712 to modify a Makefile or some configuration file used for the build to 1713 specify necessary build options. Or, perhaps you might need to run a 1714 provided, custom configure script with the appropriate options. 1715 1716 For the case involving a custom configure script, you would run 1717 ``./configure --help`` and look for the options you need to set. 1718 1719Once configuration succeeds, it is always good practice to look at the 1720``log.do_configure`` file to ensure that the appropriate options have 1721been enabled and no additional build-time dependencies need to be added 1722to :term:`DEPENDS`. For example, if the configure script reports that it 1723found something not mentioned in :term:`DEPENDS`, or that it did not find 1724something that it needed for some desired optional functionality, then 1725you would need to add those to :term:`DEPENDS`. Looking at the log might 1726also reveal items being checked for, enabled, or both that you do not 1727want, or items not being found that are in :term:`DEPENDS`, in which case 1728you would need to look at passing extra options to the configure script 1729as needed. For reference information on configure options specific to 1730the software you are building, you can consult the output of the 1731``./configure --help`` command within ``${S}`` or consult the software's 1732upstream documentation. 1733 1734Using Headers to Interface with Devices 1735--------------------------------------- 1736 1737If your recipe builds an application that needs to communicate with some 1738device or needs an API into a custom kernel, you will need to provide 1739appropriate header files. Under no circumstances should you ever modify 1740the existing 1741``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file. 1742These headers are used to build ``libc`` and must not be compromised 1743with custom or machine-specific header information. If you customize 1744``libc`` through modified headers all other applications that use 1745``libc`` thus become affected. 1746 1747.. note:: 1748 1749 Never copy and customize the ``libc`` header file (i.e. 1750 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``). 1751 1752The correct way to interface to a device or custom kernel is to use a 1753separate package that provides the additional headers for the driver or 1754other unique interfaces. When doing so, your application also becomes 1755responsible for creating a dependency on that specific provider. 1756 1757Consider the following: 1758 1759- Never modify ``linux-libc-headers.inc``. Consider that file to be 1760 part of the ``libc`` system, and not something you use to access the 1761 kernel directly. You should access ``libc`` through specific ``libc`` 1762 calls. 1763 1764- Applications that must talk directly to devices should either provide 1765 necessary headers themselves, or establish a dependency on a special 1766 headers package that is specific to that driver. 1767 1768For example, suppose you want to modify an existing header that adds I/O 1769control or network support. If the modifications are used by a small 1770number programs, providing a unique version of a header is easy and has 1771little impact. When doing so, bear in mind the guidelines in the 1772previous list. 1773 1774.. note:: 1775 1776 If for some reason your changes need to modify the behavior of the ``libc``, 1777 and subsequently all other applications on the system, use a ``.bbappend`` 1778 to modify the ``linux-kernel-headers.inc`` file. However, take care to not 1779 make the changes machine specific. 1780 1781Consider a case where your kernel is older and you need an older 1782``libc`` ABI. The headers installed by your recipe should still be a 1783standard mainline kernel, not your own custom one. 1784 1785When you use custom kernel headers you need to get them from 1786:term:`STAGING_KERNEL_DIR`, 1787which is the directory with kernel headers that are required to build 1788out-of-tree modules. Your recipe will also need the following:: 1789 1790 do_configure[depends] += "virtual/kernel:do_shared_workdir" 1791 1792Compilation 1793----------- 1794 1795During a build, the ``do_compile`` task happens after source is fetched, 1796unpacked, and configured. If the recipe passes through ``do_compile`` 1797successfully, nothing needs to be done. 1798 1799However, if the compile step fails, you need to diagnose the failure. 1800Here are some common issues that cause failures. 1801 1802.. note:: 1803 1804 For cases where improper paths are detected for configuration files 1805 or for when libraries/headers cannot be found, be sure you are using 1806 the more robust ``pkg-config``. See the note in section 1807 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information. 1808 1809- *Parallel build failures:* These failures manifest themselves as 1810 intermittent errors, or errors reporting that a file or directory 1811 that should be created by some other part of the build process could 1812 not be found. This type of failure can occur even if, upon 1813 inspection, the file or directory does exist after the build has 1814 failed, because that part of the build process happened in the wrong 1815 order. 1816 1817 To fix the problem, you need to either satisfy the missing dependency 1818 in the Makefile or whatever script produced the Makefile, or (as a 1819 workaround) set :term:`PARALLEL_MAKE` to an empty string:: 1820 1821 PARALLEL_MAKE = "" 1822 1823 For information on parallel Makefile issues, see the 1824 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section. 1825 1826- *Improper host path usage:* This failure applies to recipes building 1827 for the target or ``nativesdk`` only. The failure occurs when the 1828 compilation process uses improper headers, libraries, or other files 1829 from the host system when cross-compiling for the target. 1830 1831 To fix the problem, examine the ``log.do_compile`` file to identify 1832 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and 1833 so forth) and then either add configure options, apply a patch, or do 1834 both. 1835 1836- *Failure to find required libraries/headers:* If a build-time 1837 dependency is missing because it has not been declared in 1838 :term:`DEPENDS`, or because the 1839 dependency exists but the path used by the build process to find the 1840 file is incorrect and the configure step did not detect it, the 1841 compilation process could fail. For either of these failures, the 1842 compilation process notes that files could not be found. In these 1843 cases, you need to go back and add additional options to the 1844 configure script as well as possibly add additional build-time 1845 dependencies to :term:`DEPENDS`. 1846 1847 Occasionally, it is necessary to apply a patch to the source to 1848 ensure the correct paths are used. If you need to specify paths to 1849 find files staged into the sysroot from other recipes, use the 1850 variables that the OpenEmbedded build system provides (e.g. 1851 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so 1852 forth). 1853 1854Installing 1855---------- 1856 1857During ``do_install``, the task copies the built files along with their 1858hierarchy to locations that would mirror their locations on the target 1859device. The installation process copies files from the 1860``${``\ :term:`S`\ ``}``, 1861``${``\ :term:`B`\ ``}``, and 1862``${``\ :term:`WORKDIR`\ ``}`` 1863directories to the ``${``\ :term:`D`\ ``}`` 1864directory to create the structure as it should appear on the target 1865system. 1866 1867How your software is built affects what you must do to be sure your 1868software is installed correctly. The following list describes what you 1869must do for installation depending on the type of build system used by 1870the software being built: 1871 1872- *Autotools and CMake:* If the software your recipe is building uses 1873 Autotools or CMake, the OpenEmbedded build system understands how to 1874 install the software. Consequently, you do not have to have a 1875 ``do_install`` task as part of your recipe. You just need to make 1876 sure the install portion of the build completes with no issues. 1877 However, if you wish to install additional files not already being 1878 installed by ``make install``, you should do this using a 1879 ``do_install:append`` function using the install command as described 1880 in the "Manual" bulleted item later in this list. 1881 1882- *Other (using* ``make install``\ *)*: You need to define a ``do_install`` 1883 function in your recipe. The function should call 1884 ``oe_runmake install`` and will likely need to pass in the 1885 destination directory as well. How you pass that path is dependent on 1886 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``, 1887 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth). 1888 1889 For an example recipe using ``make install``, see the 1890 ":ref:`dev-manual/common-tasks:makefile-based package`" section. 1891 1892- *Manual:* You need to define a ``do_install`` function in your 1893 recipe. The function must first use ``install -d`` to create the 1894 directories under 1895 ``${``\ :term:`D`\ ``}``. Once the 1896 directories exist, your function can use ``install`` to manually 1897 install the built software into the directories. 1898 1899 You can find more information on ``install`` at 1900 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html. 1901 1902For the scenarios that do not use Autotools or CMake, you need to track 1903the installation and diagnose and fix any issues until everything 1904installs correctly. You need to look in the default location of 1905``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been 1906installed correctly. 1907 1908.. note:: 1909 1910 - During the installation process, you might need to modify some of 1911 the installed files to suit the target layout. For example, you 1912 might need to replace hard-coded paths in an initscript with 1913 values of variables provided by the build system, such as 1914 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such 1915 modifications during ``do_install``, be sure to modify the 1916 destination file after copying rather than before copying. 1917 Modifying after copying ensures that the build system can 1918 re-execute ``do_install`` if needed. 1919 1920 - ``oe_runmake install``, which can be run directly or can be run 1921 indirectly by the 1922 :ref:`autotools <ref-classes-autotools>` and 1923 :ref:`cmake <ref-classes-cmake>` classes, 1924 runs ``make install`` in parallel. Sometimes, a Makefile can have 1925 missing dependencies between targets that can result in race 1926 conditions. If you experience intermittent failures during 1927 ``do_install``, you might be able to work around them by disabling 1928 parallel Makefile installs by adding the following to the recipe:: 1929 1930 PARALLEL_MAKEINST = "" 1931 1932 See :term:`PARALLEL_MAKEINST` for additional information. 1933 1934 - If you need to install one or more custom CMake toolchain files 1935 that are supplied by the application you are building, install the 1936 files to ``${D}${datadir}/cmake/Modules`` during 1937 :ref:`ref-tasks-install`. 1938 1939Enabling System Services 1940------------------------ 1941 1942If you want to install a service, which is a process that usually starts 1943on boot and runs in the background, then you must include some 1944additional definitions in your recipe. 1945 1946If you are adding services and the service initialization script or the 1947service file itself is not installed, you must provide for that 1948installation in your recipe using a ``do_install:append`` function. If 1949your recipe already has a ``do_install`` function, update the function 1950near its end rather than adding an additional ``do_install:append`` 1951function. 1952 1953When you create the installation for your services, you need to 1954accomplish what is normally done by ``make install``. In other words, 1955make sure your installation arranges the output similar to how it is 1956arranged on the target system. 1957 1958The OpenEmbedded build system provides support for starting services two 1959different ways: 1960 1961- *SysVinit:* SysVinit is a system and service manager that manages the 1962 init system used to control the very basic functions of your system. 1963 The init program is the first program started by the Linux kernel 1964 when the system boots. Init then controls the startup, running and 1965 shutdown of all other programs. 1966 1967 To enable a service using SysVinit, your recipe needs to inherit the 1968 :ref:`update-rc.d <ref-classes-update-rc.d>` 1969 class. The class helps facilitate safely installing the package on 1970 the target. 1971 1972 You will need to set the 1973 :term:`INITSCRIPT_PACKAGES`, 1974 :term:`INITSCRIPT_NAME`, 1975 and 1976 :term:`INITSCRIPT_PARAMS` 1977 variables within your recipe. 1978 1979- *systemd:* System Management Daemon (systemd) was designed to replace 1980 SysVinit and to provide enhanced management of services. For more 1981 information on systemd, see the systemd homepage at 1982 https://freedesktop.org/wiki/Software/systemd/. 1983 1984 To enable a service using systemd, your recipe needs to inherit the 1985 :ref:`systemd <ref-classes-systemd>` class. See 1986 the ``systemd.bbclass`` file located in your :term:`Source Directory` 1987 section for 1988 more information. 1989 1990Packaging 1991--------- 1992 1993Successful packaging is a combination of automated processes performed 1994by the OpenEmbedded build system and some specific steps you need to 1995take. The following list describes the process: 1996 1997- *Splitting Files*: The ``do_package`` task splits the files produced 1998 by the recipe into logical components. Even software that produces a 1999 single binary might still have debug symbols, documentation, and 2000 other logical components that should be split out. The ``do_package`` 2001 task ensures that files are split up and packaged correctly. 2002 2003- *Running QA Checks*: The 2004 :ref:`insane <ref-classes-insane>` class adds a 2005 step to the package generation process so that output quality 2006 assurance checks are generated by the OpenEmbedded build system. This 2007 step performs a range of checks to be sure the build's output is free 2008 of common problems that show up during runtime. For information on 2009 these checks, see the 2010 :ref:`insane <ref-classes-insane>` class and 2011 the ":ref:`ref-manual/qa-checks:qa error and warning messages`" 2012 chapter in the Yocto Project Reference Manual. 2013 2014- *Hand-Checking Your Packages*: After you build your software, you 2015 need to be sure your packages are correct. Examine the 2016 ``${``\ :term:`WORKDIR`\ ``}/packages-split`` 2017 directory and make sure files are where you expect them to be. If you 2018 discover problems, you can set 2019 :term:`PACKAGES`, 2020 :term:`FILES`, 2021 ``do_install(:append)``, and so forth as needed. 2022 2023- *Splitting an Application into Multiple Packages*: If you need to 2024 split an application into several packages, see the 2025 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`" 2026 section for an example. 2027 2028- *Installing a Post-Installation Script*: For an example showing how 2029 to install a post-installation script, see the 2030 ":ref:`dev-manual/common-tasks:post-installation scripts`" section. 2031 2032- *Marking Package Architecture*: Depending on what your recipe is 2033 building and how it is configured, it might be important to mark the 2034 packages produced as being specific to a particular machine, or to 2035 mark them as not being specific to a particular machine or 2036 architecture at all. 2037 2038 By default, packages apply to any machine with the same architecture 2039 as the target machine. When a recipe produces packages that are 2040 machine-specific (e.g. the 2041 :term:`MACHINE` value is passed 2042 into the configure script or a patch is applied only for a particular 2043 machine), you should mark them as such by adding the following to the 2044 recipe:: 2045 2046 PACKAGE_ARCH = "${MACHINE_ARCH}" 2047 2048 On the other hand, if the recipe produces packages that do not 2049 contain anything specific to the target machine or architecture at 2050 all (e.g. recipes that simply package script files or configuration 2051 files), you should use the 2052 :ref:`allarch <ref-classes-allarch>` class to 2053 do this for you by adding this to your recipe:: 2054 2055 inherit allarch 2056 2057 Ensuring that the package architecture is correct is not critical 2058 while you are doing the first few builds of your recipe. However, it 2059 is important in order to ensure that your recipe rebuilds (or does 2060 not rebuild) appropriately in response to changes in configuration, 2061 and to ensure that you get the appropriate packages installed on the 2062 target machine, particularly if you run separate builds for more than 2063 one target machine. 2064 2065Sharing Files Between Recipes 2066----------------------------- 2067 2068Recipes often need to use files provided by other recipes on the build 2069host. For example, an application linking to a common library needs 2070access to the library itself and its associated headers. The way this 2071access is accomplished is by populating a sysroot with files. Each 2072recipe has two sysroots in its work directory, one for target files 2073(``recipe-sysroot``) and one for files that are native to the build host 2074(``recipe-sysroot-native``). 2075 2076.. note:: 2077 2078 You could find the term "staging" used within the Yocto project 2079 regarding files populating sysroots (e.g. the :term:`STAGING_DIR` 2080 variable). 2081 2082Recipes should never populate the sysroot directly (i.e. write files 2083into sysroot). Instead, files should be installed into standard 2084locations during the 2085:ref:`ref-tasks-install` task within 2086the ``${``\ :term:`D`\ ``}`` directory. The 2087reason for this limitation is that almost all files that populate the 2088sysroot are cataloged in manifests in order to ensure the files can be 2089removed later when a recipe is either modified or removed. Thus, the 2090sysroot is able to remain free from stale files. 2091 2092A subset of the files installed by the :ref:`ref-tasks-install` task are 2093used by the :ref:`ref-tasks-populate_sysroot` task as defined by the the 2094:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It 2095is possible to modify the list of directories that populate the sysroot. 2096The following example shows how you could add the ``/opt`` directory to 2097the list of directories within a recipe:: 2098 2099 SYSROOT_DIRS += "/opt" 2100 2101.. note:: 2102 2103 The `/sysroot-only` is to be used by recipes that generate artifacts 2104 that are not included in the target filesystem, allowing them to share 2105 these artifacts without needing to use the :term:`DEPLOY_DIR`. 2106 2107For a more complete description of the :ref:`ref-tasks-populate_sysroot` 2108task and its associated functions, see the 2109:ref:`staging <ref-classes-staging>` class. 2110 2111Using Virtual Providers 2112----------------------- 2113 2114Prior to a build, if you know that several different recipes provide the 2115same functionality, you can use a virtual provider (i.e. ``virtual/*``) 2116as a placeholder for the actual provider. The actual provider is 2117determined at build-time. 2118 2119A common scenario where a virtual provider is used would be for the 2120kernel recipe. Suppose you have three kernel recipes whose 2121:term:`PN` values map to ``kernel-big``, 2122``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes 2123in some way uses a :term:`PROVIDES` 2124statement that essentially identifies itself as being able to provide 2125``virtual/kernel``. Here is one way through the 2126:ref:`kernel <ref-classes-kernel>` class:: 2127 2128 PROVIDES += "virtual/kernel" 2129 2130Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is 2131going to utilize a :term:`PROVIDES` statement that identifies that recipe as 2132being able to provide the ``virtual/kernel`` item. 2133 2134Now comes the time to actually build an image and you need a kernel 2135recipe, but which one? You can configure your build to call out the 2136kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As 2137an example, consider the :yocto_git:`x86-base.inc 2138</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a 2139machine (i.e. :term:`MACHINE`) configuration file. This include file is the 2140reason all x86-based machines use the ``linux-yocto`` kernel. Here are the 2141relevant lines from the include file:: 2142 2143 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" 2144 PREFERRED_VERSION_linux-yocto ??= "4.15%" 2145 2146When you use a virtual provider, you do not have to "hard code" a recipe 2147name as a build dependency. You can use the 2148:term:`DEPENDS` variable to state the 2149build is dependent on ``virtual/kernel`` for example:: 2150 2151 DEPENDS = "virtual/kernel" 2152 2153During the build, the OpenEmbedded build system picks 2154the correct recipe needed for the ``virtual/kernel`` dependency based on 2155the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel 2156mentioned at the beginning of this section, configure your build as 2157follows:: 2158 2159 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small" 2160 2161.. note:: 2162 2163 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not 2164 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these 2165 recipes from building is usually the desired behavior since this mechanism's 2166 purpose is to select between mutually exclusive alternative providers. 2167 2168The following lists specific examples of virtual providers: 2169 2170- ``virtual/kernel``: Provides the name of the kernel recipe to use 2171 when building a kernel image. 2172 2173- ``virtual/bootloader``: Provides the name of the bootloader to use 2174 when building an image. 2175 2176- ``virtual/libgbm``: Provides ``gbm.pc``. 2177 2178- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``. 2179 2180- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL). 2181 2182- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM). 2183 2184- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2). 2185 2186.. note:: 2187 2188 Virtual providers only apply to build time dependencies specified with 2189 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime 2190 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`. 2191 2192Properly Versioning Pre-Release Recipes 2193--------------------------------------- 2194 2195Sometimes the name of a recipe can lead to versioning problems when the 2196recipe is upgraded to a final release. For example, consider the 2197``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in 2198the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section. 2199This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is 2200released, the recipe filename becomes ``irssi_0.8.16.bb``. The version 2201change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the 2202build system and package managers, so the resulting packages will not 2203correctly trigger an upgrade. 2204 2205In order to ensure the versions compare properly, the recommended 2206convention is to set :term:`PV` within the 2207recipe to "previous_version+current_version". You can use an additional 2208variable so that you can use the current version elsewhere. Here is an 2209example:: 2210 2211 REALPV = "0.8.16-rc1" 2212 PV = "0.8.15+${REALPV}" 2213 2214Post-Installation Scripts 2215------------------------- 2216 2217Post-installation scripts run immediately after installing a package on 2218the target or during image creation when a package is included in an 2219image. To add a post-installation script to a package, add a 2220``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file 2221(``.bb``) and replace `PACKAGENAME` with the name of the package you want 2222to attach to the ``postinst`` script. To apply the post-installation 2223script to the main package for the recipe, which is usually what is 2224required, specify 2225``${``\ :term:`PN`\ ``}`` in place of 2226PACKAGENAME. 2227 2228A post-installation function has the following structure:: 2229 2230 pkg_postinst:PACKAGENAME() { 2231 # Commands to carry out 2232 } 2233 2234The script defined in the post-installation function is called when the 2235root filesystem is created. If the script succeeds, the package is 2236marked as installed. 2237 2238.. note:: 2239 2240 Any RPM post-installation script that runs on the target should 2241 return a 0 exit code. RPM does not allow non-zero exit codes for 2242 these scripts, and the RPM package manager will cause the package to 2243 fail installation on the target. 2244 2245Sometimes it is necessary for the execution of a post-installation 2246script to be delayed until the first boot. For example, the script might 2247need to be executed on the device itself. To delay script execution 2248until boot time, you must explicitly mark post installs to defer to the 2249target. You can use ``pkg_postinst_ontarget()`` or call 2250``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any 2251failure of a ``pkg_postinst()`` script (including exit 1) triggers an 2252error during the 2253:ref:`ref-tasks-rootfs` task. 2254 2255If you have recipes that use ``pkg_postinst`` function and they require 2256the use of non-standard native tools that have dependencies during 2257root filesystem construction, you need to use the 2258:term:`PACKAGE_WRITE_DEPS` 2259variable in your recipe to list these tools. If you do not use this 2260variable, the tools might be missing and execution of the 2261post-installation script is deferred until first boot. Deferring the 2262script to the first boot is undesirable and impossible for read-only 2263root filesystems. 2264 2265.. note:: 2266 2267 There is equivalent support for pre-install, pre-uninstall, and post-uninstall 2268 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``, 2269 respectively. These scrips work in exactly the same way as does 2270 ``pkg_postinst`` with the exception that they run at different times. Also, 2271 because of when they run, they are not applicable to being run at image 2272 creation time like ``pkg_postinst``. 2273 2274Testing 2275------- 2276 2277The final step for completing your recipe is to be sure that the 2278software you built runs correctly. To accomplish runtime testing, add 2279the build's output packages to your image and test them on the target. 2280 2281For information on how to customize your image by adding specific 2282packages, see ":ref:`dev-manual/common-tasks:customizing images`" section. 2283 2284Examples 2285-------- 2286 2287To help summarize how to write a recipe, this section provides some 2288examples given various scenarios: 2289 2290- Recipes that use local files 2291 2292- Using an Autotooled package 2293 2294- Using a Makefile-based package 2295 2296- Splitting an application into multiple packages 2297 2298- Adding binaries to an image 2299 2300Single .c File Package (Hello World!) 2301~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2302 2303Building an application from a single file that is stored locally (e.g. 2304under ``files``) requires a recipe that has the file listed in the 2305:term:`SRC_URI` variable. Additionally, you need to manually write the 2306``do_compile`` and ``do_install`` tasks. The :term:`S` variable defines the 2307directory containing the source code, which is set to 2308:term:`WORKDIR` in this case - the 2309directory BitBake uses for the build. 2310:: 2311 2312 SUMMARY = "Simple helloworld application" 2313 SECTION = "examples" 2314 LICENSE = "MIT" 2315 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" 2316 2317 SRC_URI = "file://helloworld.c" 2318 2319 S = "${WORKDIR}" 2320 2321 do_compile() { 2322 ${CC} ${LDFLAGS} helloworld.c -o helloworld 2323 } 2324 2325 do_install() { 2326 install -d ${D}${bindir} 2327 install -m 0755 helloworld ${D}${bindir} 2328 } 2329 2330By default, the ``helloworld``, ``helloworld-dbg``, and 2331``helloworld-dev`` packages are built. For information on how to 2332customize the packaging process, see the 2333":ref:`dev-manual/common-tasks:splitting an application into multiple packages`" 2334section. 2335 2336Autotooled Package 2337~~~~~~~~~~~~~~~~~~ 2338 2339Applications that use Autotools such as ``autoconf`` and ``automake`` 2340require a recipe that has a source archive listed in :term:`SRC_URI` and 2341also inherit the 2342:ref:`autotools <ref-classes-autotools>` class, 2343which contains the definitions of all the steps needed to build an 2344Autotool-based application. The result of the build is automatically 2345packaged. And, if the application uses NLS for localization, packages 2346with local information are generated (one package per language). 2347Following is one example: (``hello_2.3.bb``) 2348:: 2349 2350 SUMMARY = "GNU Helloworld application" 2351 SECTION = "examples" 2352 LICENSE = "GPL-2.0-or-later" 2353 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe" 2354 2355 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz" 2356 2357 inherit autotools gettext 2358 2359The variable :term:`LIC_FILES_CHKSUM` is used to track source license 2360changes as described in the 2361":ref:`dev-manual/common-tasks:tracking license changes`" section in 2362the Yocto Project Overview and Concepts Manual. You can quickly create 2363Autotool-based recipes in a manner similar to the previous example. 2364 2365Makefile-Based Package 2366~~~~~~~~~~~~~~~~~~~~~~ 2367 2368Applications that use GNU ``make`` also require a recipe that has the 2369source archive listed in :term:`SRC_URI`. You do not need to add a 2370``do_compile`` step since by default BitBake starts the ``make`` command 2371to compile the application. If you need additional ``make`` options, you 2372should store them in the 2373:term:`EXTRA_OEMAKE` or 2374:term:`PACKAGECONFIG_CONFARGS` 2375variables. BitBake passes these options into the GNU ``make`` 2376invocation. Note that a ``do_install`` task is still required. 2377Otherwise, BitBake runs an empty ``do_install`` task by default. 2378 2379Some applications might require extra parameters to be passed to the 2380compiler. For example, the application might need an additional header 2381path. You can accomplish this by adding to the :term:`CFLAGS` variable. The 2382following example shows this:: 2383 2384 CFLAGS:prepend = "-I ${S}/include " 2385 2386In the following example, ``lz4`` is a makefile-based package:: 2387 2388 SUMMARY = "Extremely Fast Compression algorithm" 2389 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems." 2390 HOMEPAGE = "https://github.com/lz4/lz4" 2391 2392 LICENSE = "BSD-2-Clause | GPL-2.0-only" 2393 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \ 2394 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \ 2395 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \ 2396 " 2397 2398 PE = "1" 2399 2400 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3" 2401 2402 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \ 2403 file://CVE-2021-3520.patch \ 2404 " 2405 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)" 2406 2407 S = "${WORKDIR}/git" 2408 2409 # Fixed in r118, which is larger than the current version. 2410 CVE_CHECK_IGNORE += "CVE-2014-4715" 2411 2412 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no" 2413 2414 do_install() { 2415 oe_runmake install 2416 } 2417 2418 BBCLASSEXTEND = "native nativesdk" 2419 2420Splitting an Application into Multiple Packages 2421~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2422 2423You can use the variables :term:`PACKAGES` and :term:`FILES` to split an 2424application into multiple packages. 2425 2426Following is an example that uses the ``libxpm`` recipe. By default, 2427this recipe generates a single package that contains the library along 2428with a few binaries. You can modify the recipe to split the binaries 2429into separate packages:: 2430 2431 require xorg-lib-common.inc 2432 2433 SUMMARY = "Xpm: X Pixmap extension library" 2434 LICENSE = "MIT" 2435 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7" 2436 DEPENDS += "libxext libsm libxt" 2437 PE = "1" 2438 2439 XORG_PN = "libXpm" 2440 2441 PACKAGES =+ "sxpm cxpm" 2442 FILES:cxpm = "${bindir}/cxpm" 2443 FILES:sxpm = "${bindir}/sxpm" 2444 2445In the previous example, we want to ship the ``sxpm`` and ``cxpm`` 2446binaries in separate packages. Since ``bindir`` would be packaged into 2447the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable 2448so additional package names are added to the start of list. This results 2449in the extra ``FILES:*`` variables then containing information that 2450define which files and directories go into which packages. Files 2451included by earlier packages are skipped by latter packages. Thus, the 2452main :term:`PN` package does not include the above listed files. 2453 2454Packaging Externally Produced Binaries 2455~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2456 2457Sometimes, you need to add pre-compiled binaries to an image. For 2458example, suppose that there are binaries for proprietary code, 2459created by a particular division of a company. Your part of the company 2460needs to use those binaries as part of an image that you are building 2461using the OpenEmbedded build system. Since you only have the binaries 2462and not the source code, you cannot use a typical recipe that expects to 2463fetch the source specified in 2464:term:`SRC_URI` and then compile it. 2465 2466One method is to package the binaries and then install them as part of 2467the image. Generally, it is not a good idea to package binaries since, 2468among other things, it can hinder the ability to reproduce builds and 2469could lead to compatibility problems with ABI in the future. However, 2470sometimes you have no choice. 2471 2472The easiest solution is to create a recipe that uses the 2473:ref:`bin_package <ref-classes-bin-package>` class 2474and to be sure that you are using default locations for build artifacts. 2475In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the 2476configure and compile steps as well as sets things up to grab packages 2477from the appropriate area. In particular, this class sets ``noexec`` on 2478both the :ref:`ref-tasks-configure` 2479and :ref:`ref-tasks-compile` tasks, 2480sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a 2481:ref:`ref-tasks-install` task, which 2482effectively copies all files from ``${S}`` to ``${D}``. The 2483:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}`` 2484are already laid out in the way they should be laid out on the target. 2485For more information on these variables, see the 2486:term:`FILES`, 2487:term:`PN`, 2488:term:`S`, and 2489:term:`D` variables in the Yocto Project 2490Reference Manual's variable glossary. 2491 2492.. note:: 2493 2494 - Using :term:`DEPENDS` is a good 2495 idea even for components distributed in binary form, and is often 2496 necessary for shared libraries. For a shared library, listing the 2497 library dependencies in :term:`DEPENDS` makes sure that the libraries 2498 are available in the staging sysroot when other recipes link 2499 against the library, which might be necessary for successful 2500 linking. 2501 2502 - Using :term:`DEPENDS` also allows runtime dependencies between 2503 packages to be added automatically. See the 2504 ":ref:`overview-manual/concepts:automatically added runtime dependencies`" 2505 section in the Yocto Project Overview and Concepts Manual for more 2506 information. 2507 2508If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are 2509doing the following: 2510 2511- Create a recipe where the 2512 :ref:`ref-tasks-configure` and 2513 :ref:`ref-tasks-compile` tasks do 2514 nothing: It is usually sufficient to just not define these tasks in 2515 the recipe, because the default implementations do nothing unless a 2516 Makefile is found in 2517 ``${``\ :term:`S`\ ``}``. 2518 2519 If ``${S}`` might contain a Makefile, or if you inherit some class 2520 that replaces ``do_configure`` and ``do_compile`` with custom 2521 versions, then you can use the 2522 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` 2523 flag to turn the tasks into no-ops, as follows:: 2524 2525 do_configure[noexec] = "1" 2526 do_compile[noexec] = "1" 2527 2528 Unlike 2529 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`, 2530 using the flag preserves the dependency chain from the 2531 :ref:`ref-tasks-fetch`, 2532 :ref:`ref-tasks-unpack`, and 2533 :ref:`ref-tasks-patch` tasks to the 2534 :ref:`ref-tasks-install` task. 2535 2536- Make sure your ``do_install`` task installs the binaries 2537 appropriately. 2538 2539- Ensure that you set up :term:`FILES` 2540 (usually 2541 ``FILES:${``\ :term:`PN`\ ``}``) to 2542 point to the files you have installed, which of course depends on 2543 where you have installed them and whether those files are in 2544 different locations than the defaults. 2545 2546Following Recipe Style Guidelines 2547--------------------------------- 2548 2549When writing recipes, it is good to conform to existing style 2550guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page 2551provides rough guidelines for preferred recipe style. 2552 2553It is common for existing recipes to deviate a bit from this style. 2554However, aiming for at least a consistent style is a good idea. Some 2555practices, such as omitting spaces around ``=`` operators in assignments 2556or ordering recipe components in an erratic way, are widely seen as poor 2557style. 2558 2559Recipe Syntax 2560------------- 2561 2562Understanding recipe file syntax is important for writing recipes. The 2563following list overviews the basic items that make up a BitBake recipe 2564file. For more complete BitBake syntax descriptions, see the 2565":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" 2566chapter of the BitBake User Manual. 2567 2568- *Variable Assignments and Manipulations:* Variable assignments allow 2569 a value to be assigned to a variable. The assignment can be static 2570 text or might include the contents of other variables. In addition to 2571 the assignment, appending and prepending operations are also 2572 supported. 2573 2574 The following example shows some of the ways you can use variables in 2575 recipes:: 2576 2577 S = "${WORKDIR}/postfix-${PV}" 2578 CFLAGS += "-DNO_ASM" 2579 SRC_URI:append = " file://fixup.patch" 2580 2581- *Functions:* Functions provide a series of actions to be performed. 2582 You usually use functions to override the default implementation of a 2583 task function or to complement a default function (i.e. append or 2584 prepend to an existing function). Standard functions use ``sh`` shell 2585 syntax, although access to OpenEmbedded variables and internal 2586 methods are also available. 2587 2588 Here is an example function from the ``sed`` recipe:: 2589 2590 do_install () { 2591 autotools_do_install 2592 install -d ${D}${base_bindir} 2593 mv ${D}${bindir}/sed ${D}${base_bindir}/sed 2594 rmdir ${D}${bindir}/ 2595 } 2596 2597 It is 2598 also possible to implement new functions that are called between 2599 existing tasks as long as the new functions are not replacing or 2600 complementing the default functions. You can implement functions in 2601 Python instead of shell. Both of these options are not seen in the 2602 majority of recipes. 2603 2604- *Keywords:* BitBake recipes use only a few keywords. You use keywords 2605 to include common functions (``inherit``), load parts of a recipe 2606 from other files (``include`` and ``require``) and export variables 2607 to the environment (``export``). 2608 2609 The following example shows the use of some of these keywords:: 2610 2611 export POSTCONF = "${STAGING_BINDIR}/postconf" 2612 inherit autoconf 2613 require otherfile.inc 2614 2615- *Comments (#):* Any lines that begin with the hash character (``#``) 2616 are treated as comment lines and are ignored:: 2617 2618 # This is a comment 2619 2620This next list summarizes the most important and most commonly used 2621parts of the recipe syntax. For more information on these parts of the 2622syntax, you can reference the 2623:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter 2624in the BitBake User Manual. 2625 2626- *Line Continuation (\\):* Use the backward slash (``\``) character to 2627 split a statement over multiple lines. Place the slash character at 2628 the end of the line that is to be continued on the next line:: 2629 2630 VAR = "A really long \ 2631 line" 2632 2633 .. note:: 2634 2635 You cannot have any characters including spaces or tabs after the 2636 slash character. 2637 2638- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to 2639 access the contents of a variable:: 2640 2641 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz" 2642 2643 .. note:: 2644 2645 It is important to understand that the value of a variable 2646 expressed in this form does not get substituted automatically. The 2647 expansion of these expressions happens on-demand later (e.g. 2648 usually when a function that makes reference to the variable 2649 executes). This behavior ensures that the values are most 2650 appropriate for the context in which they are finally used. On the 2651 rare occasion that you do need the variable expression to be 2652 expanded immediately, you can use the 2653 := 2654 operator instead of 2655 = 2656 when you make the assignment, but this is not generally needed. 2657 2658- *Quote All Assignments ("value"):* Use double quotes around values in 2659 all variable assignments (e.g. ``"value"``). Following is an example:: 2660 2661 VAR1 = "${OTHERVAR}" 2662 VAR2 = "The version is ${PV}" 2663 2664- *Conditional Assignment (?=):* Conditional assignment is used to 2665 assign a value to a variable, but only when the variable is currently 2666 unset. Use the question mark followed by the equal sign (``?=``) to 2667 make a "soft" assignment used for conditional assignment. Typically, 2668 "soft" assignments are used in the ``local.conf`` file for variables 2669 that are allowed to come through from the external environment. 2670 2671 Here is an example where ``VAR1`` is set to "New value" if it is 2672 currently empty. However, if ``VAR1`` has already been set, it 2673 remains unchanged:: 2674 2675 VAR1 ?= "New value" 2676 2677 In this next example, ``VAR1`` is left with the value "Original value":: 2678 2679 VAR1 = "Original value" 2680 VAR1 ?= "New value" 2681 2682- *Appending (+=):* Use the plus character followed by the equals sign 2683 (``+=``) to append values to existing variables. 2684 2685 .. note:: 2686 2687 This operator adds a space between the existing content of the 2688 variable and the new content. 2689 2690 Here is an example:: 2691 2692 SRC_URI += "file://fix-makefile.patch" 2693 2694- *Prepending (=+):* Use the equals sign followed by the plus character 2695 (``=+``) to prepend values to existing variables. 2696 2697 .. note:: 2698 2699 This operator adds a space between the new content and the 2700 existing content of the variable. 2701 2702 Here is an example:: 2703 2704 VAR =+ "Starts" 2705 2706- *Appending (:append):* Use the ``:append`` operator to append values 2707 to existing variables. This operator does not add any additional 2708 space. Also, the operator is applied after all the ``+=``, and ``=+`` 2709 operators have been applied and after all ``=`` assignments have 2710 occurred. 2711 2712 The following example shows the space being explicitly added to the 2713 start to ensure the appended value is not merged with the existing 2714 value:: 2715 2716 SRC_URI:append = " file://fix-makefile.patch" 2717 2718 You can also use 2719 the ``:append`` operator with overrides, which results in the actions 2720 only being performed for the specified target or machine:: 2721 2722 SRC_URI:append:sh4 = " file://fix-makefile.patch" 2723 2724- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend 2725 values to existing variables. This operator does not add any 2726 additional space. Also, the operator is applied after all the ``+=``, 2727 and ``=+`` operators have been applied and after all ``=`` 2728 assignments have occurred. 2729 2730 The following example shows the space being explicitly added to the 2731 end to ensure the prepended value is not merged with the existing 2732 value:: 2733 2734 CFLAGS:prepend = "-I${S}/myincludes " 2735 2736 You can also use the 2737 ``:prepend`` operator with overrides, which results in the actions 2738 only being performed for the specified target or machine:: 2739 2740 CFLAGS:prepend:sh4 = "-I${S}/myincludes " 2741 2742- *Overrides:* You can use overrides to set a value conditionally, 2743 typically based on how the recipe is being built. For example, to set 2744 the :term:`KBRANCH` variable's 2745 value to "standard/base" for any target 2746 :term:`MACHINE`, except for 2747 qemuarm where it should be set to "standard/arm-versatile-926ejs", 2748 you would do the following:: 2749 2750 KBRANCH = "standard/base" 2751 KBRANCH:qemuarm = "standard/arm-versatile-926ejs" 2752 2753 Overrides are also used to separate 2754 alternate values of a variable in other situations. For example, when 2755 setting variables such as 2756 :term:`FILES` and 2757 :term:`RDEPENDS` that are 2758 specific to individual packages produced by a recipe, you should 2759 always use an override that specifies the name of the package. 2760 2761- *Indentation:* Use spaces for indentation rather than tabs. For 2762 shell functions, both currently work. However, it is a policy 2763 decision of the Yocto Project to use tabs in shell functions. Realize 2764 that some layers have a policy to use spaces for all indentation. 2765 2766- *Using Python for Complex Operations:* For more advanced processing, 2767 it is possible to use Python code during variable assignments (e.g. 2768 search and replacement on a variable). 2769 2770 You indicate Python code using the ``${@python_code}`` syntax for the 2771 variable assignment:: 2772 2773 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz 2774 2775- *Shell Function Syntax:* Write shell functions as if you were writing 2776 a shell script when you describe a list of actions to take. You 2777 should ensure that your script works with a generic ``sh`` and that 2778 it does not require any ``bash`` or other shell-specific 2779 functionality. The same considerations apply to various system 2780 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you 2781 might wish to use. If in doubt, you should check with multiple 2782 implementations - including those from BusyBox. 2783 2784Adding a New Machine 2785==================== 2786 2787Adding a new machine to the Yocto Project is a straightforward process. 2788This section describes how to add machines that are similar to those 2789that the Yocto Project already supports. 2790 2791.. note:: 2792 2793 Although well within the capabilities of the Yocto Project, adding a 2794 totally new architecture might require changes to ``gcc``/``glibc`` 2795 and to the site information, which is beyond the scope of this 2796 manual. 2797 2798For a complete example that shows how to add a new machine, see the 2799":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`" 2800section in the Yocto Project Board Support Package (BSP) Developer's 2801Guide. 2802 2803Adding the Machine Configuration File 2804------------------------------------- 2805 2806To add a new machine, you need to add a new machine configuration file 2807to the layer's ``conf/machine`` directory. This configuration file 2808provides details about the device you are adding. 2809 2810The OpenEmbedded build system uses the root name of the machine 2811configuration file to reference the new machine. For example, given a 2812machine configuration file named ``crownbay.conf``, the build system 2813recognizes the machine as "crownbay". 2814 2815The most important variables you must set in your machine configuration 2816file or include from a lower-level configuration file are as follows: 2817 2818- :term:`TARGET_ARCH` (e.g. "arm") 2819 2820- ``PREFERRED_PROVIDER_virtual/kernel`` 2821 2822- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi") 2823 2824You might also need these variables: 2825 2826- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1") 2827 2828- :term:`KERNEL_IMAGETYPE` (e.g. "zImage") 2829 2830- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2") 2831 2832You can find full details on these variables in the reference section. 2833You can leverage existing machine ``.conf`` files from 2834``meta-yocto-bsp/conf/machine/``. 2835 2836Adding a Kernel for the Machine 2837------------------------------- 2838 2839The OpenEmbedded build system needs to be able to build a kernel for the 2840machine. You need to either create a new kernel recipe for this machine, 2841or extend an existing kernel recipe. You can find several kernel recipe 2842examples in the Source Directory at ``meta/recipes-kernel/linux`` that 2843you can use as references. 2844 2845If you are creating a new kernel recipe, normal recipe-writing rules 2846apply for setting up a :term:`SRC_URI`. Thus, you need to specify any 2847necessary patches and set :term:`S` to point at the source code. You need to 2848create a ``do_configure`` task that configures the unpacked kernel with 2849a ``defconfig`` file. You can do this by using a ``make defconfig`` 2850command or, more commonly, by copying in a suitable ``defconfig`` file 2851and then running ``make oldconfig``. By making use of ``inherit kernel`` 2852and potentially some of the ``linux-*.inc`` files, most other 2853functionality is centralized and the defaults of the class normally work 2854well. 2855 2856If you are extending an existing kernel recipe, it is usually a matter 2857of adding a suitable ``defconfig`` file. The file needs to be added into 2858a location similar to ``defconfig`` files used for other machines in a 2859given kernel recipe. A possible way to do this is by listing the file in 2860the :term:`SRC_URI` and adding the machine to the expression in 2861:term:`COMPATIBLE_MACHINE`:: 2862 2863 COMPATIBLE_MACHINE = '(qemux86|qemumips)' 2864 2865For more information on ``defconfig`` files, see the 2866":ref:`kernel-dev/common:changing the configuration`" 2867section in the Yocto Project Linux Kernel Development Manual. 2868 2869Adding a Formfactor Configuration File 2870-------------------------------------- 2871 2872A formfactor configuration file provides information about the target 2873hardware for which the image is being built and information that the 2874build system cannot obtain from other sources such as the kernel. Some 2875examples of information contained in a formfactor configuration file 2876include framebuffer orientation, whether or not the system has a 2877keyboard, the positioning of the keyboard in relation to the screen, and 2878the screen resolution. 2879 2880The build system uses reasonable defaults in most cases. However, if 2881customization is necessary, you need to create a ``machconfig`` file in 2882the ``meta/recipes-bsp/formfactor/files`` directory. This directory 2883contains directories for specific machines such as ``qemuarm`` and 2884``qemux86``. For information about the settings available and the 2885defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file 2886found in the same area. 2887 2888Following is an example for "qemuarm" machine:: 2889 2890 HAVE_TOUCHSCREEN=1 2891 HAVE_KEYBOARD=1 2892 DISPLAY_CAN_ROTATE=0 2893 DISPLAY_ORIENTATION=0 2894 #DISPLAY_WIDTH_PIXELS=640 2895 #DISPLAY_HEIGHT_PIXELS=480 2896 #DISPLAY_BPP=16 2897 DISPLAY_DPI=150 2898 DISPLAY_SUBPIXEL_ORDER=vrgb 2899 2900Upgrading Recipes 2901================= 2902 2903Over time, upstream developers publish new versions for software built 2904by layer recipes. It is recommended to keep recipes up-to-date with 2905upstream version releases. 2906 2907While there are several methods to upgrade a recipe, you might 2908consider checking on the upgrade status of a recipe first. You can do so 2909using the ``devtool check-upgrade-status`` command. See the 2910":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`" 2911section in the Yocto Project Reference Manual for more information. 2912 2913The remainder of this section describes three ways you can upgrade a 2914recipe. You can use the Automated Upgrade Helper (AUH) to set up 2915automatic version upgrades. Alternatively, you can use 2916``devtool upgrade`` to set up semi-automatic version upgrades. Finally, 2917you can manually upgrade a recipe by editing the recipe itself. 2918 2919Using the Auto Upgrade Helper (AUH) 2920----------------------------------- 2921 2922The AUH utility works in conjunction with the OpenEmbedded build system 2923in order to automatically generate upgrades for recipes based on new 2924versions being published upstream. Use AUH when you want to create a 2925service that performs the upgrades automatically and optionally sends 2926you an email with the results. 2927 2928AUH allows you to update several recipes with a single use. You can also 2929optionally perform build and integration tests using images with the 2930results saved to your hard drive and emails of results optionally sent 2931to recipe maintainers. Finally, AUH creates Git commits with appropriate 2932commit messages in the layer's tree for the changes made to recipes. 2933 2934.. note:: 2935 2936 In some conditions, you should not use AUH to upgrade recipes 2937 and should instead use either ``devtool upgrade`` or upgrade your 2938 recipes manually: 2939 2940 - When AUH cannot complete the upgrade sequence. This situation 2941 usually results because custom patches carried by the recipe 2942 cannot be automatically rebased to the new version. In this case, 2943 ``devtool upgrade`` allows you to manually resolve conflicts. 2944 2945 - When for any reason you want fuller control over the upgrade 2946 process. For example, when you want special arrangements for 2947 testing. 2948 2949The following steps describe how to set up the AUH utility: 2950 29511. *Be Sure the Development Host is Set Up:* You need to be sure that 2952 your development host is set up to use the Yocto Project. For 2953 information on how to set up your host, see the 2954 ":ref:`dev-manual/start:Preparing the Build Host`" section. 2955 29562. *Make Sure Git is Configured:* The AUH utility requires Git to be 2957 configured because AUH uses Git to save upgrades. Thus, you must have 2958 Git user and email configured. The following command shows your 2959 configurations:: 2960 2961 $ git config --list 2962 2963 If you do not have the user and 2964 email configured, you can use the following commands to do so:: 2965 2966 $ git config --global user.name some_name 2967 $ git config --global user.email username@domain.com 2968 29693. *Clone the AUH Repository:* To use AUH, you must clone the repository 2970 onto your development host. The following command uses Git to create 2971 a local copy of the repository on your system:: 2972 2973 $ git clone git://git.yoctoproject.org/auto-upgrade-helper 2974 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done. 2975 remote: Compressing objects: 100% (300/300), done. 2976 remote: Total 768 (delta 499), reused 703 (delta 434) 2977 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done. 2978 Resolving deltas: 100% (499/499), done. 2979 Checking connectivity... done. 2980 2981 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or 2982 :term:`Poky` repositories. 2983 29844. *Create a Dedicated Build Directory:* Run the 2985 :ref:`structure-core-script` 2986 script to create a fresh build directory that you use exclusively for 2987 running the AUH utility:: 2988 2989 $ cd poky 2990 $ source oe-init-build-env your_AUH_build_directory 2991 2992 Re-using an existing build directory and its configurations is not 2993 recommended as existing settings could cause AUH to fail or behave 2994 undesirably. 2995 29965. *Make Configurations in Your Local Configuration File:* Several 2997 settings are needed in the ``local.conf`` file in the build 2998 directory you just created for AUH. Make these following 2999 configurations: 3000 3001 - If you want to enable :ref:`Build 3002 History <dev-manual/common-tasks:maintaining build output quality>`, 3003 which is optional, you need the following lines in the 3004 ``conf/local.conf`` file:: 3005 3006 INHERIT =+ "buildhistory" 3007 BUILDHISTORY_COMMIT = "1" 3008 3009 With this configuration and a successful 3010 upgrade, a build history "diff" file appears in the 3011 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in 3012 your build directory. 3013 3014 - If you want to enable testing through the 3015 :ref:`testimage <ref-classes-testimage*>` 3016 class, which is optional, you need to have the following set in 3017 your ``conf/local.conf`` file:: 3018 3019 INHERIT += "testimage" 3020 3021 .. note:: 3022 3023 If your distro does not enable by default ptest, which Poky 3024 does, you need the following in your ``local.conf`` file:: 3025 3026 DISTRO_FEATURES:append = " ptest" 3027 3028 30296. *Optionally Start a vncserver:* If you are running in a server 3030 without an X11 session, you need to start a vncserver:: 3031 3032 $ vncserver :1 3033 $ export DISPLAY=:1 3034 30357. *Create and Edit an AUH Configuration File:* You need to have the 3036 ``upgrade-helper/upgrade-helper.conf`` configuration file in your 3037 build directory. You can find a sample configuration file in the 3038 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`. 3039 3040 Read through the sample file and make configurations as needed. For 3041 example, if you enabled build history in your ``local.conf`` as 3042 described earlier, you must enable it in ``upgrade-helper.conf``. 3043 3044 Also, if you are using the default ``maintainers.inc`` file supplied 3045 with Poky and located in ``meta-yocto`` and you do not set a 3046 "maintainers_whitelist" or "global_maintainer_override" in the 3047 ``upgrade-helper.conf`` configuration, and you specify "-e all" on 3048 the AUH command-line, the utility automatically sends out emails to 3049 all the default maintainers. Please avoid this. 3050 3051This next set of examples describes how to use the AUH: 3052 3053- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the 3054 following form:: 3055 3056 $ upgrade-helper.py recipe_name 3057 3058 For example, this command upgrades the ``xmodmap`` recipe:: 3059 3060 $ upgrade-helper.py xmodmap 3061 3062- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a 3063 specific recipe to a particular version, use the following form:: 3064 3065 $ upgrade-helper.py recipe_name -t version 3066 3067 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3:: 3068 3069 $ upgrade-helper.py xmodmap -t 1.2.3 3070 3071- *Upgrading all Recipes to the Latest Versions and Suppressing Email 3072 Notifications:* To upgrade all recipes to their most recent versions 3073 and suppress the email notifications, use the following command:: 3074 3075 $ upgrade-helper.py all 3076 3077- *Upgrading all Recipes to the Latest Versions and Send Email 3078 Notifications:* To upgrade all recipes to their most recent versions 3079 and send email messages to maintainers for each attempted recipe as 3080 well as a status email, use the following command:: 3081 3082 $ upgrade-helper.py -e all 3083 3084Once you have run the AUH utility, you can find the results in the AUH 3085build directory:: 3086 3087 ${BUILDDIR}/upgrade-helper/timestamp 3088 3089The AUH utility 3090also creates recipe update commits from successful upgrade attempts in 3091the layer tree. 3092 3093You can easily set up to run the AUH utility on a regular basis by using 3094a cron job. See the 3095:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>` 3096file distributed with the utility for an example. 3097 3098Using ``devtool upgrade`` 3099------------------------- 3100 3101As mentioned earlier, an alternative method for upgrading recipes to 3102newer versions is to use 3103:doc:`devtool upgrade </ref-manual/devtool-reference>`. 3104You can read about ``devtool upgrade`` in general in the 3105":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`" 3106section in the Yocto Project Application Development and the Extensible 3107Software Development Kit (eSDK) Manual. 3108 3109To see all the command-line options available with ``devtool upgrade``, 3110use the following help command:: 3111 3112 $ devtool upgrade -h 3113 3114If you want to find out what version a recipe is currently at upstream 3115without any attempt to upgrade your local version of the recipe, you can 3116use the following command:: 3117 3118 $ devtool latest-version recipe_name 3119 3120As mentioned in the previous section describing AUH, ``devtool upgrade`` 3121works in a less-automated manner than AUH. Specifically, 3122``devtool upgrade`` only works on a single recipe that you name on the 3123command line, cannot perform build and integration testing using images, 3124and does not automatically generate commits for changes in the source 3125tree. Despite all these "limitations", ``devtool upgrade`` updates the 3126recipe file to the new upstream version and attempts to rebase custom 3127patches contained by the recipe as needed. 3128 3129.. note:: 3130 3131 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat 3132 of a "wrapper" application for ``devtool upgrade``. 3133 3134A typical scenario involves having used Git to clone an upstream 3135repository that you use during build operations. Because you have built the 3136recipe in the past, the layer is likely added to your 3137configuration already. If for some reason, the layer is not added, you 3138could add it easily using the 3139":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`" 3140script. For example, suppose you use the ``nano.bb`` recipe from the 3141``meta-oe`` layer in the ``meta-openembedded`` repository. For this 3142example, assume that the layer has been cloned into following area:: 3143 3144 /home/scottrif/meta-openembedded 3145 3146The following command from your 3147:term:`Build Directory` adds the layer to 3148your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):: 3149 3150 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe 3151 NOTE: Starting bitbake server... 3152 Parsing recipes: 100% |##########################################| Time: 0:00:55 3153 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. 3154 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00 3155 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00 3156 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00 3157 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00 3158 3159For this example, assume that the ``nano.bb`` recipe that 3160is upstream has a 2.9.3 version number. However, the version in the 3161local repository is 2.7.4. The following command from your build 3162directory automatically upgrades the recipe for you: 3163 3164.. note:: 3165 3166 Using the ``-V`` option is not necessary. Omitting the version number causes 3167 ``devtool upgrade`` to upgrade the recipe to the most recent version. 3168 3169:: 3170 3171 $ devtool upgrade nano -V 2.9.3 3172 NOTE: Starting bitbake server... 3173 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace 3174 Parsing recipes: 100% |##########################################| Time: 0:00:46 3175 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. 3176 NOTE: Extracting current version source... 3177 NOTE: Resolving any missing task queue dependencies 3178 . 3179 . 3180 . 3181 NOTE: Executing SetScene Tasks 3182 NOTE: Executing RunQueue Tasks 3183 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded. 3184 Adding changed files: 100% |#####################################| Time: 0:00:00 3185 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano 3186 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb 3187 3188Continuing with this example, you can use ``devtool build`` to build the 3189newly upgraded recipe:: 3190 3191 $ devtool build nano 3192 NOTE: Starting bitbake server... 3193 Loading cache: 100% |################################################################################################| Time: 0:00:01 3194 Loaded 2040 entries from dependency cache. 3195 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00 3196 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. 3197 NOTE: Resolving any missing task queue dependencies 3198 . 3199 . 3200 . 3201 NOTE: Executing SetScene Tasks 3202 NOTE: Executing RunQueue Tasks 3203 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano 3204 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded. 3205 3206Within the ``devtool upgrade`` workflow, you can 3207deploy and test your rebuilt software. For this example, 3208however, running ``devtool finish`` cleans up the workspace once the 3209source in your workspace is clean. This usually means using Git to stage 3210and submit commits for the changes generated by the upgrade process. 3211 3212Once the tree is clean, you can clean things up in this example with the 3213following command from the ``${BUILDDIR}/workspace/sources/nano`` 3214directory:: 3215 3216 $ devtool finish nano meta-oe 3217 NOTE: Starting bitbake server... 3218 Loading cache: 100% |################################################################################################| Time: 0:00:00 3219 Loaded 2040 entries from dependency cache. 3220 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01 3221 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. 3222 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch 3223 NOTE: Updating recipe nano_2.9.3.bb 3224 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb 3225 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano 3226 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually 3227 3228 3229Using the ``devtool finish`` command cleans up the workspace and creates a patch 3230file based on your commits. The tool puts all patch files back into the 3231source directory in a sub-directory named ``nano`` in this case. 3232 3233Manually Upgrading a Recipe 3234--------------------------- 3235 3236If for some reason you choose not to upgrade recipes using 3237:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or 3238by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``, 3239you can manually edit the recipe files to upgrade the versions. 3240 3241.. note:: 3242 3243 Manually updating multiple recipes scales poorly and involves many 3244 steps. The recommendation to upgrade recipe versions is through AUH 3245 or ``devtool upgrade``, both of which automate some steps and provide 3246 guidance for others needed for the manual process. 3247 3248To manually upgrade recipe versions, follow these general steps: 3249 32501. *Change the Version:* Rename the recipe such that the version (i.e. 3251 the :term:`PV` part of the recipe name) 3252 changes appropriately. If the version is not part of the recipe name, 3253 change the value as it is set for :term:`PV` within the recipe itself. 3254 32552. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds 3256 is fetched from Git or some other version control system, update 3257 :term:`SRCREV` to point to the 3258 commit hash that matches the new version. 3259 32603. *Build the Software:* Try to build the recipe using BitBake. Typical 3261 build failures include the following: 3262 3263 - License statements were updated for the new version. For this 3264 case, you need to review any changes to the license and update the 3265 values of :term:`LICENSE` and 3266 :term:`LIC_FILES_CHKSUM` 3267 as needed. 3268 3269 .. note:: 3270 3271 License changes are often inconsequential. For example, the 3272 license text's copyright year might have changed. 3273 3274 - Custom patches carried by the older version of the recipe might 3275 fail to apply to the new version. For these cases, you need to 3276 review the failures. Patches might not be necessary for the new 3277 version of the software if the upgraded version has fixed those 3278 issues. If a patch is necessary and failing, you need to rebase it 3279 into the new version. 3280 32814. *Optionally Attempt to Build for Several Architectures:* Once you 3282 successfully build the new software for a given architecture, you 3283 could test the build for other architectures by changing the 3284 :term:`MACHINE` variable and 3285 rebuilding the software. This optional step is especially important 3286 if the recipe is to be released publicly. 3287 32885. *Check the Upstream Change Log or Release Notes:* Checking both these 3289 reveals if there are new features that could break 3290 backwards-compatibility. If so, you need to take steps to mitigate or 3291 eliminate that situation. 3292 32936. *Optionally Create a Bootable Image and Test:* If you want, you can 3294 test the new software by booting it onto actual hardware. 3295 32967. *Create a Commit with the Change in the Layer Repository:* After all 3297 builds work and any testing is successful, you can create commits for 3298 any changes in the layer holding your upgraded recipe. 3299 3300Finding Temporary Source Code 3301============================= 3302 3303You might find it helpful during development to modify the temporary 3304source code used by recipes to build packages. For example, suppose you 3305are developing a patch and you need to experiment a bit to figure out 3306your solution. After you have initially built the package, you can 3307iteratively tweak the source code, which is located in the 3308:term:`Build Directory`, and then you can 3309force a re-compile and quickly test your altered code. Once you settle 3310on a solution, you can then preserve your changes in the form of 3311patches. 3312 3313During a build, the unpacked temporary source code used by recipes to 3314build packages is available in the Build Directory as defined by the 3315:term:`S` variable. Below is the default 3316value for the :term:`S` variable as defined in the 3317``meta/conf/bitbake.conf`` configuration file in the 3318:term:`Source Directory`:: 3319 3320 S = "${WORKDIR}/${BP}" 3321 3322You should be aware that many recipes override the 3323:term:`S` variable. For example, recipes that fetch their source from Git 3324usually set :term:`S` to ``${WORKDIR}/git``. 3325 3326.. note:: 3327 3328 The :term:`BP` represents the base recipe name, which consists of the name 3329 and version:: 3330 3331 BP = "${BPN}-${PV}" 3332 3333 3334The path to the work directory for the recipe 3335(:term:`WORKDIR`) is defined as 3336follows:: 3337 3338 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR} 3339 3340The actual directory depends on several things: 3341 3342- :term:`TMPDIR`: The top-level build 3343 output directory. 3344 3345- :term:`MULTIMACH_TARGET_SYS`: 3346 The target system identifier. 3347 3348- :term:`PN`: The recipe name. 3349 3350- :term:`EXTENDPE`: The epoch - (if 3351 :term:`PE` is not specified, which is 3352 usually the case for most recipes, then :term:`EXTENDPE` is blank). 3353 3354- :term:`PV`: The recipe version. 3355 3356- :term:`PR`: The recipe revision. 3357 3358As an example, assume a Source Directory top-level folder named 3359``poky``, a default Build Directory at ``poky/build``, and a 3360``qemux86-poky-linux`` machine target system. Furthermore, suppose your 3361recipe is named ``foo_1.3.0.bb``. In this case, the work directory the 3362build system uses to build the package would be as follows:: 3363 3364 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0 3365 3366Using Quilt in Your Workflow 3367============================ 3368 3369`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool 3370that allows you to capture source code changes without having a clean 3371source tree. This section outlines the typical workflow you can use to 3372modify source code, test changes, and then preserve the changes in the 3373form of a patch all using Quilt. 3374 3375.. note:: 3376 3377 With regard to preserving changes to source files, if you clean a 3378 recipe or have ``rm_work`` enabled, the 3379 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>` 3380 as described in the Yocto Project Application Development and the 3381 Extensible Software Development Kit (eSDK) manual is a safer 3382 development flow than the flow that uses Quilt. 3383 3384Follow these general steps: 3385 33861. *Find the Source Code:* Temporary source code used by the 3387 OpenEmbedded build system is kept in the 3388 :term:`Build Directory`. See the 3389 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to 3390 learn how to locate the directory that has the temporary source code for a 3391 particular package. 3392 33932. *Change Your Working Directory:* You need to be in the directory that 3394 has the temporary source code. That directory is defined by the 3395 :term:`S` variable. 3396 33973. *Create a New Patch:* Before modifying source code, you need to 3398 create a new patch. To create a new patch file, use ``quilt new`` as 3399 below:: 3400 3401 $ quilt new my_changes.patch 3402 34034. *Notify Quilt and Add Files:* After creating the patch, you need to 3404 notify Quilt about the files you plan to edit. You notify Quilt by 3405 adding the files to the patch you just created:: 3406 3407 $ quilt add file1.c file2.c file3.c 3408 34095. *Edit the Files:* Make your changes in the source code to the files 3410 you added to the patch. 3411 34126. *Test Your Changes:* Once you have modified the source code, the 3413 easiest way to test your changes is by calling the ``do_compile`` 3414 task as shown in the following example:: 3415 3416 $ bitbake -c compile -f package 3417 3418 The ``-f`` or ``--force`` option forces the specified task to 3419 execute. If you find problems with your code, you can just keep 3420 editing and re-testing iteratively until things work as expected. 3421 3422 .. note:: 3423 3424 All the modifications you make to the temporary source code disappear 3425 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake 3426 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``). 3427 Modifications will also disappear if you use the ``rm_work`` feature as 3428 described in the 3429 ":ref:`dev-manual/common-tasks:conserving disk space during builds`" 3430 section. 3431 34327. *Generate the Patch:* Once your changes work as expected, you need to 3433 use Quilt to generate the final patch that contains all your 3434 modifications. 3435 :: 3436 3437 $ quilt refresh 3438 3439 At this point, the 3440 ``my_changes.patch`` file has all your edits made to the ``file1.c``, 3441 ``file2.c``, and ``file3.c`` files. 3442 3443 You can find the resulting patch file in the ``patches/`` 3444 subdirectory of the source (:term:`S`) directory. 3445 34468. *Copy the Patch File:* For simplicity, copy the patch file into a 3447 directory named ``files``, which you can create in the same directory 3448 that holds the recipe (``.bb``) file or the append (``.bbappend``) 3449 file. Placing the patch here guarantees that the OpenEmbedded build 3450 system will find the patch. Next, add the patch into the :term:`SRC_URI` 3451 of the recipe. Here is an example:: 3452 3453 SRC_URI += "file://my_changes.patch" 3454 3455Using a Development Shell 3456========================= 3457 3458When debugging certain commands or even when just editing packages, 3459``devshell`` can be a useful tool. When you invoke ``devshell``, all 3460tasks up to and including 3461:ref:`ref-tasks-patch` are run for the 3462specified target. Then, a new terminal is opened and you are placed in 3463``${``\ :term:`S`\ ``}``, the source 3464directory. In the new terminal, all the OpenEmbedded build-related 3465environment variables are still defined so you can use commands such as 3466``configure`` and ``make``. The commands execute just as if the 3467OpenEmbedded build system were executing them. Consequently, working 3468this way can be helpful when debugging a build or preparing software to 3469be used with the OpenEmbedded build system. 3470 3471Following is an example that uses ``devshell`` on a target named 3472``matchbox-desktop``:: 3473 3474 $ bitbake matchbox-desktop -c devshell 3475 3476This command spawns a terminal with a shell prompt within the 3477OpenEmbedded build environment. The 3478:term:`OE_TERMINAL` variable 3479controls what type of shell is opened. 3480 3481For spawned terminals, the following occurs: 3482 3483- The ``PATH`` variable includes the cross-toolchain. 3484 3485- The ``pkgconfig`` variables find the correct ``.pc`` files. 3486 3487- The ``configure`` command finds the Yocto Project site files as well 3488 as any other necessary files. 3489 3490Within this environment, you can run configure or compile commands as if 3491they were being run by the OpenEmbedded build system itself. As noted 3492earlier, the working directory also automatically changes to the Source 3493Directory (:term:`S`). 3494 3495To manually run a specific task using ``devshell``, run the 3496corresponding ``run.*`` script in the 3497``${``\ :term:`WORKDIR`\ ``}/temp`` 3498directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does 3499not exist, which would be the case if the task was skipped by way of the 3500sstate cache, you can create the task by first running it outside of the 3501``devshell``:: 3502 3503 $ bitbake -c task 3504 3505.. note:: 3506 3507 - Execution of a task's ``run.*`` script and BitBake's execution of 3508 a task are identical. In other words, running the script re-runs 3509 the task just as it would be run using the ``bitbake -c`` command. 3510 3511 - Any ``run.*`` file that does not have a ``.pid`` extension is a 3512 symbolic link (symlink) to the most recent version of that file. 3513 3514Remember, that the ``devshell`` is a mechanism that allows you to get 3515into the BitBake task execution environment. And as such, all commands 3516must be called just as BitBake would call them. That means you need to 3517provide the appropriate options for cross-compilation and so forth as 3518applicable. 3519 3520When you are finished using ``devshell``, exit the shell or close the 3521terminal window. 3522 3523.. note:: 3524 3525 - It is worth remembering that when using ``devshell`` you need to 3526 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc`` 3527 instead of just using ``gcc``. The same applies to other 3528 applications such as ``binutils``, ``libtool`` and so forth. 3529 BitBake sets up environment variables such as :term:`CC` to assist 3530 applications, such as ``make`` to find the correct tools. 3531 3532 - It is also worth noting that ``devshell`` still works over X11 3533 forwarding and similar situations. 3534 3535Using a Python Development Shell 3536================================ 3537 3538Similar to working within a development shell as described in the 3539previous section, you can also spawn and work within an interactive 3540Python development shell. When debugging certain commands or even when 3541just editing packages, ``pydevshell`` can be a useful tool. When you 3542invoke the ``pydevshell`` task, all tasks up to and including 3543:ref:`ref-tasks-patch` are run for the 3544specified target. Then a new terminal is opened. Additionally, key 3545Python objects and code are available in the same way they are to 3546BitBake tasks, in particular, the data store 'd'. So, commands such as 3547the following are useful when exploring the data store and running 3548functions:: 3549 3550 pydevshell> d.getVar("STAGING_DIR") 3551 '/media/build1/poky/build/tmp/sysroots' 3552 pydevshell> d.getVar("STAGING_DIR", False) 3553 '${TMPDIR}/sysroots' 3554 pydevshell> d.setVar("FOO", "bar") 3555 pydevshell> d.getVar("FOO") 3556 'bar' 3557 pydevshell> d.delVar("FOO") 3558 pydevshell> d.getVar("FOO") 3559 pydevshell> bb.build.exec_func("do_unpack", d) 3560 pydevshell> 3561 3562The commands execute just as if the OpenEmbedded build 3563system were executing them. Consequently, working this way can be 3564helpful when debugging a build or preparing software to be used with the 3565OpenEmbedded build system. 3566 3567Following is an example that uses ``pydevshell`` on a target named 3568``matchbox-desktop``:: 3569 3570 $ bitbake matchbox-desktop -c pydevshell 3571 3572This command spawns a terminal and places you in an interactive Python 3573interpreter within the OpenEmbedded build environment. The 3574:term:`OE_TERMINAL` variable 3575controls what type of shell is opened. 3576 3577When you are finished using ``pydevshell``, you can exit the shell 3578either by using Ctrl+d or closing the terminal window. 3579 3580Building 3581======== 3582 3583This section describes various build procedures, such as the steps 3584needed for a simple build, building a target for multiple configurations, 3585generating an image for more than one machine, and so forth. 3586 3587Building a Simple Image 3588----------------------- 3589 3590In the development environment, you need to build an image whenever you 3591change hardware support, add or change system libraries, or add or 3592change services that have dependencies. There are several methods that allow 3593you to build an image within the Yocto Project. This section presents 3594the basic steps you need to build a simple image using BitBake from a 3595build host running Linux. 3596 3597.. note:: 3598 3599 - For information on how to build an image using 3600 :term:`Toaster`, see the 3601 :doc:`/toaster-manual/index`. 3602 3603 - For information on how to use ``devtool`` to build images, see the 3604 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`" 3605 section in the Yocto Project Application Development and the 3606 Extensible Software Development Kit (eSDK) manual. 3607 3608 - For a quick example on how to build an image using the 3609 OpenEmbedded build system, see the 3610 :doc:`/brief-yoctoprojectqs/index` document. 3611 3612The build process creates an entire Linux distribution from source and 3613places it in your :term:`Build Directory` under 3614``tmp/deploy/images``. For detailed information on the build process 3615using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the 3616Yocto Project Overview and Concepts Manual. 3617 3618The following figure and list overviews the build process: 3619 3620.. image:: figures/bitbake-build-flow.png 3621 :align: center 3622 36231. *Set up Your Host Development System to Support Development Using the 3624 Yocto Project*: See the ":doc:`start`" section for options on how to get a 3625 build host ready to use the Yocto Project. 3626 36272. *Initialize the Build Environment:* Initialize the build environment 3628 by sourcing the build environment script (i.e. 3629 :ref:`structure-core-script`):: 3630 3631 $ source oe-init-build-env [build_dir] 3632 3633 When you use the initialization script, the OpenEmbedded build system 3634 uses ``build`` as the default :term:`Build Directory` in your current work 3635 directory. You can use a `build_dir` argument with the script to 3636 specify a different build directory. 3637 3638 .. note:: 3639 3640 A common practice is to use a different Build Directory for 3641 different targets; for example, ``~/build/x86`` for a ``qemux86`` 3642 target, and ``~/build/arm`` for a ``qemuarm`` target. In any 3643 event, it's typically cleaner to locate the build directory 3644 somewhere outside of your source directory. 3645 36463. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the 3647 ``conf/local.conf`` configuration file, which is found in the Build 3648 Directory, is set up how you want it. This file defines many aspects 3649 of the build environment including the target machine architecture 3650 through the :term:`MACHINE` variable, the packaging format used during 3651 the build 3652 (:term:`PACKAGE_CLASSES`), 3653 and a centralized tarball download directory through the 3654 :term:`DL_DIR` variable. 3655 36564. *Build the Image:* Build the image using the ``bitbake`` command:: 3657 3658 $ bitbake target 3659 3660 .. note:: 3661 3662 For information on BitBake, see the :doc:`bitbake:index`. 3663 3664 The target is the name of the recipe you want to build. Common 3665 targets are the images in ``meta/recipes-core/images``, 3666 ``meta/recipes-sato/images``, and so forth all found in the 3667 :term:`Source Directory`. Alternatively, the target 3668 can be the name of a recipe for a specific piece of software such as 3669 BusyBox. For more details about the images the OpenEmbedded build 3670 system supports, see the 3671 ":ref:`ref-manual/images:Images`" chapter in the Yocto 3672 Project Reference Manual. 3673 3674 As an example, the following command builds the 3675 ``core-image-minimal`` image:: 3676 3677 $ bitbake core-image-minimal 3678 3679 Once an 3680 image has been built, it often needs to be installed. The images and 3681 kernels built by the OpenEmbedded build system are placed in the 3682 Build Directory in ``tmp/deploy/images``. For information on how to 3683 run pre-built images such as ``qemux86`` and ``qemuarm``, see the 3684 :doc:`/sdk-manual/index` manual. For 3685 information about how to install these images, see the documentation 3686 for your particular board or machine. 3687 3688Building Images for Multiple Targets Using Multiple Configurations 3689------------------------------------------------------------------ 3690 3691You can use a single ``bitbake`` command to build multiple images or 3692packages for different targets where each image or package requires a 3693different configuration (multiple configuration builds). The builds, in 3694this scenario, are sometimes referred to as "multiconfigs", and this 3695section uses that term throughout. 3696 3697This section describes how to set up for multiple configuration builds 3698and how to account for cross-build dependencies between the 3699multiconfigs. 3700 3701Setting Up and Running a Multiple Configuration Build 3702~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3703 3704To accomplish a multiple configuration build, you must define each 3705target's configuration separately using a parallel configuration file in 3706the :term:`Build Directory`, and you 3707must follow a required file hierarchy. Additionally, you must enable the 3708multiple configuration builds in your ``local.conf`` file. 3709 3710Follow these steps to set up and execute multiple configuration builds: 3711 3712- *Create Separate Configuration Files*: You need to create a single 3713 configuration file for each build target (each multiconfig). 3714 Minimally, each configuration file must define the machine and the 3715 temporary directory BitBake uses for the build. Suggested practice 3716 dictates that you do not overlap the temporary directories used 3717 during the builds. However, it is possible that you can share the 3718 temporary directory 3719 (:term:`TMPDIR`). For example, 3720 consider a scenario with two different multiconfigs for the same 3721 :term:`MACHINE`: "qemux86" built 3722 for two distributions such as "poky" and "poky-lsb". In this case, 3723 you might want to use the same :term:`TMPDIR`. 3724 3725 Here is an example showing the minimal statements needed in a 3726 configuration file for a "qemux86" target whose temporary build 3727 directory is ``tmpmultix86``:: 3728 3729 MACHINE = "qemux86" 3730 TMPDIR = "${TOPDIR}/tmpmultix86" 3731 3732 The location for these multiconfig configuration files is specific. 3733 They must reside in the current build directory in a sub-directory of 3734 ``conf`` named ``multiconfig``. Following is an example that defines 3735 two configuration files for the "x86" and "arm" multiconfigs: 3736 3737 .. image:: figures/multiconfig_files.png 3738 :align: center 3739 3740 The reason for this required file hierarchy is because the :term:`BBPATH` 3741 variable is not constructed until the layers are parsed. 3742 Consequently, using the configuration file as a pre-configuration 3743 file is not possible unless it is located in the current working 3744 directory. 3745 3746- *Add the BitBake Multi-configuration Variable to the Local 3747 Configuration File*: Use the 3748 :term:`BBMULTICONFIG` 3749 variable in your ``conf/local.conf`` configuration file to specify 3750 each multiconfig. Continuing with the example from the previous 3751 figure, the :term:`BBMULTICONFIG` variable needs to enable two 3752 multiconfigs: "x86" and "arm" by specifying each configuration file:: 3753 3754 BBMULTICONFIG = "x86 arm" 3755 3756 .. note:: 3757 3758 A "default" configuration already exists by definition. This 3759 configuration is named: "" (i.e. empty string) and is defined by 3760 the variables coming from your ``local.conf`` 3761 file. Consequently, the previous example actually adds two 3762 additional configurations to your build: "arm" and "x86" along 3763 with "". 3764 3765- *Launch BitBake*: Use the following BitBake command form to launch 3766 the multiple configuration build:: 3767 3768 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ] 3769 3770 For the example in this section, the following command applies:: 3771 3772 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base 3773 3774 The previous BitBake command builds a ``core-image-minimal`` image 3775 that is configured through the ``x86.conf`` configuration file, a 3776 ``core-image-sato`` image that is configured through the ``arm.conf`` 3777 configuration file and a ``core-image-base`` that is configured 3778 through your ``local.conf`` configuration file. 3779 3780.. note:: 3781 3782 Support for multiple configuration builds in the Yocto Project &DISTRO; 3783 (&DISTRO_NAME;) Release does not include Shared State (sstate) 3784 optimizations. Consequently, if a build uses the same object twice 3785 in, for example, two different :term:`TMPDIR` 3786 directories, the build either loads from an existing sstate cache for 3787 that build at the start or builds the object fresh. 3788 3789Enabling Multiple Configuration Build Dependencies 3790~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3791 3792Sometimes dependencies can exist between targets (multiconfigs) in a 3793multiple configuration build. For example, suppose that in order to 3794build a ``core-image-sato`` image for an "x86" multiconfig, the root 3795filesystem of an "arm" multiconfig must exist. This dependency is 3796essentially that the 3797:ref:`ref-tasks-image` task in the 3798``core-image-sato`` recipe depends on the completion of the 3799:ref:`ref-tasks-rootfs` task of the 3800``core-image-minimal`` recipe. 3801 3802To enable dependencies in a multiple configuration build, you must 3803declare the dependencies in the recipe using the following statement 3804form:: 3805 3806 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" 3807 3808To better show how to use this statement, consider the example scenario 3809from the first paragraph of this section. The following statement needs 3810to be added to the recipe that builds the ``core-image-sato`` image:: 3811 3812 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs" 3813 3814In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The 3815task on which the ``do_image`` task in the recipe depends is the 3816``do_rootfs`` task from the ``core-image-minimal`` recipe associated 3817with the "arm" multiconfig. 3818 3819Once you set up this dependency, you can build the "x86" multiconfig 3820using a BitBake command as follows:: 3821 3822 $ bitbake mc:x86:core-image-sato 3823 3824This command executes all the tasks needed to create the 3825``core-image-sato`` image for the "x86" multiconfig. Because of the 3826dependency, BitBake also executes through the ``do_rootfs`` task for the 3827"arm" multiconfig build. 3828 3829Having a recipe depend on the root filesystem of another build might not 3830seem that useful. Consider this change to the statement in the 3831``core-image-sato`` recipe:: 3832 3833 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image" 3834 3835In this case, BitBake must 3836create the ``core-image-minimal`` image for the "arm" build since the 3837"x86" build depends on it. 3838 3839Because "x86" and "arm" are enabled for multiple configuration builds 3840and have separate configuration files, BitBake places the artifacts for 3841each build in the respective temporary build directories (i.e. 3842:term:`TMPDIR`). 3843 3844Building an Initial RAM Filesystem (initramfs) Image 3845---------------------------------------------------- 3846 3847An initial RAM filesystem (initramfs) image provides a temporary root 3848filesystem used for early system initialization (e.g. loading of modules 3849needed to locate and mount the "real" root filesystem). 3850 3851.. note:: 3852 3853 The initramfs image is the successor of initial RAM disk (initrd). It 3854 is a "copy in and out" (cpio) archive of the initial filesystem that 3855 gets loaded into memory during the Linux startup process. Because 3856 Linux uses the contents of the archive during initialization, the 3857 initramfs image needs to contain all of the device drivers and tools 3858 needed to mount the final root filesystem. 3859 3860Follow these steps to create an initramfs image: 3861 38621. *Create the initramfs Image Recipe:* You can reference the 3863 ``core-image-minimal-initramfs.bb`` recipe found in the 3864 ``meta/recipes-core`` directory of the :term:`Source Directory` 3865 as an example 3866 from which to work. 3867 38682. *Decide if You Need to Bundle the initramfs Image Into the Kernel 3869 Image:* If you want the initramfs image that is built to be bundled 3870 in with the kernel image, set the 3871 :term:`INITRAMFS_IMAGE_BUNDLE` 3872 variable to "1" in your ``local.conf`` configuration file and set the 3873 :term:`INITRAMFS_IMAGE` 3874 variable in the recipe that builds the kernel image. 3875 3876 .. note:: 3877 3878 It is recommended that you bundle the initramfs image with the 3879 kernel image to avoid circular dependencies between the kernel 3880 recipe and the initramfs recipe should the initramfs image include 3881 kernel modules. 3882 3883 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the initramfs 3884 image to be unpacked into the ``${B}/usr/`` directory. The unpacked 3885 initramfs image is then passed to the kernel's ``Makefile`` using the 3886 :term:`CONFIG_INITRAMFS_SOURCE` 3887 variable, allowing the initramfs image to be built into the kernel 3888 normally. 3889 3890 .. note:: 3891 3892 Bundling the initramfs with the kernel conflates the code in the initramfs 3893 with the GPLv2 licensed Linux kernel binary. Thus only GPLv2 compatible 3894 software may be part of a bundled initramfs. 3895 3896 .. note:: 3897 3898 If you choose to not bundle the initramfs image with the kernel 3899 image, you are essentially using an 3900 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__. 3901 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`, 3902 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more 3903 information, see the :ref:`ref-classes-image-live` file. 3904 39053. *Optionally Add Items to the initramfs Image Through the initramfs 3906 Image Recipe:* If you add items to the initramfs image by way of its 3907 recipe, you should use 3908 :term:`PACKAGE_INSTALL` 3909 rather than 3910 :term:`IMAGE_INSTALL`. 3911 :term:`PACKAGE_INSTALL` gives more direct control of what is added to the 3912 image as compared to the defaults you might not necessarily want that 3913 are set by the :ref:`image <ref-classes-image>` 3914 or :ref:`core-image <ref-classes-core-image>` 3915 classes. 3916 39174. *Build the Kernel Image and the initramfs Image:* Build your kernel 3918 image using BitBake. Because the initramfs image recipe is a 3919 dependency of the kernel image, the initramfs image is built as well 3920 and bundled with the kernel image if you used the 3921 :term:`INITRAMFS_IMAGE_BUNDLE` 3922 variable described earlier. 3923 3924Bundling an Initramfs Image From a Separate Multiconfig 3925~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3926 3927There may be a case where we want to build an initramfs image which does not 3928inherit the same distro policy as our main image, for example, we may want 3929our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our initramfs 3930image to keep a smaller footprint. However, by performing the steps mentioned 3931above the initramfs image will inherit ``TCLIBC="glibc"`` without allowing us 3932to override it. 3933 3934To achieve this, you need to perform some additional steps: 3935 39361. *Create a multiconfig for your initramfs image:* You can perform the steps 3937 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig. 3938 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and 3939 contains the variables:: 3940 3941 TMPDIR="${TOPDIR}/tmp-initramfscfg" 3942 TCLIBC="musl" 3943 39442. *Set additional initramfs variables on your main configuration:* 3945 Additionally, on your main configuration (``local.conf``) you need to set the 3946 variables:: 3947 3948 INITRAMFS_MULTICONFIG = "initramfscfg" 3949 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}" 3950 3951 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE` 3952 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE` 3953 to be built coming from the ``initramfscfg`` multiconfig, and to let the 3954 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located. 3955 3956 Building a system with such configuration will build the kernel using the 3957 main configuration but the ``do_bundle_initramfs`` task will grab the 3958 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE` 3959 instead, resulting in a musl based initramfs image bundled in the kernel 3960 but a glibc based main image. 3961 3962 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE` 3963 or to build a different :term:`DISTRO` for it such as ``poky-tiny``. 3964 3965 3966Building a Tiny System 3967---------------------- 3968 3969Very small distributions have some significant advantages such as 3970requiring less on-die or in-package memory (cheaper), better performance 3971through efficient cache usage, lower power requirements due to less 3972memory, faster boot times, and reduced development overhead. Some 3973real-world examples where a very small distribution gives you distinct 3974advantages are digital cameras, medical devices, and small headless 3975systems. 3976 3977This section presents information that shows you how you can trim your 3978distribution to even smaller sizes than the ``poky-tiny`` distribution, 3979which is around 5 Mbytes, that can be built out-of-the-box using the 3980Yocto Project. 3981 3982Tiny System Overview 3983~~~~~~~~~~~~~~~~~~~~ 3984 3985The following list presents the overall steps you need to consider and 3986perform to create distributions with smaller root filesystems, achieve 3987faster boot times, maintain your critical functionality, and avoid 3988initial RAM disks: 3989 3990- :ref:`Determine your goals and guiding principles 3991 <dev-manual/common-tasks:goals and guiding principles>` 3992 3993- :ref:`dev-manual/common-tasks:understand what contributes to your image size` 3994 3995- :ref:`Reduce the size of the root filesystem 3996 <dev-manual/common-tasks:trim the root filesystem>` 3997 3998- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>` 3999 4000- :ref:`dev-manual/common-tasks:remove package management requirements` 4001 4002- :ref:`dev-manual/common-tasks:look for other ways to minimize size` 4003 4004- :ref:`dev-manual/common-tasks:iterate on the process` 4005 4006Goals and Guiding Principles 4007~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4008 4009Before you can reach your destination, you need to know where you are 4010going. Here is an example list that you can use as a guide when creating 4011very small distributions: 4012 4013- Determine how much space you need (e.g. a kernel that is 1 Mbyte or 4014 less and a root filesystem that is 3 Mbytes or less). 4015 4016- Find the areas that are currently taking 90% of the space and 4017 concentrate on reducing those areas. 4018 4019- Do not create any difficult "hacks" to achieve your goals. 4020 4021- Leverage the device-specific options. 4022 4023- Work in a separate layer so that you keep changes isolated. For 4024 information on how to create layers, see the 4025 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section. 4026 4027Understand What Contributes to Your Image Size 4028~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4029 4030It is easiest to have something to start with when creating your own 4031distribution. You can use the Yocto Project out-of-the-box to create the 4032``poky-tiny`` distribution. Ultimately, you will want to make changes in 4033your own distribution that are likely modeled after ``poky-tiny``. 4034 4035.. note:: 4036 4037 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your 4038 ``local.conf`` file to "poky-tiny" as described in the 4039 ":ref:`dev-manual/common-tasks:creating your own distribution`" 4040 section. 4041 4042Understanding some memory concepts will help you reduce the system size. 4043Memory consists of static, dynamic, and temporary memory. Static memory 4044is the TEXT (code), DATA (initialized data in the code), and BSS 4045(uninitialized data) sections. Dynamic memory represents memory that is 4046allocated at runtime: stacks, hash tables, and so forth. Temporary 4047memory is recovered after the boot process. This memory consists of 4048memory used for decompressing the kernel and for the ``__init__`` 4049functions. 4050 4051To help you see where you currently are with kernel and root filesystem 4052sizes, you can use two tools found in the :term:`Source Directory` 4053in the 4054``scripts/tiny/`` directory: 4055 4056- ``ksize.py``: Reports component sizes for the kernel build objects. 4057 4058- ``dirsize.py``: Reports component sizes for the root filesystem. 4059 4060This next tool and command help you organize configuration fragments and 4061view file dependencies in a human-readable form: 4062 4063- ``merge_config.sh``: Helps you manage configuration files and 4064 fragments within the kernel. With this tool, you can merge individual 4065 configuration fragments together. The tool allows you to make 4066 overrides and warns you of any missing configuration options. The 4067 tool is ideal for allowing you to iterate on configurations, create 4068 minimal configurations, and create configuration files for different 4069 machines without having to duplicate your process. 4070 4071 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git 4072 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``, 4073 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig`` 4074 directory. 4075 4076 For more information on configuration fragments, see the 4077 ":ref:`kernel-dev/common:creating configuration fragments`" 4078 section in the Yocto Project Linux Kernel Development Manual. 4079 4080- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command 4081 with these options brings up a Dependency Explorer from which you can 4082 view file dependencies. Understanding these dependencies allows you 4083 to make informed decisions when cutting out various pieces of the 4084 kernel and root filesystem. 4085 4086Trim the Root Filesystem 4087~~~~~~~~~~~~~~~~~~~~~~~~ 4088 4089The root filesystem is made up of packages for booting, libraries, and 4090applications. To change things, you can configure how the packaging 4091happens, which changes the way you build them. You can also modify the 4092filesystem itself or select a different filesystem. 4093 4094First, find out what is hogging your root filesystem by running the 4095``dirsize.py`` script from your root directory:: 4096 4097 $ cd root-directory-of-image 4098 $ dirsize.py 100000 > dirsize-100k.log 4099 $ cat dirsize-100k.log 4100 4101You can apply a filter to the script to ignore files 4102under a certain size. The previous example filters out any files below 4103100 Kbytes. The sizes reported by the tool are uncompressed, and thus 4104will be smaller by a relatively constant factor in a compressed root 4105filesystem. When you examine your log file, you can focus on areas of 4106the root filesystem that take up large amounts of memory. 4107 4108You need to be sure that what you eliminate does not cripple the 4109functionality you need. One way to see how packages relate to each other 4110is by using the Dependency Explorer UI with the BitBake command:: 4111 4112 $ cd image-directory 4113 $ bitbake -u taskexp -g image 4114 4115Use the interface to 4116select potential packages you wish to eliminate and see their dependency 4117relationships. 4118 4119When deciding how to reduce the size, get rid of packages that result in 4120minimal impact on the feature set. For example, you might not need a VGA 4121display. Or, you might be able to get by with ``devtmpfs`` and ``mdev`` 4122instead of ``udev``. 4123 4124Use your ``local.conf`` file to make changes. For example, to eliminate 4125``udev`` and ``glib``, set the following in the local configuration 4126file:: 4127 4128 VIRTUAL-RUNTIME_dev_manager = "" 4129 4130Finally, you should consider exactly the type of root filesystem you 4131need to meet your needs while also reducing its size. For example, 4132consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an 4133``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1 4134Mbyte journal. If you are okay with running read-only, you do not need 4135this journal. 4136 4137.. note:: 4138 4139 After each round of elimination, you need to rebuild your system and 4140 then use the tools to see the effects of your reductions. 4141 4142Trim the Kernel 4143~~~~~~~~~~~~~~~ 4144 4145The kernel is built by including policies for hardware-independent 4146aspects. What subsystems do you enable? For what architecture are you 4147building? Which drivers do you build by default? 4148 4149.. note:: 4150 4151 You can modify the kernel source if you want to help with boot time. 4152 4153Run the ``ksize.py`` script from the top-level Linux build directory to 4154get an idea of what is making up the kernel:: 4155 4156 $ cd top-level-linux-build-directory 4157 $ ksize.py > ksize.log 4158 $ cat ksize.log 4159 4160When you examine the log, you will see how much space is taken up with 4161the built-in ``.o`` files for drivers, networking, core kernel files, 4162filesystem, sound, and so forth. The sizes reported by the tool are 4163uncompressed, and thus will be smaller by a relatively constant factor 4164in a compressed kernel image. Look to reduce the areas that are large 4165and taking up around the "90% rule." 4166 4167To examine, or drill down, into any particular area, use the ``-d`` 4168option with the script:: 4169 4170 $ ksize.py -d > ksize.log 4171 4172Using this option 4173breaks out the individual file information for each area of the kernel 4174(e.g. drivers, networking, and so forth). 4175 4176Use your log file to see what you can eliminate from the kernel based on 4177features you can let go. For example, if you are not going to need 4178sound, you do not need any drivers that support sound. 4179 4180After figuring out what to eliminate, you need to reconfigure the kernel 4181to reflect those changes during the next build. You could run 4182``menuconfig`` and make all your changes at once. However, that makes it 4183difficult to see the effects of your individual eliminations and also 4184makes it difficult to replicate the changes for perhaps another target 4185device. A better method is to start with no configurations using 4186``allnoconfig``, create configuration fragments for individual changes, 4187and then manage the fragments into a single configuration file using 4188``merge_config.sh``. The tool makes it easy for you to iterate using the 4189configuration change and build cycle. 4190 4191Each time you make configuration changes, you need to rebuild the kernel 4192and check to see what impact your changes had on the overall size. 4193 4194Remove Package Management Requirements 4195~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4196 4197Packaging requirements add size to the image. One way to reduce the size 4198of the image is to remove all the packaging requirements from the image. 4199This reduction includes both removing the package manager and its unique 4200dependencies as well as removing the package management data itself. 4201 4202To eliminate all the packaging requirements for an image, be sure that 4203"package-management" is not part of your 4204:term:`IMAGE_FEATURES` 4205statement for the image. When you remove this feature, you are removing 4206the package manager as well as its dependencies from the root 4207filesystem. 4208 4209Look for Other Ways to Minimize Size 4210~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4211 4212Depending on your particular circumstances, other areas that you can 4213trim likely exist. The key to finding these areas is through tools and 4214methods described here combined with experimentation and iteration. Here 4215are a couple of areas to experiment with: 4216 4217- ``glibc``: In general, follow this process: 4218 4219 1. Remove ``glibc`` features from 4220 :term:`DISTRO_FEATURES` 4221 that you think you do not need. 4222 4223 2. Build your distribution. 4224 4225 3. If the build fails due to missing symbols in a package, determine 4226 if you can reconfigure the package to not need those features. For 4227 example, change the configuration to not support wide character 4228 support as is done for ``ncurses``. Or, if support for those 4229 characters is needed, determine what ``glibc`` features provide 4230 the support and restore the configuration. 4231 4232 4. Rebuild and repeat the process. 4233 4234- ``busybox``: For BusyBox, use a process similar as described for 4235 ``glibc``. A difference is you will need to boot the resulting system 4236 to see if you are able to do everything you expect from the running 4237 system. You need to be sure to integrate configuration fragments into 4238 Busybox because BusyBox handles its own core features and then allows 4239 you to add configuration fragments on top. 4240 4241Iterate on the Process 4242~~~~~~~~~~~~~~~~~~~~~~ 4243 4244If you have not reached your goals on system size, you need to iterate 4245on the process. The process is the same. Use the tools and see just what 4246is taking up 90% of the root filesystem and the kernel. Decide what you 4247can eliminate without limiting your device beyond what you need. 4248 4249Depending on your system, a good place to look might be Busybox, which 4250provides a stripped down version of Unix tools in a single, executable 4251file. You might be able to drop virtual terminal services or perhaps 4252ipv6. 4253 4254Building Images for More than One Machine 4255----------------------------------------- 4256 4257A common scenario developers face is creating images for several 4258different machines that use the same software environment. In this 4259situation, it is tempting to set the tunings and optimization flags for 4260each build specifically for the targeted hardware (i.e. "maxing out" the 4261tunings). Doing so can considerably add to build times and package feed 4262maintenance collectively for the machines. For example, selecting tunes 4263that are extremely specific to a CPU core used in a system might enable 4264some micro optimizations in GCC for that particular system but would 4265otherwise not gain you much of a performance difference across the other 4266systems as compared to using a more general tuning across all the builds 4267(e.g. setting :term:`DEFAULTTUNE` 4268specifically for each machine's build). Rather than "max out" each 4269build's tunings, you can take steps that cause the OpenEmbedded build 4270system to reuse software across the various machines where it makes 4271sense. 4272 4273If build speed and package feed maintenance are considerations, you 4274should consider the points in this section that can help you optimize 4275your tunings to best consider build times and package feed maintenance. 4276 4277- *Share the Build Directory:* If at all possible, share the 4278 :term:`TMPDIR` across builds. The 4279 Yocto Project supports switching between different 4280 :term:`MACHINE` values in the same 4281 :term:`TMPDIR`. This practice is well supported and regularly used by 4282 developers when building for multiple machines. When you use the same 4283 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system 4284 can reuse the existing native and often cross-recipes for multiple 4285 machines. Thus, build time decreases. 4286 4287 .. note:: 4288 4289 If :term:`DISTRO` settings change or fundamental configuration settings 4290 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`. 4291 Sharing :term:`TMPDIR` under these circumstances might work but since it is 4292 not guaranteed, you should use a clean :term:`TMPDIR`. 4293 4294- *Enable the Appropriate Package Architecture:* By default, the 4295 OpenEmbedded build system enables three levels of package 4296 architectures: "all", "tune" or "package", and "machine". Any given 4297 recipe usually selects one of these package architectures (types) for 4298 its output. Depending for what a given recipe creates packages, 4299 making sure you enable the appropriate package architecture can 4300 directly impact the build time. 4301 4302 A recipe that just generates scripts can enable "all" architecture 4303 because there are no binaries to build. To specifically enable "all" 4304 architecture, be sure your recipe inherits the 4305 :ref:`allarch <ref-classes-allarch>` class. 4306 This class is useful for "all" architectures because it configures 4307 many variables so packages can be used across multiple architectures. 4308 4309 If your recipe needs to generate packages that are machine-specific 4310 or when one of the build or runtime dependencies is already 4311 machine-architecture dependent, which makes your recipe also 4312 machine-architecture dependent, make sure your recipe enables the 4313 "machine" package architecture through the 4314 :term:`MACHINE_ARCH` 4315 variable:: 4316 4317 PACKAGE_ARCH = "${MACHINE_ARCH}" 4318 4319 When you do not 4320 specifically enable a package architecture through the 4321 :term:`PACKAGE_ARCH`, The 4322 OpenEmbedded build system defaults to the 4323 :term:`TUNE_PKGARCH` setting:: 4324 4325 PACKAGE_ARCH = "${TUNE_PKGARCH}" 4326 4327- *Choose a Generic Tuning File if Possible:* Some tunes are more 4328 generic and can run on multiple targets (e.g. an ``armv5`` set of 4329 packages could run on ``armv6`` and ``armv7`` processors in most 4330 cases). Similarly, ``i486`` binaries could work on ``i586`` and 4331 higher processors. You should realize, however, that advances on 4332 newer processor versions would not be used. 4333 4334 If you select the same tune for several different machines, the 4335 OpenEmbedded build system reuses software previously built, thus 4336 speeding up the overall build time. Realize that even though a new 4337 sysroot for each machine is generated, the software is not recompiled 4338 and only one package feed exists. 4339 4340- *Manage Granular Level Packaging:* Sometimes there are cases where 4341 injecting another level of package architecture beyond the three 4342 higher levels noted earlier can be useful. For example, consider how 4343 NXP (formerly Freescale) allows for the easy reuse of binary packages 4344 in their layer 4345 :yocto_git:`meta-freescale </meta-freescale/>`. 4346 In this example, the 4347 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>` 4348 class shares GPU packages for i.MX53 boards because all boards share 4349 the AMD GPU. The i.MX6-based boards can do the same because all 4350 boards share the Vivante GPU. This class inspects the BitBake 4351 datastore to identify if the package provides or depends on one of 4352 the sub-architecture values. If so, the class sets the 4353 :term:`PACKAGE_ARCH` value 4354 based on the ``MACHINE_SUBARCH`` value. If the package does not 4355 provide or depend on one of the sub-architecture values but it 4356 matches a value in the machine-specific filter, it sets 4357 :term:`MACHINE_ARCH`. This 4358 behavior reduces the number of packages built and saves build time by 4359 reusing binaries. 4360 4361- *Use Tools to Debug Issues:* Sometimes you can run into situations 4362 where software is being rebuilt when you think it should not be. For 4363 example, the OpenEmbedded build system might not be using shared 4364 state between machines when you think it should be. These types of 4365 situations are usually due to references to machine-specific 4366 variables such as :term:`MACHINE`, 4367 :term:`SERIAL_CONSOLES`, 4368 :term:`XSERVER`, 4369 :term:`MACHINE_FEATURES`, 4370 and so forth in code that is supposed to only be tune-specific or 4371 when the recipe depends 4372 (:term:`DEPENDS`, 4373 :term:`RDEPENDS`, 4374 :term:`RRECOMMENDS`, 4375 :term:`RSUGGESTS`, and so forth) 4376 on some other recipe that already has 4377 :term:`PACKAGE_ARCH` defined 4378 as "${MACHINE_ARCH}". 4379 4380 .. note:: 4381 4382 Patches to fix any issues identified are most welcome as these 4383 issues occasionally do occur. 4384 4385 For such cases, you can use some tools to help you sort out the 4386 situation: 4387 4388 - ``state-diff-machines.sh``*:* You can find this tool in the 4389 ``scripts`` directory of the Source Repositories. See the comments 4390 in the script for information on how to use the tool. 4391 4392 - *BitBake's "-S printdiff" Option:* Using this option causes 4393 BitBake to try to establish the closest signature match it can 4394 (e.g. in the shared state cache) and then run ``bitbake-diffsigs`` 4395 over the matches to determine the stamps and delta where these two 4396 stamp trees diverge. 4397 4398Building Software from an External Source 4399----------------------------------------- 4400 4401By default, the OpenEmbedded build system uses the 4402:term:`Build Directory` when building source 4403code. The build process involves fetching the source files, unpacking 4404them, and then patching them if necessary before the build takes place. 4405 4406There are situations where you might want to build software from source 4407files that are external to and thus outside of the OpenEmbedded build 4408system. For example, suppose you have a project that includes a new BSP 4409with a heavily customized kernel. And, you want to minimize exposing the 4410build system to the development team so that they can focus on their 4411project and maintain everyone's workflow as much as possible. In this 4412case, you want a kernel source directory on the development machine 4413where the development occurs. You want the recipe's 4414:term:`SRC_URI` variable to point to 4415the external directory and use it as is, not copy it. 4416 4417To build from software that comes from an external source, all you need 4418to do is inherit the 4419:ref:`externalsrc <ref-classes-externalsrc>` class 4420and then set the 4421:term:`EXTERNALSRC` variable to 4422point to your external source code. Here are the statements to put in 4423your ``local.conf`` file:: 4424 4425 INHERIT += "externalsrc" 4426 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree" 4427 4428This next example shows how to accomplish the same thing by setting 4429:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file:: 4430 4431 EXTERNALSRC = "path" 4432 EXTERNALSRC_BUILD = "path" 4433 4434.. note:: 4435 4436 In order for these settings to take effect, you must globally or 4437 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>` 4438 class. 4439 4440By default, :ref:`ref-classes-externalsrc` builds the source code in a 4441directory separate from the external source directory as specified by 4442:term:`EXTERNALSRC`. If you need 4443to have the source built in the same directory in which it resides, or 4444some other nominated directory, you can set 4445:term:`EXTERNALSRC_BUILD` 4446to point to that directory:: 4447 4448 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree" 4449 4450Replicating a Build Offline 4451--------------------------- 4452 4453It can be useful to take a "snapshot" of upstream sources used in a 4454build and then use that "snapshot" later to replicate the build offline. 4455To do so, you need to first prepare and populate your downloads 4456directory your "snapshot" of files. Once your downloads directory is 4457ready, you can use it at any time and from any machine to replicate your 4458build. 4459 4460Follow these steps to populate your Downloads directory: 4461 44621. *Create a Clean Downloads Directory:* Start with an empty downloads 4463 directory (:term:`DL_DIR`). You 4464 start with an empty downloads directory by either removing the files 4465 in the existing directory or by setting :term:`DL_DIR` to point to either 4466 an empty location or one that does not yet exist. 4467 44682. *Generate Tarballs of the Source Git Repositories:* Edit your 4469 ``local.conf`` configuration file as follows:: 4470 4471 DL_DIR = "/home/your-download-dir/" 4472 BB_GENERATE_MIRROR_TARBALLS = "1" 4473 4474 During 4475 the fetch process in the next step, BitBake gathers the source files 4476 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See 4477 the 4478 :term:`BB_GENERATE_MIRROR_TARBALLS` 4479 variable for more information. 4480 44813. *Populate Your Downloads Directory Without Building:* Use BitBake to 4482 fetch your sources but inhibit the build:: 4483 4484 $ bitbake target --runonly=fetch 4485 4486 The downloads directory (i.e. ``${DL_DIR}``) now has 4487 a "snapshot" of the source files in the form of tarballs, which can 4488 be used for the build. 4489 44904. *Optionally Remove Any Git or other SCM Subdirectories From the 4491 Downloads Directory:* If you want, you can clean up your downloads 4492 directory by removing any Git or other Source Control Management 4493 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs 4494 already contain these subdirectories. 4495 4496Once your downloads directory has everything it needs regarding source 4497files, you can create your "own-mirror" and build your target. 4498Understand that you can use the files to build the target offline from 4499any machine and at any time. 4500 4501Follow these steps to build your target using the files in the downloads 4502directory: 4503 45041. *Using Local Files Only:* Inside your ``local.conf`` file, add the 4505 :term:`SOURCE_MIRROR_URL` variable, inherit the 4506 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the 4507 :term:`BB_NO_NETWORK` variable to your ``local.conf``. 4508 :: 4509 4510 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/" 4511 INHERIT += "own-mirrors" 4512 BB_NO_NETWORK = "1" 4513 4514 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>` 4515 class set up the system to use the downloads directory as your "own 4516 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that 4517 BitBake's fetching process in step 3 stays local, which means files 4518 from your "own-mirror" are used. 4519 45202. *Start With a Clean Build:* You can start with a clean build by 4521 removing the 4522 ``${``\ :term:`TMPDIR`\ ``}`` 4523 directory or using a new :term:`Build Directory`. 4524 45253. *Build Your Target:* Use BitBake to build your target:: 4526 4527 $ bitbake target 4528 4529 The build completes using the known local "snapshot" of source 4530 files from your mirror. The resulting tarballs for your "snapshot" of 4531 source files are in the downloads directory. 4532 4533 .. note:: 4534 4535 The offline build does not work if recipes attempt to find the 4536 latest version of software by setting 4537 :term:`SRCREV` to 4538 ``${``\ :term:`AUTOREV`\ ``}``:: 4539 4540 SRCREV = "${AUTOREV}" 4541 4542 When a recipe sets :term:`SRCREV` to 4543 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an 4544 attempt to determine the latest version of software from the SCM. 4545 Typically, recipes that use :term:`AUTOREV` are custom or modified 4546 recipes. Recipes that reside in public repositories usually do not 4547 use :term:`AUTOREV`. 4548 4549 If you do have recipes that use :term:`AUTOREV`, you can take steps to 4550 still use the recipes in an offline build. Do the following: 4551 4552 1. Use a configuration generated by enabling :ref:`build 4553 history <dev-manual/common-tasks:maintaining build output quality>`. 4554 4555 2. Use the ``buildhistory-collect-srcrevs`` command to collect the 4556 stored :term:`SRCREV` values from the build's history. For more 4557 information on collecting these values, see the 4558 ":ref:`dev-manual/common-tasks:build history package information`" 4559 section. 4560 4561 3. Once you have the correct source revisions, you can modify 4562 those recipes to set :term:`SRCREV` to specific versions of the 4563 software. 4564 4565Speeding Up a Build 4566=================== 4567 4568Build time can be an issue. By default, the build system uses simple 4569controls to try and maximize build efficiency. In general, the default 4570settings for all the following variables result in the most efficient 4571build times when dealing with single socket systems (i.e. a single CPU). 4572If you have multiple CPUs, you might try increasing the default values 4573to gain more speed. See the descriptions in the glossary for each 4574variable for more information: 4575 4576- :term:`BB_NUMBER_THREADS`: 4577 The maximum number of threads BitBake simultaneously executes. 4578 4579- :term:`BB_NUMBER_PARSE_THREADS`: 4580 The number of threads BitBake uses during parsing. 4581 4582- :term:`PARALLEL_MAKE`: Extra 4583 options passed to the ``make`` command during the 4584 :ref:`ref-tasks-compile` task in 4585 order to specify parallel compilation on the local build host. 4586 4587- :term:`PARALLEL_MAKEINST`: 4588 Extra options passed to the ``make`` command during the 4589 :ref:`ref-tasks-install` task in 4590 order to specify parallel installation on the local build host. 4591 4592As mentioned, these variables all scale to the number of processor cores 4593available on the build system. For single socket systems, this 4594auto-scaling ensures that the build system fundamentally takes advantage 4595of potential parallel operations during the build based on the build 4596machine's capabilities. 4597 4598Following are additional factors that can affect build speed: 4599 4600- File system type: The file system type that the build is being 4601 performed on can also influence performance. Using ``ext4`` is 4602 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4`` 4603 improved features such as extents. 4604 4605- Disabling the updating of access time using ``noatime``: The 4606 ``noatime`` mount option prevents the build system from updating file 4607 and directory access times. 4608 4609- Setting a longer commit: Using the "commit=" mount option increases 4610 the interval in seconds between disk cache writes. Changing this 4611 interval from the five second default to something longer increases 4612 the risk of data loss but decreases the need to write to the disk, 4613 thus increasing the build performance. 4614 4615- Choosing the packaging backend: Of the available packaging backends, 4616 IPK is the fastest. Additionally, selecting a singular packaging 4617 backend also helps. 4618 4619- Using ``tmpfs`` for :term:`TMPDIR` 4620 as a temporary file system: While this can help speed up the build, 4621 the benefits are limited due to the compiler using ``-pipe``. The 4622 build system goes to some lengths to avoid ``sync()`` calls into the 4623 file system on the principle that if there was a significant failure, 4624 the :term:`Build Directory` 4625 contents could easily be rebuilt. 4626 4627- Inheriting the 4628 :ref:`rm_work <ref-classes-rm-work>` class: 4629 Inheriting this class has shown to speed up builds due to 4630 significantly lower amounts of data stored in the data cache as well 4631 as on disk. Inheriting this class also makes cleanup of 4632 :term:`TMPDIR` faster, at the 4633 expense of being easily able to dive into the source code. File 4634 system maintainers have recommended that the fastest way to clean up 4635 large numbers of files is to reformat partitions rather than delete 4636 files due to the linear nature of partitions. This, of course, 4637 assumes you structure the disk partitions and file systems in a way 4638 that this is practical. 4639 4640Aside from the previous list, you should keep some trade offs in mind 4641that can help you speed up the build: 4642 4643- Remove items from 4644 :term:`DISTRO_FEATURES` 4645 that you might not need. 4646 4647- Exclude debug symbols and other debug information: If you do not need 4648 these symbols and other debug information, disabling the ``*-dbg`` 4649 package generation can speed up the build. You can disable this 4650 generation by setting the 4651 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT` 4652 variable to "1". 4653 4654- Disable static library generation for recipes derived from 4655 ``autoconf`` or ``libtool``: Following is an example showing how to 4656 disable static libraries and still provide an override to handle 4657 exceptions:: 4658 4659 STATICLIBCONF = "--disable-static" 4660 STATICLIBCONF:sqlite3-native = "" 4661 EXTRA_OECONF += "${STATICLIBCONF}" 4662 4663 .. note:: 4664 4665 - Some recipes need static libraries in order to work correctly 4666 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides, 4667 as in the previous example, account for these kinds of 4668 exceptions. 4669 4670 - Some packages have packaging code that assumes the presence of 4671 the static libraries. If so, you might need to exclude them as 4672 well. 4673 4674Working With Libraries 4675====================== 4676 4677Libraries are an integral part of your system. This section describes 4678some common practices you might find helpful when working with libraries 4679to build your system: 4680 4681- :ref:`How to include static library files 4682 <dev-manual/common-tasks:including static library files>` 4683 4684- :ref:`How to use the Multilib feature to combine multiple versions of 4685 library files into a single image 4686 <dev-manual/common-tasks:combining multiple versions of library files into one image>` 4687 4688- :ref:`How to install multiple versions of the same library in parallel on 4689 the same system 4690 <dev-manual/common-tasks:installing multiple versions of the same library>` 4691 4692Including Static Library Files 4693------------------------------ 4694 4695If you are building a library and the library offers static linking, you 4696can control which static library files (``*.a`` files) get included in 4697the built library. 4698 4699The :term:`PACKAGES` and 4700:term:`FILES:* <FILES>` variables in the 4701``meta/conf/bitbake.conf`` configuration file define how files installed 4702by the ``do_install`` task are packaged. By default, the :term:`PACKAGES` 4703variable includes ``${PN}-staticdev``, which represents all static 4704library files. 4705 4706.. note:: 4707 4708 Some previously released versions of the Yocto Project defined the 4709 static library files through ``${PN}-dev``. 4710 4711Following is part of the BitBake configuration file, where you can see 4712how the static library files are defined:: 4713 4714 PACKAGE_BEFORE_PN ?= "" 4715 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" 4716 PACKAGES_DYNAMIC = "^${PN}-locale-.*" 4717 FILES = "" 4718 4719 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \ 4720 ${sysconfdir} ${sharedstatedir} ${localstatedir} \ 4721 ${base_bindir}/* ${base_sbindir}/* \ 4722 ${base_libdir}/*${SOLIBS} \ 4723 ${base_prefix}/lib/udev ${prefix}/lib/udev \ 4724 ${base_libdir}/udev ${libdir}/udev \ 4725 ${datadir}/${BPN} ${libdir}/${BPN}/* \ 4726 ${datadir}/pixmaps ${datadir}/applications \ 4727 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \ 4728 ${libdir}/bonobo/servers" 4729 4730 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*" 4731 4732 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \ 4733 ${datadir}/gnome/help" 4734 SECTION:${PN}-doc = "doc" 4735 4736 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}" 4737 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \ 4738 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \ 4739 ${datadir}/aclocal ${base_libdir}/*.o \ 4740 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \ 4741 ${libdir}/cmake ${datadir}/cmake" 4742 SECTION:${PN}-dev = "devel" 4743 ALLOW_EMPTY:${PN}-dev = "1" 4744 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})" 4745 4746 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a" 4747 SECTION:${PN}-staticdev = "devel" 4748 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})" 4749 4750Combining Multiple Versions of Library Files into One Image 4751----------------------------------------------------------- 4752 4753The build system offers the ability to build libraries with different 4754target optimizations or architecture formats and combine these together 4755into one system image. You can link different binaries in the image 4756against the different libraries as needed for specific use cases. This 4757feature is called "Multilib". 4758 4759An example would be where you have most of a system compiled in 32-bit 4760mode using 32-bit libraries, but you have something large, like a 4761database engine, that needs to be a 64-bit application and uses 64-bit 4762libraries. Multilib allows you to get the best of both 32-bit and 64-bit 4763libraries. 4764 4765While the Multilib feature is most commonly used for 32 and 64-bit 4766differences, the approach the build system uses facilitates different 4767target optimizations. You could compile some binaries to use one set of 4768libraries and other binaries to use a different set of libraries. The 4769libraries could differ in architecture, compiler options, or other 4770optimizations. 4771 4772There are several examples in the ``meta-skeleton`` layer found in the 4773:term:`Source Directory`: 4774 4775- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>` 4776 configuration file. 4777 4778- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>` 4779 configuration file. 4780 4781- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>` 4782 recipe 4783 4784Preparing to Use Multilib 4785~~~~~~~~~~~~~~~~~~~~~~~~~ 4786 4787User-specific requirements drive the Multilib feature. Consequently, 4788there is no one "out-of-the-box" configuration that would 4789meet your needs. 4790 4791In order to enable Multilib, you first need to ensure your recipe is 4792extended to support multiple libraries. Many standard recipes are 4793already extended and support multiple libraries. You can check in the 4794``meta/conf/multilib.conf`` configuration file in the 4795:term:`Source Directory` to see how this is 4796done using the 4797:term:`BBCLASSEXTEND` variable. 4798Eventually, all recipes will be covered and this list will not be 4799needed. 4800 4801For the most part, the :ref:`Multilib <ref-classes-multilib*>` 4802class extension works automatically to 4803extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where 4804:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-"). 4805Standard variables such as 4806:term:`DEPENDS`, 4807:term:`RDEPENDS`, 4808:term:`RPROVIDES`, 4809:term:`RRECOMMENDS`, 4810:term:`PACKAGES`, and 4811:term:`PACKAGES_DYNAMIC` are 4812automatically extended by the system. If you are extending any manual 4813code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure 4814those names are extended correctly. 4815 4816Using Multilib 4817~~~~~~~~~~~~~~ 4818 4819After you have set up the recipes, you need to define the actual 4820combination of multiple libraries you want to build. You accomplish this 4821through your ``local.conf`` configuration file in the 4822:term:`Build Directory`. An example 4823configuration would be as follows:: 4824 4825 MACHINE = "qemux86-64" 4826 require conf/multilib.conf 4827 MULTILIBS = "multilib:lib32" 4828 DEFAULTTUNE:virtclass-multilib-lib32 = "x86" 4829 IMAGE_INSTALL:append = "lib32-glib-2.0" 4830 4831This example enables an additional library named 4832``lib32`` alongside the normal target packages. When combining these 4833"lib32" alternatives, the example uses "x86" for tuning. For information 4834on this particular tuning, see 4835``meta/conf/machine/include/ia32/arch-ia32.inc``. 4836 4837The example then includes ``lib32-glib-2.0`` in all the images, which 4838illustrates one method of including a multiple library dependency. You 4839can use a normal image build to include this dependency, for example:: 4840 4841 $ bitbake core-image-sato 4842 4843You can also build Multilib packages 4844specifically with a command like this:: 4845 4846 $ bitbake lib32-glib-2.0 4847 4848Additional Implementation Details 4849~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4850 4851There are generic implementation details as well as details that are specific to 4852package management systems. Following are implementation details 4853that exist regardless of the package management system: 4854 4855- The typical convention used for the class extension code as used by 4856 Multilib assumes that all package names specified in 4857 :term:`PACKAGES` that contain 4858 ``${PN}`` have ``${PN}`` at the start of the name. When that 4859 convention is not followed and ``${PN}`` appears at the middle or the 4860 end of a name, problems occur. 4861 4862- The :term:`TARGET_VENDOR` 4863 value under Multilib will be extended to "-vendormlmultilib" (e.g. 4864 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this 4865 slightly unwieldy contraction is that any "-" characters in the 4866 vendor string presently break Autoconf's ``config.sub``, and other 4867 separators are problematic for different reasons. 4868 4869Here are the implementation details for the RPM Package Management System: 4870 4871- A unique architecture is defined for the Multilib packages, along 4872 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the 4873 :term:`Build Directory`. For 4874 example, consider ``lib32`` in a ``qemux86-64`` image. The possible 4875 architectures in the system are "all", "qemux86_64", 4876 "lib32:qemux86_64", and "lib32:x86". 4877 4878- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM 4879 packaging. The naming for a normal RPM package and a Multilib RPM 4880 package in a ``qemux86-64`` system resolves to something similar to 4881 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``, 4882 respectively. 4883 4884- When installing a Multilib image, the RPM backend first installs the 4885 base image and then installs the Multilib libraries. 4886 4887- The build system relies on RPM to resolve the identical files in the 4888 two (or more) Multilib packages. 4889 4890Here are the implementation details for the IPK Package Management System: 4891 4892- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK 4893 packaging. The naming for a normal RPM package and a Multilib IPK 4894 package in a ``qemux86-64`` system resolves to something like 4895 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``, 4896 respectively. 4897 4898- The IPK deploy folder is not modified with ``${MLPREFIX}`` because 4899 packages with and without the Multilib feature can exist in the same 4900 folder due to the ``${PN}`` differences. 4901 4902- IPK defines a sanity check for Multilib installation using certain 4903 rules for file comparison, overridden, etc. 4904 4905Installing Multiple Versions of the Same Library 4906------------------------------------------------ 4907 4908There are be situations where you need to install and use multiple versions 4909of the same library on the same system at the same time. This 4910almost always happens when a library API changes and you have 4911multiple pieces of software that depend on the separate versions of the 4912library. To accommodate these situations, you can install multiple 4913versions of the same library in parallel on the same system. 4914 4915The process is straightforward as long as the libraries use proper 4916versioning. With properly versioned libraries, all you need to do to 4917individually specify the libraries is create separate, appropriately 4918named recipes where the :term:`PN` part of 4919the name includes a portion that differentiates each library version 4920(e.g. the major part of the version number). Thus, instead of having a 4921single recipe that loads one version of a library (e.g. ``clutter``), 4922you provide multiple recipes that result in different versions of the 4923libraries you want. As an example, the following two recipes would allow 4924the two separate versions of the ``clutter`` library to co-exist on the 4925same system: 4926 4927.. code-block:: none 4928 4929 clutter-1.6_1.6.20.bb 4930 clutter-1.8_1.8.4.bb 4931 4932Additionally, if 4933you have other recipes that depend on a given library, you need to use 4934the :term:`DEPENDS` variable to 4935create the dependency. Continuing with the same example, if you want to 4936have a recipe depend on the 1.8 version of the ``clutter`` library, use 4937the following in your recipe:: 4938 4939 DEPENDS = "clutter-1.8" 4940 4941Working with Pre-Built Libraries 4942================================ 4943 4944Introduction 4945------------- 4946 4947Some library vendors do not release source code for their software but do 4948release pre-built binaries. When shared libraries are built, they should 4949be versioned (see `this article 4950<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__ 4951for some background), but sometimes this is not done. 4952 4953To summarize, a versioned library must meet two conditions: 4954 4955#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``. 4956#. The library must have the ELF tag ``SONAME`` set to the major version 4957 of the library, for example: ``libfoo.so.1``. You can check this by 4958 running ``readelf -d filename | grep SONAME``. 4959 4960This section shows how to deal with both versioned and unversioned 4961pre-built libraries. 4962 4963Versioned Libraries 4964------------------- 4965 4966In this example we work with pre-built libraries for the FT4222H USB I/O chip. 4967Libraries are built for several target architecture variants and packaged in 4968an archive as follows:: 4969 4970 ├── build-arm-hisiv300 4971 │ └── libft4222.so.1.4.4.44 4972 ├── build-arm-v5-sf 4973 │ └── libft4222.so.1.4.4.44 4974 ├── build-arm-v6-hf 4975 │ └── libft4222.so.1.4.4.44 4976 ├── build-arm-v7-hf 4977 │ └── libft4222.so.1.4.4.44 4978 ├── build-arm-v8 4979 │ └── libft4222.so.1.4.4.44 4980 ├── build-i386 4981 │ └── libft4222.so.1.4.4.44 4982 ├── build-i486 4983 │ └── libft4222.so.1.4.4.44 4984 ├── build-mips-eglibc-hf 4985 │ └── libft4222.so.1.4.4.44 4986 ├── build-pentium 4987 │ └── libft4222.so.1.4.4.44 4988 ├── build-x86_64 4989 │ └── libft4222.so.1.4.4.44 4990 ├── examples 4991 │ ├── get-version.c 4992 │ ├── i2cm.c 4993 │ ├── spim.c 4994 │ └── spis.c 4995 ├── ftd2xx.h 4996 ├── install4222.sh 4997 ├── libft4222.h 4998 ├── ReadMe.txt 4999 └── WinTypes.h 5000 5001To write a recipe to use such a library in your system: 5002 5003- The vendor will probably have a proprietary licence, so set 5004 :term:`LICENSE_FLAGS` in your recipe. 5005- The vendor provides a tarball containing libraries so set :term:`SRC_URI` 5006 appropriately. 5007- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an 5008 unsupported architecture. In the following example, we only support the 32 5009 and 64 bit variants of the ``x86`` architecture. 5010- As the vendor provides versioned libraries, we can use ``oe_soinstall`` 5011 from :ref:`ref-classes-utils` to install the shared library and create 5012 symbolic links. If the vendor does not do this, we need to follow the 5013 non-versioned library guidelines in the next section. 5014- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto 5015 Project build, disable the corresponding checks by adding ``ldflags`` 5016 to :term:`INSANE_SKIP`. 5017- The vendor will typically ship release builds without debugging symbols. 5018 Avoid errors by preventing the packaging task from stripping out the symbols 5019 and adding them to a separate debug package. This is done by setting the 5020 ``INHIBIT_`` flags shown below. 5021 5022The complete recipe would look like this:: 5023 5024 SUMMARY = "FTDI FT4222H Library" 5025 SECTION = "libs" 5026 LICENSE_FLAGS = "ftdi" 5027 LICENSE = "CLOSED" 5028 5029 COMPATIBLE_HOST = "(i.86|x86_64).*-linux" 5030 5031 # Sources available in a .tgz file in .zip archive 5032 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip 5033 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/ 5034 # Since dealing with this particular type of archive is out of topic here, 5035 # we use a local link. 5036 SRC_URI = "file://libft4222-linux-${PV}.tgz" 5037 5038 S = "${WORKDIR}" 5039 5040 ARCH_DIR:x86-64 = "build-x86_64" 5041 ARCH_DIR:i586 = "build-i386" 5042 ARCH_DIR:i686 = "build-i386" 5043 5044 INSANE_SKIP:${PN} = "ldflags" 5045 INHIBIT_PACKAGE_STRIP = "1" 5046 INHIBIT_SYSROOT_STRIP = "1" 5047 INHIBIT_PACKAGE_DEBUG_SPLIT = "1" 5048 5049 do_install () { 5050 install -m 0755 -d ${D}${libdir} 5051 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir} 5052 install -d ${D}${includedir} 5053 install -m 0755 ${S}/*.h ${D}${includedir} 5054 } 5055 5056If the precompiled binaries are not statically linked and have dependencies on 5057other libraries, then by adding those libraries to :term:`DEPENDS`, the linking 5058can be examined and the appropriate :term:`RDEPENDS` automatically added. 5059 5060Non-Versioned Libraries 5061----------------------- 5062 5063Some Background 5064~~~~~~~~~~~~~~~ 5065 5066Libraries in Linux systems are generally versioned so that it is possible 5067to have multiple versions of the same library installed, which eases upgrades 5068and support for older software. For example, suppose that in a versioned 5069library, an actual library is called ``libfoo.so.1.2``, a symbolic link named 5070``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named 5071``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you 5072link a binary against a library, you typically provide the unversioned file 5073name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic 5074link and actually links against the versioned filename. The unversioned symbolic 5075link is only used at development time. Consequently, the library is packaged 5076along with the headers in the development package ``${PN}-dev`` along with the 5077actual library and versioned symbolic links in ``${PN}``. Because versioned 5078libraries are far more common than unversioned libraries, the default packaging 5079rules assume versioned libraries. 5080 5081Yocto Library Packaging Overview 5082~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5083 5084It follows that packaging an unversioned library requires a bit of work in the 5085recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which 5086triggers a QA warning that a non-symlink library is in a ``-dev`` package, 5087and binaries in the same recipe link to the library in ``${PN}-dev``, 5088which triggers more QA warnings. To solve this problem, you need to package the 5089unversioned library into ``${PN}`` where it belongs. The following are the abridged 5090default :term:`FILES` variables in ``bitbake.conf``:: 5091 5092 SOLIBS = ".so.*" 5093 SOLIBSDEV = ".so" 5094 FILES:${PN} = "... ${libdir}/lib*${SOLIBS} ..." 5095 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..." 5096 FILES:${PN}-dev = "... ${FILES_SOLIBSDEV} ..." 5097 5098:term:`SOLIBS` defines a pattern that matches real shared object libraries. 5099:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two 5100variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts 5101the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``. 5102To package unversioned libraries, you need to modify the variables in the recipe 5103as follows:: 5104 5105 SOLIBS = ".so" 5106 FILES_SOLIBSDEV = "" 5107 5108The modifications cause the ``.so`` file to be the real library 5109and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into 5110``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed, 5111``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of 5112the files you want in ``${PN}``. 5113 5114Finally, loadable modules, essentially unversioned libraries that are linked 5115at runtime using ``dlopen()`` instead of at build time, should generally be 5116installed in a private directory. However, if they are installed in ``${libdir}``, 5117then the modules can be treated as unversioned libraries. 5118 5119Example 5120~~~~~~~ 5121 5122The example below installs an unversioned x86-64 pre-built library named 5123``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the 5124x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP` 5125and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above 5126versioned library example. The "magic" is setting the :term:`SOLIBS` and 5127:term:`FILES_SOLIBSDEV` variables as explained above:: 5128 5129 SUMMARY = "libfoo sample recipe" 5130 SECTION = "libs" 5131 LICENSE = "CLOSED" 5132 5133 SRC_URI = "file://libfoo.so" 5134 5135 COMPATIBLE_HOST = "x86_64.*-linux" 5136 5137 INSANE_SKIP:${PN} = "ldflags" 5138 INHIBIT_PACKAGE_STRIP = "1" 5139 INHIBIT_SYSROOT_STRIP = "1" 5140 SOLIBS = ".so" 5141 FILES_SOLIBSDEV = "" 5142 5143 do_install () { 5144 install -d ${D}${libdir} 5145 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir} 5146 } 5147 5148Using x32 psABI 5149=============== 5150 5151x32 processor-specific Application Binary Interface (`x32 5152psABI <https://software.intel.com/en-us/node/628948>`__) is a native 515332-bit processor-specific ABI for Intel 64 (x86-64) architectures. An 5154ABI defines the calling conventions between functions in a processing 5155environment. The interface determines what registers are used and what 5156the sizes are for various C data types. 5157 5158Some processing environments prefer using 32-bit applications even when 5159running on Intel 64-bit platforms. Consider the i386 psABI, which is a 5160very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not 5161provide efficient use and access of the Intel 64-bit processor 5162resources, leaving the system underutilized. Now consider the x86_64 5163psABI. This ABI is newer and uses 64-bits for data sizes and program 5164pointers. The extra bits increase the footprint size of the programs, 5165libraries, and also increases the memory and file system size 5166requirements. Executing under the x32 psABI enables user programs to 5167utilize CPU and system resources more efficiently while keeping the 5168memory footprint of the applications low. Extra bits are used for 5169registers but not for addressing mechanisms. 5170 5171The Yocto Project supports the final specifications of x32 psABI as 5172follows: 5173 5174- You can create packages and images in x32 psABI format on x86_64 5175 architecture targets. 5176 5177- You can successfully build recipes with the x32 toolchain. 5178 5179- You can create and boot ``core-image-minimal`` and 5180 ``core-image-sato`` images. 5181 5182- There is RPM Package Manager (RPM) support for x32 binaries. 5183 5184- There is support for large images. 5185 5186To use the x32 psABI, you need to edit your ``conf/local.conf`` 5187configuration file as follows:: 5188 5189 MACHINE = "qemux86-64" 5190 DEFAULTTUNE = "x86-64-x32" 5191 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \ 5192 or 'INVALID')) or 'lib'}" 5193 5194Once you have set 5195up your configuration file, use BitBake to build an image that supports 5196the x32 psABI. Here is an example:: 5197 5198 $ bitbake core-image-sato 5199 5200Enabling GObject Introspection Support 5201====================================== 5202 5203`GObject introspection <https://gi.readthedocs.io/en/latest/>`__ 5204is the standard mechanism for accessing GObject-based software from 5205runtime environments. GObject is a feature of the GLib library that 5206provides an object framework for the GNOME desktop and related software. 5207GObject Introspection adds information to GObject that allows objects 5208created within it to be represented across different programming 5209languages. If you want to construct GStreamer pipelines using Python, or 5210control UPnP infrastructure using Javascript and GUPnP, GObject 5211introspection is the only way to do it. 5212 5213This section describes the Yocto Project support for generating and 5214packaging GObject introspection data. GObject introspection data is a 5215description of the API provided by libraries built on top of the GLib 5216framework, and, in particular, that framework's GObject mechanism. 5217GObject Introspection Repository (GIR) files go to ``-dev`` packages, 5218``typelib`` files go to main packages as they are packaged together with 5219libraries that are introspected. 5220 5221The data is generated when building such a library, by linking the 5222library with a small executable binary that asks the library to describe 5223itself, and then executing the binary and processing its output. 5224 5225Generating this data in a cross-compilation environment is difficult 5226because the library is produced for the target architecture, but its 5227code needs to be executed on the build host. This problem is solved with 5228the OpenEmbedded build system by running the code through QEMU, which 5229allows precisely that. Unfortunately, QEMU does not always work 5230perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`" 5231section. 5232 5233Enabling the Generation of Introspection Data 5234--------------------------------------------- 5235 5236Enabling the generation of introspection data (GIR files) in your 5237library package involves the following: 5238 52391. Inherit the 5240 :ref:`gobject-introspection <ref-classes-gobject-introspection>` 5241 class. 5242 52432. Make sure introspection is not disabled anywhere in the recipe or 5244 from anything the recipe includes. Also, make sure that 5245 "gobject-introspection-data" is not in 5246 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED` 5247 and that "qemu-usermode" is not in 5248 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`. 5249 In either of these conditions, nothing will happen. 5250 52513. Try to build the recipe. If you encounter build errors that look like 5252 something is unable to find ``.so`` libraries, check where these 5253 libraries are located in the source tree and add the following to the 5254 recipe:: 5255 5256 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs" 5257 5258 .. note:: 5259 5260 See recipes in the ``oe-core`` repository that use that 5261 :term:`GIR_EXTRA_LIBS_PATH` variable as an example. 5262 52634. Look for any other errors, which probably mean that introspection 5264 support in a package is not entirely standard, and thus breaks down 5265 in a cross-compilation environment. For such cases, custom-made fixes 5266 are needed. A good place to ask and receive help in these cases is 5267 the :ref:`Yocto Project mailing 5268 lists <resources-mailinglist>`. 5269 5270.. note:: 5271 5272 Using a library that no longer builds against the latest Yocto 5273 Project release and prints introspection related errors is a good 5274 candidate for the previous procedure. 5275 5276Disabling the Generation of Introspection Data 5277---------------------------------------------- 5278 5279You might find that you do not want to generate introspection data. Or, 5280perhaps QEMU does not work on your build host and target architecture 5281combination. If so, you can use either of the following methods to 5282disable GIR file generations: 5283 5284- Add the following to your distro configuration:: 5285 5286 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data" 5287 5288 Adding this statement disables generating introspection data using 5289 QEMU but will still enable building introspection tools and libraries 5290 (i.e. building them does not require the use of QEMU). 5291 5292- Add the following to your machine configuration:: 5293 5294 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode" 5295 5296 Adding this statement disables the use of QEMU when building packages for your 5297 machine. Currently, this feature is used only by introspection 5298 recipes and has the same effect as the previously described option. 5299 5300 .. note:: 5301 5302 Future releases of the Yocto Project might have other features 5303 affected by this option. 5304 5305If you disable introspection data, you can still obtain it through other 5306means such as copying the data from a suitable sysroot, or by generating 5307it on the target hardware. The OpenEmbedded build system does not 5308currently provide specific support for these techniques. 5309 5310Testing that Introspection Works in an Image 5311-------------------------------------------- 5312 5313Use the following procedure to test if generating introspection data is 5314working in an image: 5315 53161. Make sure that "gobject-introspection-data" is not in 5317 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED` 5318 and that "qemu-usermode" is not in 5319 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`. 5320 53212. Build ``core-image-sato``. 5322 53233. Launch a Terminal and then start Python in the terminal. 5324 53254. Enter the following in the terminal:: 5326 5327 >>> from gi.repository import GLib 5328 >>> GLib.get_host_name() 5329 53305. For something a little more advanced, enter the following see: 5331 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html 5332 5333Known Issues 5334------------ 5335 5336Here are know issues in GObject Introspection Support: 5337 5338- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build 5339 introspection data on that architecture. 5340 5341- x32 is not supported by QEMU. Consequently, introspection data is 5342 disabled. 5343 5344- musl causes transient GLib binaries to crash on assertion failures. 5345 Consequently, generating introspection data is disabled. 5346 5347- Because QEMU is not able to run the binaries correctly, introspection 5348 is disabled for some specific packages under specific architectures 5349 (e.g. ``gcr``, ``libsecret``, and ``webkit``). 5350 5351- QEMU usermode might not work properly when running 64-bit binaries 5352 under 32-bit host machines. In particular, "qemumips64" is known to 5353 not work under i686. 5354 5355Optionally Using an External Toolchain 5356====================================== 5357 5358You might want to use an external toolchain as part of your development. 5359If this is the case, the fundamental steps you need to accomplish are as 5360follows: 5361 5362- Understand where the installed toolchain resides. For cases where you 5363 need to build the external toolchain, you would need to take separate 5364 steps to build and install the toolchain. 5365 5366- Make sure you add the layer that contains the toolchain to your 5367 ``bblayers.conf`` file through the 5368 :term:`BBLAYERS` variable. 5369 5370- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file 5371 to the location in which you installed the toolchain. 5372 5373A good example of an external toolchain used with the Yocto Project is 5374Mentor Graphics Sourcery G++ Toolchain. You can see information on how 5375to use that particular layer in the ``README`` file at 5376https://github.com/MentorEmbedded/meta-sourcery/. You can find 5377further information by reading about the 5378:term:`TCMODE` variable in the Yocto 5379Project Reference Manual's variable glossary. 5380 5381Creating Partitioned Images Using Wic 5382===================================== 5383 5384Creating an image for a particular hardware target using the 5385OpenEmbedded build system does not necessarily mean you can boot that 5386image as is on your device. Physical devices accept and boot images in 5387various ways depending on the specifics of the device. Usually, 5388information about the hardware can tell you what image format the device 5389requires. Should your device require multiple partitions on an SD card, 5390flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to 5391create the properly partitioned image. 5392 5393The ``wic`` command generates partitioned images from existing 5394OpenEmbedded build artifacts. Image generation is driven by partitioning 5395commands contained in an Openembedded kickstart file (``.wks``) 5396specified either directly on the command line or as one of a selection 5397of canned kickstart files as shown with the ``wic list images`` command 5398in the 5399":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`" 5400section. When you apply the command to a given set of build artifacts, the 5401result is an image or set of images that can be directly written onto media and 5402used on a particular system. 5403 5404.. note:: 5405 5406 For a kickstart file reference, see the 5407 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`" 5408 Chapter in the Yocto Project Reference Manual. 5409 5410The ``wic`` command and the infrastructure it is based on is by 5411definition incomplete. The purpose of the command is to allow the 5412generation of customized images, and as such, was designed to be 5413completely extensible through a plugin interface. See the 5414":ref:`dev-manual/common-tasks:using the wic plugin interface`" section 5415for information on these plugins. 5416 5417This section provides some background information on Wic, describes what 5418you need to have in place to run the tool, provides instruction on how 5419to use the Wic utility, provides information on using the Wic plugins 5420interface, and provides several examples that show how to use Wic. 5421 5422Background 5423---------- 5424 5425This section provides some background on the Wic utility. While none of 5426this information is required to use Wic, you might find it interesting. 5427 5428- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The 5429 "oe" diphthong in "oeic" was promoted to the letter "w", because 5430 "oeic" is both difficult to remember and to pronounce. 5431 5432- Wic is loosely based on the Meego Image Creator (``mic``) framework. 5433 The Wic implementation has been heavily modified to make direct use 5434 of OpenEmbedded build artifacts instead of package installation and 5435 configuration, which are already incorporated within the OpenEmbedded 5436 artifacts. 5437 5438- Wic is a completely independent standalone utility that initially 5439 provides easier-to-use and more flexible replacements for an existing 5440 functionality in OE-Core's 5441 :ref:`image-live <ref-classes-image-live>` 5442 class. The difference between Wic and those examples is that with Wic 5443 the functionality of those scripts is implemented by a 5444 general-purpose partitioning language, which is based on Redhat 5445 kickstart syntax. 5446 5447Requirements 5448------------ 5449 5450In order to use the Wic utility with the OpenEmbedded Build system, your 5451system needs to meet the following requirements: 5452 5453- The Linux distribution on your development host must support the 5454 Yocto Project. See the ":ref:`detailed-supported-distros`" 5455 section in the Yocto Project Reference Manual for the list of 5456 distributions that support the Yocto Project. 5457 5458- The standard system utilities, such as ``cp``, must be installed on 5459 your development host system. 5460 5461- You must have sourced the build environment setup script (i.e. 5462 :ref:`structure-core-script`) found in the 5463 :term:`Build Directory`. 5464 5465- You need to have the build artifacts already available, which 5466 typically means that you must have already created an image using the 5467 Openembedded build system (e.g. ``core-image-minimal``). While it 5468 might seem redundant to generate an image in order to create an image 5469 using Wic, the current version of Wic requires the artifacts in the 5470 form generated by the OpenEmbedded build system. 5471 5472- You must build several native tools, which are built to run on the 5473 build system:: 5474 5475 $ bitbake parted-native dosfstools-native mtools-native 5476 5477- Include "wic" as part of the 5478 :term:`IMAGE_FSTYPES` 5479 variable. 5480 5481- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>` 5482 as part of the :term:`WKS_FILE` variable 5483 5484Getting Help 5485------------ 5486 5487You can get general help for the ``wic`` command by entering the ``wic`` 5488command by itself or by entering the command with a help argument as 5489follows:: 5490 5491 $ wic -h 5492 $ wic --help 5493 $ wic help 5494 5495Currently, Wic supports seven commands: ``cp``, ``create``, ``help``, 5496``list``, ``ls``, ``rm``, and ``write``. You can get help for all these 5497commands except "help" by using the following form:: 5498 5499 $ wic help command 5500 5501For example, the following command returns help for the ``write`` 5502command:: 5503 5504 $ wic help write 5505 5506Wic supports help for three topics: ``overview``, ``plugins``, and 5507``kickstart``. You can get help for any topic using the following form:: 5508 5509 $ wic help topic 5510 5511For example, the following returns overview help for Wic:: 5512 5513 $ wic help overview 5514 5515There is one additional level of help for Wic. You can get help on 5516individual images through the ``list`` command. You can use the ``list`` 5517command to return the available Wic images as follows:: 5518 5519 $ wic list images 5520 genericx86 Create an EFI disk image for genericx86* 5521 edgerouter Create SD card image for Edgerouter 5522 beaglebone-yocto Create SD card image for Beaglebone 5523 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image 5524 systemd-bootdisk Create an EFI disk image with systemd-boot 5525 mkhybridiso Create a hybrid ISO image 5526 mkefidisk Create an EFI disk image 5527 sdimage-bootpart Create SD card image with a boot partition 5528 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin 5529 directdisk Create a 'pcbios' direct disk image 5530 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config 5531 qemuriscv Create qcow2 image for RISC-V QEMU machines 5532 directdisk-gpt Create a 'pcbios' direct disk image 5533 efi-bootdisk 5534 5535Once you know the list of available 5536Wic images, you can use ``help`` with the command to get help on a 5537particular image. For example, the following command returns help on the 5538"beaglebone-yocto" image:: 5539 5540 $ wic list beaglebone-yocto help 5541 5542 Creates a partitioned SD card image for Beaglebone. 5543 Boot files are located in the first vfat partition. 5544 5545Operational Modes 5546----------------- 5547 5548You can use Wic in two different modes, depending on how much control 5549you need for specifying the Openembedded build artifacts that are used 5550for creating the image: Raw and Cooked: 5551 5552- *Raw Mode:* You explicitly specify build artifacts through Wic 5553 command-line arguments. 5554 5555- *Cooked Mode:* The current 5556 :term:`MACHINE` setting and image 5557 name are used to automatically locate and provide the build 5558 artifacts. You just supply a kickstart file and the name of the image 5559 from which to use artifacts. 5560 5561Regardless of the mode you use, you need to have the build artifacts 5562ready and available. 5563 5564Raw Mode 5565~~~~~~~~ 5566 5567Running Wic in raw mode allows you to specify all the partitions through 5568the ``wic`` command line. The primary use for raw mode is if you have 5569built your kernel outside of the Yocto Project 5570:term:`Build Directory`. In other words, you 5571can point to arbitrary kernel, root filesystem locations, and so forth. 5572Contrast this behavior with cooked mode where Wic looks in the Build 5573Directory (e.g. ``tmp/deploy/images/``\ machine). 5574 5575The general form of the ``wic`` command in raw mode is:: 5576 5577 $ wic create wks_file options ... 5578 5579 Where: 5580 5581 wks_file: 5582 An OpenEmbedded kickstart file. You can provide 5583 your own custom file or use a file from a set of 5584 existing files as described by further options. 5585 5586 optional arguments: 5587 -h, --help show this help message and exit 5588 -o OUTDIR, --outdir OUTDIR 5589 name of directory to create image in 5590 -e IMAGE_NAME, --image-name IMAGE_NAME 5591 name of the image to use the artifacts from e.g. core- 5592 image-sato 5593 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR 5594 path to the /rootfs dir to use as the .wks rootfs 5595 source 5596 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR 5597 path to the dir containing the boot artifacts (e.g. 5598 /EFI or /syslinux dirs) to use as the .wks bootimg 5599 source 5600 -k KERNEL_DIR, --kernel-dir KERNEL_DIR 5601 path to the dir containing the kernel to use in the 5602 .wks bootimg 5603 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT 5604 path to the native sysroot containing the tools to use 5605 to build the image 5606 -s, --skip-build-check 5607 skip the build check 5608 -f, --build-rootfs build rootfs 5609 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz} 5610 compress image with specified compressor 5611 -m, --bmap generate .bmap 5612 --no-fstab-update Do not change fstab file. 5613 -v VARS_DIR, --vars VARS_DIR 5614 directory with <image>.env files that store bitbake 5615 variables 5616 -D, --debug output debug information 5617 5618.. note:: 5619 5620 You do not need root privileges to run Wic. In fact, you should not 5621 run as root when using the utility. 5622 5623Cooked Mode 5624~~~~~~~~~~~ 5625 5626Running Wic in cooked mode leverages off artifacts in the Build 5627Directory. In other words, you do not have to specify kernel or root 5628filesystem locations as part of the command. All you need to provide is 5629a kickstart file and the name of the image from which to use artifacts 5630by using the "-e" option. Wic looks in the Build Directory (e.g. 5631``tmp/deploy/images/``\ machine) for artifacts. 5632 5633The general form of the ``wic`` command using Cooked Mode is as follows:: 5634 5635 $ wic create wks_file -e IMAGE_NAME 5636 5637 Where: 5638 5639 wks_file: 5640 An OpenEmbedded kickstart file. You can provide 5641 your own custom file or use a file from a set of 5642 existing files provided with the Yocto Project 5643 release. 5644 5645 required argument: 5646 -e IMAGE_NAME, --image-name IMAGE_NAME 5647 name of the image to use the artifacts from e.g. core- 5648 image-sato 5649 5650Using an Existing Kickstart File 5651-------------------------------- 5652 5653If you do not want to create your own kickstart file, you can use an 5654existing file provided by the Wic installation. As shipped, kickstart 5655files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the 5656following two locations:: 5657 5658 poky/meta-yocto-bsp/wic 5659 poky/scripts/lib/wic/canned-wks 5660 5661Use the following command to list the available kickstart files:: 5662 5663 $ wic list images 5664 genericx86 Create an EFI disk image for genericx86* 5665 beaglebone-yocto Create SD card image for Beaglebone 5666 edgerouter Create SD card image for Edgerouter 5667 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image 5668 directdisk-gpt Create a 'pcbios' direct disk image 5669 mkefidisk Create an EFI disk image 5670 directdisk Create a 'pcbios' direct disk image 5671 systemd-bootdisk Create an EFI disk image with systemd-boot 5672 mkhybridiso Create a hybrid ISO image 5673 sdimage-bootpart Create SD card image with a boot partition 5674 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin 5675 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config 5676 5677When you use an existing file, you 5678do not have to use the ``.wks`` extension. Here is an example in Raw 5679Mode that uses the ``directdisk`` file:: 5680 5681 $ wic create directdisk -r rootfs_dir -b bootimg_dir \ 5682 -k kernel_dir -n native_sysroot 5683 5684Here are the actual partition language commands used in the 5685``genericx86.wks`` file to generate an image:: 5686 5687 # short-description: Create an EFI disk image for genericx86* 5688 # long-description: Creates a partitioned EFI disk image for genericx86* machines 5689 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024 5690 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid 5691 part swap --ondisk sda --size 44 --label swap1 --fstype=swap 5692 5693 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0" 5694 5695Using the Wic Plugin Interface 5696------------------------------ 5697 5698You can extend and specialize Wic functionality by using Wic plugins. 5699This section explains the Wic plugin interface. 5700 5701.. note:: 5702 5703 Wic plugins consist of "source" and "imager" plugins. Imager plugins 5704 are beyond the scope of this section. 5705 5706Source plugins provide a mechanism to customize partition content during 5707the Wic image generation process. You can use source plugins to map 5708values that you specify using ``--source`` commands in kickstart files 5709(i.e. ``*.wks``) to a plugin implementation used to populate a given 5710partition. 5711 5712.. note:: 5713 5714 If you use plugins that have build-time dependencies (e.g. native 5715 tools, bootloaders, and so forth) when building a Wic image, you need 5716 to specify those dependencies using the :term:`WKS_FILE_DEPENDS` 5717 variable. 5718 5719Source plugins are subclasses defined in plugin files. As shipped, the 5720Yocto Project provides several plugin files. You can see the source 5721plugin files that ship with the Yocto Project 5722:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`. 5723Each of these plugin files contains source plugins that are designed to 5724populate a specific Wic image partition. 5725 5726Source plugins are subclasses of the ``SourcePlugin`` class, which is 5727defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example, 5728the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py`` 5729file is a subclass of the ``SourcePlugin`` class, which is found in the 5730``pluginbase.py`` file. 5731 5732You can also implement source plugins in a layer outside of the Source 5733Repositories (external layer). To do so, be sure that your plugin files 5734are located in a directory whose path is 5735``scripts/lib/wic/plugins/source/`` within your external layer. When the 5736plugin files are located there, the source plugins they contain are made 5737available to Wic. 5738 5739When the Wic implementation needs to invoke a partition-specific 5740implementation, it looks for the plugin with the same name as the 5741``--source`` parameter used in the kickstart file given to that 5742partition. For example, if the partition is set up using the following 5743command in a kickstart file:: 5744 5745 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 5746 5747The methods defined as class 5748members of the matching source plugin (i.e. ``bootimg-pcbios``) in the 5749``bootimg-pcbios.py`` plugin file are used. 5750 5751To be more concrete, here is the corresponding plugin definition from 5752the ``bootimg-pcbios.py`` file for the previous command along with an 5753example method called by the Wic implementation when it needs to prepare 5754a partition using an implementation-specific function:: 5755 5756 . 5757 . 5758 . 5759 class BootimgPcbiosPlugin(SourcePlugin): 5760 """ 5761 Create MBR boot partition and install syslinux on it. 5762 """ 5763 5764 name = 'bootimg-pcbios' 5765 . 5766 . 5767 . 5768 @classmethod 5769 def do_prepare_partition(cls, part, source_params, creator, cr_workdir, 5770 oe_builddir, bootimg_dir, kernel_dir, 5771 rootfs_dir, native_sysroot): 5772 """ 5773 Called to do the actual content population for a partition i.e. it 5774 'prepares' the partition to be incorporated into the image. 5775 In this case, prepare content for legacy bios boot partition. 5776 """ 5777 . 5778 . 5779 . 5780 5781If a 5782subclass (plugin) itself does not implement a particular function, Wic 5783locates and uses the default version in the superclass. It is for this 5784reason that all source plugins are derived from the ``SourcePlugin`` 5785class. 5786 5787The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines 5788a set of methods that source plugins can implement or override. Any 5789plugins (subclass of ``SourcePlugin``) that do not implement a 5790particular method inherit the implementation of the method from the 5791``SourcePlugin`` class. For more information, see the ``SourcePlugin`` 5792class in the ``pluginbase.py`` file for details: 5793 5794The following list describes the methods implemented in the 5795``SourcePlugin`` class: 5796 5797- ``do_prepare_partition()``: Called to populate a partition with 5798 actual content. In other words, the method prepares the final 5799 partition image that is incorporated into the disk image. 5800 5801- ``do_configure_partition()``: Called before 5802 ``do_prepare_partition()`` to create custom configuration files for a 5803 partition (e.g. syslinux or grub configuration files). 5804 5805- ``do_install_disk()``: Called after all partitions have been 5806 prepared and assembled into a disk image. This method provides a hook 5807 to allow finalization of a disk image (e.g. writing an MBR). 5808 5809- ``do_stage_partition()``: Special content-staging hook called 5810 before ``do_prepare_partition()``. This method is normally empty. 5811 5812 Typically, a partition just uses the passed-in parameters (e.g. the 5813 unmodified value of ``bootimg_dir``). However, in some cases, things 5814 might need to be more tailored. As an example, certain files might 5815 additionally need to be taken from ``bootimg_dir + /boot``. This hook 5816 allows those files to be staged in a customized fashion. 5817 5818 .. note:: 5819 5820 ``get_bitbake_var()`` allows you to access non-standard variables that 5821 you might want to use for this behavior. 5822 5823You can extend the source plugin mechanism. To add more hooks, create 5824more source plugin methods within ``SourcePlugin`` and the corresponding 5825derived subclasses. The code that calls the plugin methods uses the 5826``plugin.get_source_plugin_methods()`` function to find the method or 5827methods needed by the call. Retrieval of those methods is accomplished 5828by filling up a dict with keys that contain the method names of 5829interest. On success, these will be filled in with the actual methods. 5830See the Wic implementation for examples and details. 5831 5832Wic Examples 5833------------ 5834 5835This section provides several examples that show how to use the Wic 5836utility. All the examples assume the list of requirements in the 5837":ref:`dev-manual/common-tasks:requirements`" section have been met. The 5838examples assume the previously generated image is 5839``core-image-minimal``. 5840 5841Generate an Image using an Existing Kickstart File 5842~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5843 5844This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart 5845file:: 5846 5847 $ wic create mkefidisk -e core-image-minimal 5848 INFO: Building wic-tools... 5849 . 5850 . 5851 . 5852 INFO: The new image(s) can be found here: 5853 ./mkefidisk-201804191017-sda.direct 5854 5855 The following build artifacts were used to create the image(s): 5856 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs 5857 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share 5858 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86 5859 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native 5860 5861 INFO: The image(s) were created using OE kickstart file: 5862 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks 5863 5864The previous example shows the easiest way to create an image by running 5865in cooked mode and supplying a kickstart file and the "-e" option to 5866point to the existing build artifacts. Your ``local.conf`` file needs to 5867have the :term:`MACHINE` variable set 5868to the machine you are using, which is "qemux86" in this example. 5869 5870Once the image builds, the output provides image location, artifact use, 5871and kickstart file information. 5872 5873.. note:: 5874 5875 You should always verify the details provided in the output to make 5876 sure that the image was indeed created exactly as expected. 5877 5878Continuing with the example, you can now write the image from the Build 5879Directory onto a USB stick, or whatever media for which you built your 5880image, and boot from the media. You can write the image by using 5881``bmaptool`` or ``dd``:: 5882 5883 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX 5884 5885or :: 5886 5887 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX 5888 5889.. note:: 5890 5891 For more information on how to use the ``bmaptool`` 5892 to flash a device with an image, see the 5893 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``" 5894 section. 5895 5896Using a Modified Kickstart File 5897~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5898 5899Because partitioned image creation is driven by the kickstart file, it 5900is easy to affect image creation by changing the parameters in the file. 5901This next example demonstrates that through modification of the 5902``directdisk-gpt`` kickstart file. 5903 5904As mentioned earlier, you can use the command ``wic list images`` to 5905show the list of existing kickstart files. The directory in which the 5906``directdisk-gpt.wks`` file resides is 5907``scripts/lib/image/canned-wks/``, which is located in the 5908:term:`Source Directory` (e.g. ``poky``). 5909Because available files reside in this directory, you can create and add 5910your own custom files to the directory. Subsequent use of the 5911``wic list images`` command would then include your kickstart files. 5912 5913In this example, the existing ``directdisk-gpt`` file already does most 5914of what is needed. However, for the hardware in this example, the image 5915will need to boot from ``sdb`` instead of ``sda``, which is what the 5916``directdisk-gpt`` kickstart file uses. 5917 5918The example begins by making a copy of the ``directdisk-gpt.wks`` file 5919in the ``scripts/lib/image/canned-wks`` directory and then by changing 5920the lines that specify the target disk from which to boot. 5921:: 5922 5923 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \ 5924 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks 5925 5926Next, the example modifies the ``directdisksdb-gpt.wks`` file and 5927changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The 5928example changes the following two lines and leaves the remaining lines 5929untouched:: 5930 5931 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024 5932 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid 5933 5934Once the lines are changed, the 5935example generates the ``directdisksdb-gpt`` image. The command points 5936the process at the ``core-image-minimal`` artifacts for the Next Unit of 5937Computing (nuc) :term:`MACHINE` the 5938``local.conf``. 5939:: 5940 5941 $ wic create directdisksdb-gpt -e core-image-minimal 5942 INFO: Building wic-tools... 5943 . 5944 . 5945 . 5946 Initialising tasks: 100% |#######################################| Time: 0:00:01 5947 NOTE: Executing SetScene Tasks 5948 NOTE: Executing RunQueue Tasks 5949 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded. 5950 INFO: Creating image(s)... 5951 5952 INFO: The new image(s) can be found here: 5953 ./directdisksdb-gpt-201710090938-sdb.direct 5954 5955 The following build artifacts were used to create the image(s): 5956 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs 5957 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share 5958 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86 5959 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native 5960 5961 INFO: The image(s) were created using OE kickstart file: 5962 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks 5963 5964Continuing with the example, you can now directly ``dd`` the image to a 5965USB stick, or whatever media for which you built your image, and boot 5966the resulting media:: 5967 5968 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb 5969 140966+0 records in 5970 140966+0 records out 5971 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s 5972 $ sudo eject /dev/sdb 5973 5974Using a Modified Kickstart File and Running in Raw Mode 5975~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5976 5977This next example manually specifies each build artifact (runs in Raw 5978Mode) and uses a modified kickstart file. The example also uses the 5979``-o`` option to cause Wic to create the output somewhere other than the 5980default output directory, which is the current directory:: 5981 5982 $ wic create test.wks -o /home/stephano/testwic \ 5983 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \ 5984 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \ 5985 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \ 5986 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native 5987 5988 INFO: Creating image(s)... 5989 5990 INFO: The new image(s) can be found here: 5991 /home/stephano/testwic/test-201710091445-sdb.direct 5992 5993 The following build artifacts were used to create the image(s): 5994 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs 5995 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share 5996 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86 5997 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native 5998 5999 INFO: The image(s) were created using OE kickstart file: 6000 test.wks 6001 6002For this example, 6003:term:`MACHINE` did not have to be 6004specified in the ``local.conf`` file since the artifact is manually 6005specified. 6006 6007Using Wic to Manipulate an Image 6008~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6009 6010Wic image manipulation allows you to shorten turnaround time during 6011image development. For example, you can use Wic to delete the kernel 6012partition of a Wic image and then insert a newly built kernel. This 6013saves you time from having to rebuild the entire image each time you 6014modify the kernel. 6015 6016.. note:: 6017 6018 In order to use Wic to manipulate a Wic image as in this example, 6019 your development machine must have the ``mtools`` package installed. 6020 6021The following example examines the contents of the Wic image, deletes 6022the existing kernel, and then inserts a new kernel: 6023 60241. *List the Partitions:* Use the ``wic ls`` command to list all the 6025 partitions in the Wic image:: 6026 6027 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic 6028 Num Start End Size Fstype 6029 1 1048576 25041919 23993344 fat16 6030 2 25165824 72157183 46991360 ext4 6031 6032 The previous output shows two partitions in the 6033 ``core-image-minimal-qemux86.wic`` image. 6034 60352. *Examine a Particular Partition:* Use the ``wic ls`` command again 6036 but in a different form to examine a particular partition. 6037 6038 .. note:: 6039 6040 You can get command usage on any Wic command using the following 6041 form:: 6042 6043 $ wic help command 6044 6045 6046 For example, the following command shows you the various ways to 6047 use the 6048 wic ls 6049 command:: 6050 6051 $ wic help ls 6052 6053 6054 The following command shows what is in partition one:: 6055 6056 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1 6057 Volume in drive : is boot 6058 Volume Serial Number is E894-1809 6059 Directory for ::/ 6060 6061 libcom32 c32 186500 2017-10-09 16:06 6062 libutil c32 24148 2017-10-09 16:06 6063 syslinux cfg 220 2017-10-09 16:06 6064 vesamenu c32 27104 2017-10-09 16:06 6065 vmlinuz 6904608 2017-10-09 16:06 6066 5 files 7 142 580 bytes 6067 16 582 656 bytes free 6068 6069 The previous output shows five files, with the 6070 ``vmlinuz`` being the kernel. 6071 6072 .. note:: 6073 6074 If you see the following error, you need to update or create a 6075 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1" 6076 in the file. Then, run the Wic command again:: 6077 6078 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0 6079 output: Total number of sectors (47824) not a multiple of sectors per track (32)! 6080 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test 6081 6082 60833. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the 6084 ``vmlinuz`` file (kernel):: 6085 6086 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz 6087 60884. *Add In the New Kernel:* Use the ``wic cp`` command to add the 6089 updated kernel to the Wic image. Depending on how you built your 6090 kernel, it could be in different places. If you used ``devtool`` and 6091 an SDK to build your kernel, it resides in the ``tmp/work`` directory 6092 of the extensible SDK. If you used ``make`` to build the kernel, the 6093 kernel will be in the ``workspace/sources`` area. 6094 6095 The following example assumes ``devtool`` was used to build the 6096 kernel:: 6097 6098 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \ 6099 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz 6100 6101 Once the new kernel is added back into the image, you can use the 6102 ``dd`` command or :ref:`bmaptool 6103 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>` 6104 to flash your wic image onto an SD card or USB stick and test your 6105 target. 6106 6107 .. note:: 6108 6109 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``. 6110 6111Flashing Images Using ``bmaptool`` 6112================================== 6113 6114A fast and easy way to flash an image to a bootable device is to use 6115Bmaptool, which is integrated into the OpenEmbedded build system. 6116Bmaptool is a generic tool that creates a file's block map (bmap) and 6117then uses that map to copy the file. As compared to traditional tools 6118such as dd or cp, Bmaptool can copy (or flash) large files like raw 6119system image files much faster. 6120 6121.. note:: 6122 6123 - If you are using Ubuntu or Debian distributions, you can install 6124 the ``bmap-tools`` package using the following command and then 6125 use the tool without specifying ``PATH`` even from the root 6126 account:: 6127 6128 $ sudo apt install bmap-tools 6129 6130 - If you are unable to install the ``bmap-tools`` package, you will 6131 need to build Bmaptool before using it. Use the following command:: 6132 6133 $ bitbake bmap-tools-native 6134 6135Following, is an example that shows how to flash a Wic image. Realize 6136that while this example uses a Wic image, you can use Bmaptool to flash 6137any type of image. Use these steps to flash an image using Bmaptool: 6138 61391. *Update your local.conf File:* You need to have the following set 6140 in your ``local.conf`` file before building your image:: 6141 6142 IMAGE_FSTYPES += "wic wic.bmap" 6143 61442. *Get Your Image:* Either have your image ready (pre-built with the 6145 :term:`IMAGE_FSTYPES` 6146 setting previously mentioned) or take the step to build the image:: 6147 6148 $ bitbake image 6149 61503. *Flash the Device:* Flash the device with the image by using Bmaptool 6151 depending on your particular setup. The following commands assume the 6152 image resides in the Build Directory's ``deploy/images/`` area: 6153 6154 - If you have write access to the media, use this command form:: 6155 6156 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX 6157 6158 - If you do not have write access to the media, set your permissions 6159 first and then use the same command form:: 6160 6161 $ sudo chmod 666 /dev/sdX 6162 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX 6163 6164For help on the ``bmaptool`` command, use the following command:: 6165 6166 $ bmaptool --help 6167 6168Making Images More Secure 6169========================= 6170 6171Security is of increasing concern for embedded devices. Consider the 6172issues and problems discussed in just this sampling of work found across 6173the Internet: 6174 6175- *"*\ `Security Risks of Embedded 6176 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"* 6177 by Bruce Schneier 6178 6179- *"*\ `Internet Census 6180 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna 6181 Botnet 6182 6183- *"*\ `Security Issues for Embedded 6184 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"* 6185 by Jake Edge 6186 6187When securing your image is of concern, there are steps, tools, and 6188variables that you can consider to help you reach the security goals you 6189need for your particular device. Not all situations are identical when 6190it comes to making an image secure. Consequently, this section provides 6191some guidance and suggestions for consideration when you want to make 6192your image more secure. 6193 6194.. note:: 6195 6196 Because the security requirements and risks are different for every 6197 type of device, this section cannot provide a complete reference on 6198 securing your custom OS. It is strongly recommended that you also 6199 consult other sources of information on embedded Linux system 6200 hardening and on security. 6201 6202General Considerations 6203---------------------- 6204 6205There are general considerations that help you create more secure images. 6206You should consider the following suggestions to make your device 6207more secure: 6208 6209- Scan additional code you are adding to the system (e.g. application 6210 code) by using static analysis tools. Look for buffer overflows and 6211 other potential security problems. 6212 6213- Pay particular attention to the security for any web-based 6214 administration interface. 6215 6216 Web interfaces typically need to perform administrative functions and 6217 tend to need to run with elevated privileges. Thus, the consequences 6218 resulting from the interface's security becoming compromised can be 6219 serious. Look for common web vulnerabilities such as 6220 cross-site-scripting (XSS), unvalidated inputs, and so forth. 6221 6222 As with system passwords, the default credentials for accessing a 6223 web-based interface should not be the same across all devices. This 6224 is particularly true if the interface is enabled by default as it can 6225 be assumed that many end-users will not change the credentials. 6226 6227- Ensure you can update the software on the device to mitigate 6228 vulnerabilities discovered in the future. This consideration 6229 especially applies when your device is network-enabled. 6230 6231- Ensure you remove or disable debugging functionality before producing 6232 the final image. For information on how to do this, see the 6233 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`" 6234 section. 6235 6236- Ensure you have no network services listening that are not needed. 6237 6238- Remove any software from the image that is not needed. 6239 6240- Enable hardware support for secure boot functionality when your 6241 device supports this functionality. 6242 6243Security Flags 6244-------------- 6245 6246The Yocto Project has security flags that you can enable that help make 6247your build output more secure. The security flags are in the 6248``meta/conf/distro/include/security_flags.inc`` file in your 6249:term:`Source Directory` (e.g. ``poky``). 6250 6251.. note:: 6252 6253 Depending on the recipe, certain security flags are enabled and 6254 disabled by default. 6255 6256Use the following line in your ``local.conf`` file or in your custom 6257distribution configuration file to enable the security compiler and 6258linker flags for your build:: 6259 6260 require conf/distro/include/security_flags.inc 6261 6262Considerations Specific to the OpenEmbedded Build System 6263-------------------------------------------------------- 6264 6265You can take some steps that are specific to the OpenEmbedded build 6266system to make your images more secure: 6267 6268- Ensure "debug-tweaks" is not one of your selected 6269 :term:`IMAGE_FEATURES`. 6270 When creating a new project, the default is to provide you with an 6271 initial ``local.conf`` file that enables this feature using the 6272 :term:`EXTRA_IMAGE_FEATURES` 6273 variable with the line:: 6274 6275 EXTRA_IMAGE_FEATURES = "debug-tweaks" 6276 6277 To disable that feature, simply comment out that line in your 6278 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain 6279 "debug-tweaks" before producing your final image. Among other things, 6280 leaving this in place sets the root password as blank, which makes 6281 logging in for debugging or inspection easy during development but 6282 also means anyone can easily log in during production. 6283 6284- It is possible to set a root password for the image and also to set 6285 passwords for any extra users you might add (e.g. administrative or 6286 service type users). When you set up passwords for multiple images or 6287 users, you should not duplicate passwords. 6288 6289 To set up passwords, use the 6290 :ref:`extrausers <ref-classes-extrausers>` 6291 class, which is the preferred method. For an example on how to set up 6292 both root and user passwords, see the 6293 ":ref:`ref-classes-extrausers`" section. 6294 6295 .. note:: 6296 6297 When adding extra user accounts or setting a root password, be 6298 cautious about setting the same password on every device. If you 6299 do this, and the password you have set is exposed, then every 6300 device is now potentially compromised. If you need this access but 6301 want to ensure security, consider setting a different, random 6302 password for each device. Typically, you do this as a separate 6303 step after you deploy the image onto the device. 6304 6305- Consider enabling a Mandatory Access Control (MAC) framework such as 6306 SMACK or SELinux and tuning it appropriately for your device's usage. 6307 You can find more information in the 6308 :yocto_git:`meta-selinux </meta-selinux/>` layer. 6309 6310Tools for Hardening Your Image 6311------------------------------ 6312 6313The Yocto Project provides tools for making your image more secure. You 6314can find these tools in the ``meta-security`` layer of the 6315:yocto_git:`Yocto Project Source Repositories <>`. 6316 6317Creating Your Own Distribution 6318============================== 6319 6320When you build an image using the Yocto Project and do not alter any 6321distribution :term:`Metadata`, you are 6322creating a Poky distribution. If you wish to gain more control over 6323package alternative selections, compile-time options, and other 6324low-level configurations, you can create your own distribution. 6325 6326To create your own distribution, the basic steps consist of creating 6327your own distribution layer, creating your own distribution 6328configuration file, and then adding any needed code and Metadata to the 6329layer. The following steps provide some more detail: 6330 6331- *Create a layer for your new distro:* Create your distribution layer 6332 so that you can keep your Metadata and code for the distribution 6333 separate. It is strongly recommended that you create and use your own 6334 layer for configuration and code. Using your own layer as compared to 6335 just placing configurations in a ``local.conf`` configuration file 6336 makes it easier to reproduce the same build configuration when using 6337 multiple build machines. See the 6338 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 6339 section for information on how to quickly set up a layer. 6340 6341- *Create the distribution configuration file:* The distribution 6342 configuration file needs to be created in the ``conf/distro`` 6343 directory of your layer. You need to name it using your distribution 6344 name (e.g. ``mydistro.conf``). 6345 6346 .. note:: 6347 6348 The :term:`DISTRO` variable in your ``local.conf`` file determines the 6349 name of your distribution. 6350 6351 You can split out parts of your configuration file into include files 6352 and then "require" them from within your distribution configuration 6353 file. Be sure to place the include files in the 6354 ``conf/distro/include`` directory of your layer. A common example 6355 usage of include files would be to separate out the selection of 6356 desired version and revisions for individual recipes. 6357 6358 Your configuration file needs to set the following required 6359 variables: 6360 6361 - :term:`DISTRO_NAME` 6362 6363 - :term:`DISTRO_VERSION` 6364 6365 These following variables are optional and you typically set them 6366 from the distribution configuration file: 6367 6368 - :term:`DISTRO_FEATURES` 6369 6370 - :term:`DISTRO_EXTRA_RDEPENDS` 6371 6372 - :term:`DISTRO_EXTRA_RRECOMMENDS` 6373 6374 - :term:`TCLIBC` 6375 6376 .. tip:: 6377 6378 If you want to base your distribution configuration file on the 6379 very basic configuration from OE-Core, you can use 6380 ``conf/distro/defaultsetup.conf`` as a reference and just include 6381 variables that differ as compared to ``defaultsetup.conf``. 6382 Alternatively, you can create a distribution configuration file 6383 from scratch using the ``defaultsetup.conf`` file or configuration files 6384 from another distribution such as Poky as a reference. 6385 6386- *Provide miscellaneous variables:* Be sure to define any other 6387 variables for which you want to create a default or enforce as part 6388 of the distribution configuration. You can include nearly any 6389 variable from the ``local.conf`` file. The variables you use are not 6390 limited to the list in the previous bulleted item. 6391 6392- *Point to Your distribution configuration file:* In your 6393 ``local.conf`` file in the :term:`Build Directory`, 6394 set your 6395 :term:`DISTRO` variable to point to 6396 your distribution's configuration file. For example, if your 6397 distribution's configuration file is named ``mydistro.conf``, then 6398 you point to it as follows:: 6399 6400 DISTRO = "mydistro" 6401 6402- *Add more to the layer if necessary:* Use your layer to hold other 6403 information needed for the distribution: 6404 6405 - Add recipes for installing distro-specific configuration files 6406 that are not already installed by another recipe. If you have 6407 distro-specific configuration files that are included by an 6408 existing recipe, you should add an append file (``.bbappend``) for 6409 those. For general information and recommendations on how to add 6410 recipes to your layer, see the 6411 ":ref:`dev-manual/common-tasks:creating your own layer`" and 6412 ":ref:`dev-manual/common-tasks:following best practices when creating layers`" 6413 sections. 6414 6415 - Add any image recipes that are specific to your distribution. 6416 6417 - Add a ``psplash`` append file for a branded splash screen. For 6418 information on append files, see the 6419 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 6420 section. 6421 6422 - Add any other append files to make custom changes that are 6423 specific to individual recipes. 6424 6425Creating a Custom Template Configuration Directory 6426================================================== 6427 6428If you are producing your own customized version of the build system for 6429use by other users, you might want to customize the message shown by the 6430setup script or you might want to change the template configuration 6431files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a 6432new build directory. 6433 6434The OpenEmbedded build system uses the environment variable 6435``TEMPLATECONF`` to locate the directory from which it gathers 6436configuration information that ultimately ends up in the 6437:term:`Build Directory` ``conf`` directory. 6438By default, ``TEMPLATECONF`` is set as follows in the ``poky`` 6439repository:: 6440 6441 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf} 6442 6443This is the 6444directory used by the build system to find templates from which to build 6445some key configuration files. If you look at this directory, you will 6446see the ``bblayers.conf.sample``, ``local.conf.sample``, and 6447``conf-notes.txt`` files. The build system uses these files to form the 6448respective ``bblayers.conf`` file, ``local.conf`` file, and display the 6449list of BitBake targets when running the setup script. 6450 6451To override these default configuration files with configurations you 6452want used within every new Build Directory, simply set the 6453``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF`` 6454variable is set in the ``.templateconf`` file, which is in the top-level 6455:term:`Source Directory` folder 6456(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your 6457directory. 6458 6459Best practices dictate that you should keep your template configuration 6460directory in your custom distribution layer. For example, suppose you 6461have a layer named ``meta-mylayer`` located in your home directory and 6462you want your template configuration directory named ``myconf``. 6463Changing the ``.templateconf`` as follows causes the OpenEmbedded build 6464system to look in your directory and base its configuration files on the 6465``*.sample`` configuration files it finds. The final configuration files 6466(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in 6467your Build Directory, but they are based on your ``*.sample`` files. 6468:: 6469 6470 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf} 6471 6472Aside from the ``*.sample`` configuration files, the ``conf-notes.txt`` 6473also resides in the default ``meta-poky/conf`` directory. The script 6474that sets up the build environment (i.e. 6475:ref:`structure-core-script`) uses this file to 6476display BitBake targets as part of the script output. Customizing this 6477``conf-notes.txt`` file is a good way to make sure your list of custom 6478targets appears as part of the script's output. 6479 6480Here is the default list of targets displayed as a result of running 6481either of the setup scripts:: 6482 6483 You can now run 'bitbake <target>' 6484 6485 Common targets are: 6486 core-image-minimal 6487 core-image-sato 6488 meta-toolchain 6489 meta-ide-support 6490 6491Changing the listed common targets is as easy as editing your version of 6492``conf-notes.txt`` in your custom template configuration directory and 6493making sure you have ``TEMPLATECONF`` set to your directory. 6494 6495Conserving Disk Space 6496===================== 6497 6498Conserving Disk Space During Builds 6499----------------------------------- 6500 6501To help conserve disk space during builds, you can add the following 6502statement to your project's ``local.conf`` configuration file found in 6503the :term:`Build Directory`:: 6504 6505 INHERIT += "rm_work" 6506 6507Adding this statement deletes the work directory used for 6508building a recipe once the recipe is built. For more information on 6509"rm_work", see the 6510:ref:`rm_work <ref-classes-rm-work>` class in the 6511Yocto Project Reference Manual. 6512 6513Purging Duplicate Shared State Cache Files 6514------------------------------------------- 6515 6516After multiple build iterations, the Shared State (sstate) cache can contain 6517duplicate cache files for a given package, while only the most recent one 6518is likely to be reusable. The following command purges all but the 6519newest sstate cache file for each package:: 6520 6521 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache 6522 6523This command will ask you to confirm the deletions it identifies. 6524 6525Note:: 6526 6527 The duplicated sstate cache files of one package must have the same 6528 architecture, which means that sstate cache files with multiple 6529 architectures are not considered as duplicate. 6530 6531Run ``sstate-cache-management.sh`` for more details about this script. 6532 6533Working with Packages 6534===================== 6535 6536This section describes a few tasks that involve packages: 6537 6538- :ref:`dev-manual/common-tasks:excluding packages from an image` 6539 6540- :ref:`dev-manual/common-tasks:incrementing a package version` 6541 6542- :ref:`dev-manual/common-tasks:handling optional module packaging` 6543 6544- :ref:`dev-manual/common-tasks:using runtime package management` 6545 6546- :ref:`dev-manual/common-tasks:generating and using signed packages` 6547 6548- :ref:`Setting up and running package test 6549 (ptest) <dev-manual/common-tasks:testing packages with ptest>` 6550 6551- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages` 6552 6553- :ref:`dev-manual/common-tasks:adding custom metadata to packages` 6554 6555Excluding Packages from an Image 6556-------------------------------- 6557 6558You might find it necessary to prevent specific packages from being 6559installed into an image. If so, you can use several variables to direct 6560the build system to essentially ignore installing recommended packages 6561or to not install a package at all. 6562 6563The following list introduces variables you can use to prevent packages 6564from being installed into your image. Each of these variables only works 6565with IPK and RPM package types, not for Debian packages. 6566Also, you can use these variables from your ``local.conf`` file 6567or attach them to a specific image recipe by using a recipe name 6568override. For more detail on the variables, see the descriptions in the 6569Yocto Project Reference Manual's glossary chapter. 6570 6571- :term:`BAD_RECOMMENDATIONS`: 6572 Use this variable to specify "recommended-only" packages that you do 6573 not want installed. 6574 6575- :term:`NO_RECOMMENDATIONS`: 6576 Use this variable to prevent all "recommended-only" packages from 6577 being installed. 6578 6579- :term:`PACKAGE_EXCLUDE`: 6580 Use this variable to prevent specific packages from being installed 6581 regardless of whether they are "recommended-only" or not. You need to 6582 realize that the build process could fail with an error when you 6583 prevent the installation of a package whose presence is required by 6584 an installed package. 6585 6586Incrementing a Package Version 6587------------------------------ 6588 6589This section provides some background on how binary package versioning 6590is accomplished and presents some of the services, variables, and 6591terminology involved. 6592 6593In order to understand binary package versioning, you need to consider 6594the following: 6595 6596- Binary Package: The binary package that is eventually built and 6597 installed into an image. 6598 6599- Binary Package Version: The binary package version is composed of two 6600 components - a version and a revision. 6601 6602 .. note:: 6603 6604 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved 6605 but this discussion for the most part ignores :term:`PE`. 6606 6607 The version and revision are taken from the 6608 :term:`PV` and 6609 :term:`PR` variables, respectively. 6610 6611- :term:`PV`: The recipe version. :term:`PV` represents the version of the 6612 software being packaged. Do not confuse :term:`PV` with the binary 6613 package version. 6614 6615- :term:`PR`: The recipe revision. 6616 6617- :term:`SRCPV`: The OpenEmbedded 6618 build system uses this string to help define the value of :term:`PV` when 6619 the source code revision needs to be included in it. 6620 6621- :yocto_wiki:`PR Service </PR_Service>`: A 6622 network-based service that helps automate keeping package feeds 6623 compatible with existing package manager applications such as RPM, 6624 APT, and OPKG. 6625 6626Whenever the binary package content changes, the binary package version 6627must change. Changing the binary package version is accomplished by 6628changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these 6629values occurs one of two ways: 6630 6631- Automatically using a Package Revision Service (PR Service). 6632 6633- Manually incrementing the :term:`PR` and/or :term:`PV` variables. 6634 6635Given a primary challenge of any build system and its users is how to 6636maintain a package feed that is compatible with existing package manager 6637applications such as RPM, APT, and OPKG, using an automated system is 6638much preferred over a manual system. In either system, the main 6639requirement is that binary package version numbering increases in a 6640linear fashion and that there is a number of version components that 6641support that linear progression. For information on how to ensure 6642package revisioning remains linear, see the 6643":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" 6644section. 6645 6646The following three sections provide related information on the PR 6647Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on 6648how to ensure binary package revisioning remains linear. 6649 6650Working With a PR Service 6651~~~~~~~~~~~~~~~~~~~~~~~~~ 6652 6653As mentioned, attempting to maintain revision numbers in the 6654:term:`Metadata` is error prone, inaccurate, 6655and causes problems for people submitting recipes. Conversely, the PR 6656Service automatically generates increasing numbers, particularly the 6657revision field, which removes the human element. 6658 6659.. note:: 6660 6661 For additional information on using a PR Service, you can see the 6662 :yocto_wiki:`PR Service </PR_Service>` wiki page. 6663 6664The Yocto Project uses variables in order of decreasing priority to 6665facilitate revision numbering (i.e. 6666:term:`PE`, 6667:term:`PV`, and 6668:term:`PR` for epoch, version, and 6669revision, respectively). The values are highly dependent on the policies 6670and procedures of a given distribution and package feed. 6671 6672Because the OpenEmbedded build system uses 6673":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are 6674unique to a given build, the build system knows when to rebuild 6675packages. All the inputs into a given task are represented by a 6676signature, which can trigger a rebuild when different. Thus, the build 6677system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to 6678trigger a rebuild. The signatures, however, can be used to generate 6679these values. 6680 6681The PR Service works with both ``OEBasic`` and ``OEBasicHash`` 6682generators. The value of :term:`PR` bumps when the checksum changes and the 6683different generator mechanisms change signatures under different 6684circumstances. 6685 6686As implemented, the build system includes values from the PR Service 6687into the :term:`PR` field as an addition using the form "``.x``" so ``r0`` 6688becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing 6689:term:`PR` values to be used for whatever reasons, which include manual 6690:term:`PR` bumps, should it be necessary. 6691 6692By default, the PR Service is not enabled or running. Thus, the packages 6693generated are just "self consistent". The build system adds and removes 6694packages and there are no guarantees about upgrade paths but images will 6695be consistent and correct with the latest changes. 6696 6697The simplest form for a PR Service is for a single host 6698development system that builds the package feed (building system). For 6699this scenario, you can enable a local PR Service by setting 6700:term:`PRSERV_HOST` in your 6701``local.conf`` file in the :term:`Build Directory`:: 6702 6703 PRSERV_HOST = "localhost:0" 6704 6705Once the service is started, packages will automatically 6706get increasing :term:`PR` values and BitBake takes care of starting and 6707stopping the server. 6708 6709If you have a more complex setup where multiple host development systems 6710work against a common, shared package feed, you have a single PR Service 6711running and it is connected to each building system. For this scenario, 6712you need to start the PR Service using the ``bitbake-prserv`` command:: 6713 6714 bitbake-prserv --host ip --port port --start 6715 6716In addition to 6717hand-starting the service, you need to update the ``local.conf`` file of 6718each building system as described earlier so each system points to the 6719server and port. 6720 6721It is also recommended you use build history, which adds some sanity 6722checks to binary package versions, in conjunction with the server that 6723is running the PR Service. To enable build history, add the following to 6724each building system's ``local.conf`` file:: 6725 6726 # It is recommended to activate "buildhistory" for testing the PR service 6727 INHERIT += "buildhistory" 6728 BUILDHISTORY_COMMIT = "1" 6729 6730For information on build 6731history, see the 6732":ref:`dev-manual/common-tasks:maintaining build output quality`" section. 6733 6734.. note:: 6735 6736 The OpenEmbedded build system does not maintain :term:`PR` information as 6737 part of the shared state (sstate) packages. If you maintain an sstate 6738 feed, it's expected that either all your building systems that 6739 contribute to the sstate feed use a shared PR Service, or you do not 6740 run a PR Service on any of your building systems. Having some systems 6741 use a PR Service while others do not leads to obvious problems. 6742 6743 For more information on shared state, see the 6744 ":ref:`overview-manual/concepts:shared state cache`" 6745 section in the Yocto Project Overview and Concepts Manual. 6746 6747Manually Bumping PR 6748~~~~~~~~~~~~~~~~~~~ 6749 6750The alternative to setting up a PR Service is to manually "bump" the 6751:term:`PR` variable. 6752 6753If a committed change results in changing the package output, then the 6754value of the PR variable needs to be increased (or "bumped") as part of 6755that commit. For new recipes you should add the :term:`PR` variable and set 6756its initial value equal to "r0", which is the default. Even though the 6757default value is "r0", the practice of adding it to a new recipe makes 6758it harder to forget to bump the variable when you make changes to the 6759recipe in future. 6760 6761If you are sharing a common ``.inc`` file with multiple recipes, you can 6762also use the :term:`INC_PR` variable to ensure that the recipes sharing the 6763``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The 6764``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes 6765referring to it should set :term:`PR` to "${INC_PR}.0" initially, 6766incrementing the last number when the recipe is changed. If the ``.inc`` 6767file is changed then its :term:`INC_PR` should be incremented. 6768 6769When upgrading the version of a binary package, assuming the :term:`PV` 6770changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0" 6771if you are using :term:`INC_PR`). 6772 6773Usually, version increases occur only to binary packages. However, if 6774for some reason :term:`PV` changes but does not increase, you can increase 6775the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to 6776"0". 6777 6778Binary package version numbering strives to follow the `Debian Version 6779Field Policy 6780Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__. 6781These guidelines define how versions are compared and what "increasing" 6782a version means. 6783 6784Automatically Incrementing a Package Version Number 6785~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6786 6787When fetching a repository, BitBake uses the 6788:term:`SRCREV` variable to determine 6789the specific source code revision from which to build. You set the 6790:term:`SRCREV` variable to 6791:term:`AUTOREV` to cause the 6792OpenEmbedded build system to automatically use the latest revision of 6793the software:: 6794 6795 SRCREV = "${AUTOREV}" 6796 6797Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to 6798automatically update the version whenever the revision of the source 6799code changes. Here is an example:: 6800 6801 PV = "1.0+git${SRCPV}" 6802 6803The OpenEmbedded build system substitutes :term:`SRCPV` with the following: 6804 6805.. code-block:: none 6806 6807 AUTOINC+source_code_revision 6808 6809The build system replaces the ``AUTOINC`` 6810with a number. The number used depends on the state of the PR Service: 6811 6812- If PR Service is enabled, the build system increments the number, 6813 which is similar to the behavior of 6814 :term:`PR`. This behavior results in 6815 linearly increasing package versions, which is desirable. Here is an 6816 example: 6817 6818 .. code-block:: none 6819 6820 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk 6821 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk 6822 6823- If PR Service is not enabled, the build system replaces the 6824 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in 6825 changing the package version since the source revision is included. 6826 However, package versions are not increased linearly. Here is an 6827 example: 6828 6829 .. code-block:: none 6830 6831 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk 6832 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk 6833 6834In summary, the OpenEmbedded build system does not track the history of 6835binary package versions for this purpose. ``AUTOINC``, in this case, is 6836comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the 6837package version is simply replaced by "0". If PR server is enabled, the 6838build system keeps track of the package versions and bumps the number 6839when the package revision changes. 6840 6841Handling Optional Module Packaging 6842---------------------------------- 6843 6844Many pieces of software split functionality into optional modules (or 6845plugins) and the plugins that are built might depend on configuration 6846options. To avoid having to duplicate the logic that determines what 6847modules are available in your recipe or to avoid having to package each 6848module by hand, the OpenEmbedded build system provides functionality to 6849handle module packaging dynamically. 6850 6851To handle optional module packaging, you need to do two things: 6852 6853- Ensure the module packaging is actually done. 6854 6855- Ensure that any dependencies on optional modules from other recipes 6856 are satisfied by your recipe. 6857 6858Making Sure the Packaging is Done 6859~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6860 6861To ensure the module packaging actually gets done, you use the 6862``do_split_packages`` function within the ``populate_packages`` Python 6863function in your recipe. The ``do_split_packages`` function searches for 6864a pattern of files or directories under a specified path and creates a 6865package for each one it finds by appending to the 6866:term:`PACKAGES` variable and 6867setting the appropriate values for ``FILES:packagename``, 6868``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth. 6869Here is an example from the ``lighttpd`` recipe:: 6870 6871 python populate_packages:prepend () { 6872 lighttpd_libdir = d.expand('${libdir}') 6873 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$', 6874 'lighttpd-module-%s', 'Lighttpd module for %s', 6875 extra_depends='') 6876 } 6877 6878The previous example specifies a number of things in the call to 6879``do_split_packages``. 6880 6881- A directory within the files installed by your recipe through 6882 ``do_install`` in which to search. 6883 6884- A regular expression used to match module files in that directory. In 6885 the example, note the parentheses () that mark the part of the 6886 expression from which the module name should be derived. 6887 6888- A pattern to use for the package names. 6889 6890- A description for each package. 6891 6892- An empty string for ``extra_depends``, which disables the default 6893 dependency on the main ``lighttpd`` package. Thus, if a file in 6894 ``${libdir}`` called ``mod_alias.so`` is found, a package called 6895 ``lighttpd-module-alias`` is created for it and the 6896 :term:`DESCRIPTION` is set to 6897 "Lighttpd module for alias". 6898 6899Often, packaging modules is as simple as the previous example. However, 6900there are more advanced options that you can use within 6901``do_split_packages`` to modify its behavior. And, if you need to, you 6902can add more logic by specifying a hook function that is called for each 6903package. It is also perfectly acceptable to call ``do_split_packages`` 6904multiple times if you have more than one set of modules to package. 6905 6906For more examples that show how to use ``do_split_packages``, see the 6907``connman.inc`` file in the ``meta/recipes-connectivity/connman/`` 6908directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can 6909also find examples in ``meta/classes/kernel.bbclass``. 6910 6911Following is a reference that shows ``do_split_packages`` mandatory and 6912optional arguments:: 6913 6914 Mandatory arguments 6915 6916 root 6917 The path in which to search 6918 file_regex 6919 Regular expression to match searched files. 6920 Use parentheses () to mark the part of this 6921 expression that should be used to derive the 6922 module name (to be substituted where %s is 6923 used in other function arguments as noted below) 6924 output_pattern 6925 Pattern to use for the package names. Must 6926 include %s. 6927 description 6928 Description to set for each package. Must 6929 include %s. 6930 6931 Optional arguments 6932 6933 postinst 6934 Postinstall script to use for all packages 6935 (as a string) 6936 recursive 6937 True to perform a recursive search - default 6938 False 6939 hook 6940 A hook function to be called for every match. 6941 The function will be called with the following 6942 arguments (in the order listed): 6943 6944 f 6945 Full path to the file/directory match 6946 pkg 6947 The package name 6948 file_regex 6949 As above 6950 output_pattern 6951 As above 6952 modulename 6953 The module name derived using file_regex 6954 extra_depends 6955 Extra runtime dependencies (RDEPENDS) to be 6956 set for all packages. The default value of None 6957 causes a dependency on the main package 6958 (${PN}) - if you do not want this, pass empty 6959 string '' for this parameter. 6960 aux_files_pattern 6961 Extra item(s) to be added to FILES for each 6962 package. Can be a single string item or a list 6963 of strings for multiple items. Must include %s. 6964 postrm 6965 postrm script to use for all packages (as a 6966 string) 6967 allow_dirs 6968 True to allow directories to be matched - 6969 default False 6970 prepend 6971 If True, prepend created packages to PACKAGES 6972 instead of the default False which appends them 6973 match_path 6974 match file_regex on the whole relative path to 6975 the root rather than just the filename 6976 aux_files_pattern_verbatim 6977 Extra item(s) to be added to FILES for each 6978 package, using the actual derived module name 6979 rather than converting it to something legal 6980 for a package name. Can be a single string item 6981 or a list of strings for multiple items. Must 6982 include %s. 6983 allow_links 6984 True to allow symlinks to be matched - default 6985 False 6986 summary 6987 Summary to set for each package. Must include %s; 6988 defaults to description if not set. 6989 6990 6991 6992Satisfying Dependencies 6993~~~~~~~~~~~~~~~~~~~~~~~ 6994 6995The second part for handling optional module packaging is to ensure that 6996any dependencies on optional modules from other recipes are satisfied by 6997your recipe. You can be sure these dependencies are satisfied by using 6998the :term:`PACKAGES_DYNAMIC` 6999variable. Here is an example that continues with the ``lighttpd`` recipe 7000shown earlier:: 7001 7002 PACKAGES_DYNAMIC = "lighttpd-module-.*" 7003 7004The name 7005specified in the regular expression can of course be anything. In this 7006example, it is ``lighttpd-module-`` and is specified as the prefix to 7007ensure that any :term:`RDEPENDS` and 7008:term:`RRECOMMENDS` on a package 7009name starting with the prefix are satisfied during build time. If you 7010are using ``do_split_packages`` as described in the previous section, 7011the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name 7012pattern specified in the call to ``do_split_packages``. 7013 7014Using Runtime Package Management 7015-------------------------------- 7016 7017During a build, BitBake always transforms a recipe into one or more 7018packages. For example, BitBake takes the ``bash`` recipe and produces a 7019number of packages (e.g. ``bash``, ``bash-bashbug``, 7020``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``, 7021``bash-completion-extra``, ``bash-dbg``, and so forth). Not all 7022generated packages are included in an image. 7023 7024In several situations, you might need to update, add, remove, or query 7025the packages on a target device at runtime (i.e. without having to 7026generate a new image). Examples of such situations include: 7027 7028- You want to provide in-the-field updates to deployed devices (e.g. 7029 security updates). 7030 7031- You want to have a fast turn-around development cycle for one or more 7032 applications that run on your device. 7033 7034- You want to temporarily install the "debug" packages of various 7035 applications on your device so that debugging can be greatly improved 7036 by allowing access to symbols and source debugging. 7037 7038- You want to deploy a more minimal package selection of your device 7039 but allow in-the-field updates to add a larger selection for 7040 customization. 7041 7042In all these situations, you have something similar to a more 7043traditional Linux distribution in that in-field devices are able to 7044receive pre-compiled packages from a server for installation or update. 7045Being able to install these packages on a running, in-field device is 7046what is termed "runtime package management". 7047 7048In order to use runtime package management, you need a host or server 7049machine that serves up the pre-compiled packages plus the required 7050metadata. You also need package manipulation tools on the target. The 7051build machine is a likely candidate to act as the server. However, that 7052machine does not necessarily have to be the package server. The build 7053machine could push its artifacts to another machine that acts as the 7054server (e.g. Internet-facing). In fact, doing so is advantageous for a 7055production environment as getting the packages away from the development 7056system's build directory prevents accidental overwrites. 7057 7058A simple build that targets just one device produces more than one 7059package database. In other words, the packages produced by a build are 7060separated out into a couple of different package groupings based on 7061criteria such as the target's CPU architecture, the target board, or the 7062C library used on the target. For example, a build targeting the 7063``qemux86`` device produces the following three package databases: 7064``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86`` 7065device to be aware of all the packages that were available to it, you 7066would need to point it to each of these databases individually. In a 7067similar way, a traditional Linux distribution usually is configured to 7068be aware of a number of software repositories from which it retrieves 7069packages. 7070 7071Using runtime package management is completely optional and not required 7072for a successful build or deployment in any way. But if you want to make 7073use of runtime package management, you need to do a couple things above 7074and beyond the basics. The remainder of this section describes what you 7075need to do. 7076 7077Build Considerations 7078~~~~~~~~~~~~~~~~~~~~ 7079 7080This section describes build considerations of which you need to be 7081aware in order to provide support for runtime package management. 7082 7083When BitBake generates packages, it needs to know what format or formats 7084to use. In your configuration, you use the 7085:term:`PACKAGE_CLASSES` 7086variable to specify the format: 7087 70881. Open the ``local.conf`` file inside your 7089 :term:`Build Directory` (e.g. 7090 ``poky/build/conf/local.conf``). 7091 70922. Select the desired package format as follows:: 7093 7094 PACKAGE_CLASSES ?= "package_packageformat" 7095 7096 where packageformat can be "ipk", "rpm", 7097 "deb", or "tar" which are the supported package formats. 7098 7099 .. note:: 7100 7101 Because the Yocto Project supports four different package formats, 7102 you can set the variable with more than one argument. However, the 7103 OpenEmbedded build system only uses the first argument when 7104 creating an image or Software Development Kit (SDK). 7105 7106If you would like your image to start off with a basic package database 7107containing the packages in your current build as well as to have the 7108relevant tools available on the target for runtime package management, 7109you can include "package-management" in the 7110:term:`IMAGE_FEATURES` 7111variable. Including "package-management" in this configuration variable 7112ensures that when the image is assembled for your target, the image 7113includes the currently-known package databases as well as the 7114target-specific tools required for runtime package management to be 7115performed on the target. However, this is not strictly necessary. You 7116could start your image off without any databases but only include the 7117required on-target package tool(s). As an example, you could include 7118"opkg" in your 7119:term:`IMAGE_INSTALL` variable 7120if you are using the IPK package format. You can then initialize your 7121target's package database(s) later once your image is up and running. 7122 7123Whenever you perform any sort of build step that can potentially 7124generate a package or modify existing package, it is always a good idea 7125to re-generate the package index after the build by using the following 7126command:: 7127 7128 $ bitbake package-index 7129 7130It might be tempting to build the 7131package and the package index at the same time with a command such as 7132the following:: 7133 7134 $ bitbake some-package package-index 7135 7136Do not do this as 7137BitBake does not schedule the package index for after the completion of 7138the package you are building. Consequently, you cannot be sure of the 7139package index including information for the package you just built. 7140Thus, be sure to run the package update step separately after building 7141any packages. 7142 7143You can use the 7144:term:`PACKAGE_FEED_ARCHS`, 7145:term:`PACKAGE_FEED_BASE_PATHS`, 7146and 7147:term:`PACKAGE_FEED_URIS` 7148variables to pre-configure target images to use a package feed. If you 7149do not define these variables, then manual steps as described in the 7150subsequent sections are necessary to configure the target. You should 7151set these variables before building the image in order to produce a 7152correctly configured image. 7153 7154When your build is complete, your packages reside in the 7155``${TMPDIR}/deploy/packageformat`` directory. For example, if 7156``${``\ :term:`TMPDIR`\ ``}`` is 7157``tmp`` and your selected package type is RPM, then your RPM packages 7158are available in ``tmp/deploy/rpm``. 7159 7160Host or Server Machine Setup 7161~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7162 7163Although other protocols are possible, a server using HTTP typically 7164serves packages. If you want to use HTTP, then set up and configure a 7165web server such as Apache 2, lighttpd, or Python web server on the 7166machine serving the packages. 7167 7168To keep things simple, this section describes how to set up a 7169Python web server to share package feeds from the developer's 7170machine. Although this server might not be the best for a production 7171environment, the setup is simple and straight forward. Should you want 7172to use a different server more suited for production (e.g. Apache 2, 7173Lighttpd, or Nginx), take the appropriate steps to do so. 7174 7175From within the build directory where you have built an image based on 7176your packaging choice (i.e. the 7177:term:`PACKAGE_CLASSES` 7178setting), simply start the server. The following example assumes a build 7179directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES` 7180setting of "package_rpm":: 7181 7182 $ cd poky/build/tmp/deploy/rpm 7183 $ python3 -m http.server 7184 7185Target Setup 7186~~~~~~~~~~~~ 7187 7188Setting up the target differs depending on the package management 7189system. This section provides information for RPM, IPK, and DEB. 7190 7191Using RPM 7192^^^^^^^^^ 7193 7194The `Dandified Packaging 7195Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs 7196runtime package management of RPM packages. In order to use DNF for 7197runtime package management, you must perform an initial setup on the 7198target machine for cases where the ``PACKAGE_FEED_*`` variables were not 7199set as part of the image that is running on the target. This means if 7200you built your image and did not use these variables as part of the 7201build and your image is now running on the target, you need to perform 7202the steps in this section if you want to use runtime package management. 7203 7204.. note:: 7205 7206 For information on the ``PACKAGE_FEED_*`` variables, see 7207 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and 7208 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables 7209 glossary. 7210 7211On the target, you must inform DNF that package databases are available. 7212You do this by creating a file named 7213``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``. 7214 7215As an example, assume the target is able to use the following package 7216databases: ``all``, ``i586``, and ``qemux86`` from a server named 7217``my.server``. The specifics for setting up the web server are up to 7218you. The critical requirement is that the URIs in the target repository 7219configuration point to the correct remote location for the feeds. 7220 7221.. note:: 7222 7223 For development purposes, you can point the web server to the build 7224 system's ``deploy`` directory. However, for production use, it is better to 7225 copy the package directories to a location outside of the build area and use 7226 that location. Doing so avoids situations where the build system 7227 overwrites or changes the ``deploy`` directory. 7228 7229When telling DNF where to look for the package databases, you must 7230declare individual locations per architecture or a single location used 7231for all architectures. You cannot do both: 7232 7233- *Create an Explicit List of Architectures:* Define individual base 7234 URLs to identify where each package database is located: 7235 7236 .. code-block:: none 7237 7238 [oe-packages] 7239 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all 7240 7241 This example 7242 informs DNF about individual package databases for all three 7243 architectures. 7244 7245- *Create a Single (Full) Package Index:* Define a single base URL that 7246 identifies where a full package database is located:: 7247 7248 [oe-packages] 7249 baseurl=http://my.server/rpm 7250 7251 This example informs DNF about a single 7252 package database that contains all the package index information for 7253 all supported architectures. 7254 7255Once you have informed DNF where to find the package databases, you need 7256to fetch them: 7257 7258.. code-block:: none 7259 7260 # dnf makecache 7261 7262DNF is now able to find, install, and 7263upgrade packages from the specified repository or repositories. 7264 7265.. note:: 7266 7267 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for 7268 additional information. 7269 7270Using IPK 7271^^^^^^^^^ 7272 7273The ``opkg`` application performs runtime package management of IPK 7274packages. You must perform an initial setup for ``opkg`` on the target 7275machine if the 7276:term:`PACKAGE_FEED_ARCHS`, 7277:term:`PACKAGE_FEED_BASE_PATHS`, 7278and 7279:term:`PACKAGE_FEED_URIS` 7280variables have not been set or the target image was built before the 7281variables were set. 7282 7283The ``opkg`` application uses configuration files to find available 7284package databases. Thus, you need to create a configuration file inside 7285the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository 7286you want to use. 7287 7288As an example, suppose you are serving packages from a ``ipk/`` 7289directory containing the ``i586``, ``all``, and ``qemux86`` databases 7290through an HTTP server named ``my.server``. On the target, create a 7291configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/`` 7292directory containing the following: 7293 7294.. code-block:: none 7295 7296 src/gz all http://my.server/ipk/all 7297 src/gz i586 http://my.server/ipk/i586 7298 src/gz qemux86 http://my.server/ipk/qemux86 7299 7300Next, instruct ``opkg`` to fetch the 7301repository information: 7302 7303.. code-block:: none 7304 7305 # opkg update 7306 7307The ``opkg`` application is now able to find, install, and upgrade packages 7308from the specified repository. 7309 7310Using DEB 7311^^^^^^^^^ 7312 7313The ``apt`` application performs runtime package management of DEB 7314packages. This application uses a source list file to find available 7315package databases. You must perform an initial setup for ``apt`` on the 7316target machine if the 7317:term:`PACKAGE_FEED_ARCHS`, 7318:term:`PACKAGE_FEED_BASE_PATHS`, 7319and 7320:term:`PACKAGE_FEED_URIS` 7321variables have not been set or the target image was built before the 7322variables were set. 7323 7324To inform ``apt`` of the repository you want to use, you might create a 7325list file (e.g. ``my_repo.list``) inside the 7326``/etc/apt/sources.list.d/`` directory. As an example, suppose you are 7327serving packages from a ``deb/`` directory containing the ``i586``, 7328``all``, and ``qemux86`` databases through an HTTP server named 7329``my.server``. The list file should contain: 7330 7331.. code-block:: none 7332 7333 deb http://my.server/deb/all ./ 7334 deb http://my.server/deb/i586 ./ 7335 deb http://my.server/deb/qemux86 ./ 7336 7337Next, instruct the ``apt`` application 7338to fetch the repository information: 7339 7340.. code-block:: none 7341 7342 $ sudo apt update 7343 7344After this step, 7345``apt`` is able to find, install, and upgrade packages from the 7346specified repository. 7347 7348Generating and Using Signed Packages 7349------------------------------------ 7350 7351In order to add security to RPM packages used during a build, you can 7352take steps to securely sign them. Once a signature is verified, the 7353OpenEmbedded build system can use the package in the build. If security 7354fails for a signed package, the build system stops the build. 7355 7356This section describes how to sign RPM packages during a build and how 7357to use signed package feeds (repositories) when doing a build. 7358 7359Signing RPM Packages 7360~~~~~~~~~~~~~~~~~~~~ 7361 7362To enable signing RPM packages, you must set up the following 7363configurations in either your ``local.config`` or ``distro.config`` 7364file:: 7365 7366 # Inherit sign_rpm.bbclass to enable signing functionality 7367 INHERIT += " sign_rpm" 7368 # Define the GPG key that will be used for signing. 7369 RPM_GPG_NAME = "key_name" 7370 # Provide passphrase for the key 7371 RPM_GPG_PASSPHRASE = "passphrase" 7372 7373.. note:: 7374 7375 Be sure to supply appropriate values for both `key_name` and 7376 `passphrase`. 7377 7378Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in 7379the previous example, two optional variables related to signing are available: 7380 7381- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed 7382 when the package is signed. 7383 7384- *GPG_PATH:* Specifies the ``gpg`` home directory used when the 7385 package is signed. 7386 7387Processing Package Feeds 7388~~~~~~~~~~~~~~~~~~~~~~~~ 7389 7390In addition to being able to sign RPM packages, you can also enable 7391signed package feeds for IPK and RPM packages. 7392 7393The steps you need to take to enable signed package feed use are similar 7394to the steps used to sign RPM packages. You must define the following in 7395your ``local.config`` or ``distro.config`` file:: 7396 7397 INHERIT += "sign_package_feed" 7398 PACKAGE_FEED_GPG_NAME = "key_name" 7399 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase" 7400 7401For signed package feeds, the passphrase must be specified in a separate file, 7402which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` 7403variable. Regarding security, keeping a plain text passphrase out of the 7404configuration is more secure. 7405 7406Aside from the ``PACKAGE_FEED_GPG_NAME`` and 7407``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables 7408related to signed package feeds are available: 7409 7410- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed 7411 when the package is signed. 7412 7413- *GPG_PATH:* Specifies the ``gpg`` home directory used when the 7414 package is signed. 7415 7416- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg`` 7417 signature. This variable applies only to RPM and IPK package feeds. 7418 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are 7419 "ASC", which is the default and specifies ascii armored, and "BIN", 7420 which specifies binary. 7421 7422Testing Packages With ptest 7423--------------------------- 7424 7425A Package Test (ptest) runs tests against packages built by the 7426OpenEmbedded build system on the target machine. A ptest contains at 7427least two items: the actual test, and a shell script (``run-ptest``) 7428that starts the test. The shell script that starts the test must not 7429contain the actual test - the script only starts the test. On the other 7430hand, the test can be anything from a simple shell script that runs a 7431binary and checks the output to an elaborate system of test binaries and 7432data files. 7433 7434The test generates output in the format used by Automake:: 7435 7436 result: testname 7437 7438where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and 7439the testname can be any identifying string. 7440 7441For a list of Yocto Project recipes that are already enabled with ptest, 7442see the :yocto_wiki:`Ptest </Ptest>` wiki page. 7443 7444.. note:: 7445 7446 A recipe is "ptest-enabled" if it inherits the 7447 :ref:`ptest <ref-classes-ptest>` class. 7448 7449Adding ptest to Your Build 7450~~~~~~~~~~~~~~~~~~~~~~~~~~ 7451 7452To add package testing to your build, add the 7453:term:`DISTRO_FEATURES` and 7454:term:`EXTRA_IMAGE_FEATURES` 7455variables to your ``local.conf`` file, which is found in the 7456:term:`Build Directory`:: 7457 7458 DISTRO_FEATURES:append = " ptest" 7459 EXTRA_IMAGE_FEATURES += "ptest-pkgs" 7460 7461Once your build is complete, the ptest files are installed into the 7462``/usr/lib/package/ptest`` directory within the image, where ``package`` 7463is the name of the package. 7464 7465Running ptest 7466~~~~~~~~~~~~~ 7467 7468The ``ptest-runner`` package installs a shell script that loops through 7469all installed ptest test suites and runs them in sequence. Consequently, 7470you might want to add this package to your image. 7471 7472Getting Your Package Ready 7473~~~~~~~~~~~~~~~~~~~~~~~~~~ 7474 7475In order to enable a recipe to run installed ptests on target hardware, 7476you need to prepare the recipes that build the packages you want to 7477test. Here is what you have to do for each recipe: 7478 7479- *Be sure the recipe inherits 7480 the* :ref:`ptest <ref-classes-ptest>` *class:* 7481 Include the following line in each recipe:: 7482 7483 inherit ptest 7484 7485- *Create run-ptest:* This script starts your test. Locate the 7486 script where you will refer to it using 7487 :term:`SRC_URI`. Here is an 7488 example that starts a test for ``dbus``:: 7489 7490 #!/bin/sh 7491 cd test 7492 make -k runtest-TESTS 7493 7494- *Ensure dependencies are met:* If the test adds build or runtime 7495 dependencies that normally do not exist for the package (such as 7496 requiring "make" to run the test suite), use the 7497 :term:`DEPENDS` and 7498 :term:`RDEPENDS` variables in 7499 your recipe in order for the package to meet the dependencies. Here 7500 is an example where the package has a runtime dependency on "make":: 7501 7502 RDEPENDS:${PN}-ptest += "make" 7503 7504- *Add a function to build the test suite:* Not many packages support 7505 cross-compilation of their test suites. Consequently, you usually 7506 need to add a cross-compilation function to the package. 7507 7508 Many packages based on Automake compile and run the test suite by 7509 using a single command such as ``make check``. However, the host 7510 ``make check`` builds and runs on the same computer, while 7511 cross-compiling requires that the package is built on the host but 7512 executed for the target architecture (though often, as in the case 7513 for ptest, the execution occurs on the host). The built version of 7514 Automake that ships with the Yocto Project includes a patch that 7515 separates building and execution. Consequently, packages that use the 7516 unaltered, patched version of ``make check`` automatically 7517 cross-compiles. 7518 7519 Regardless, you still must add a ``do_compile_ptest`` function to 7520 build the test suite. Add a function similar to the following to your 7521 recipe:: 7522 7523 do_compile_ptest() { 7524 oe_runmake buildtest-TESTS 7525 } 7526 7527- *Ensure special configurations are set:* If the package requires 7528 special configurations prior to compiling the test code, you must 7529 insert a ``do_configure_ptest`` function into the recipe. 7530 7531- *Install the test suite:* The ``ptest`` class automatically copies 7532 the file ``run-ptest`` to the target and then runs make 7533 ``install-ptest`` to run the tests. If this is not enough, you need 7534 to create a ``do_install_ptest`` function and make sure it gets 7535 called after the "make install-ptest" completes. 7536 7537Creating Node Package Manager (NPM) Packages 7538-------------------------------------------- 7539 7540`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package 7541manager for the JavaScript programming language. The Yocto Project 7542supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can 7543use this fetcher in combination with 7544:doc:`devtool </ref-manual/devtool-reference>` to create 7545recipes that produce NPM packages. 7546 7547There are two workflows that allow you to create NPM packages using 7548``devtool``: the NPM registry modules method and the NPM project code 7549method. 7550 7551.. note:: 7552 7553 While it is possible to create NPM recipes manually, using 7554 ``devtool`` is far simpler. 7555 7556Additionally, some requirements and caveats exist. 7557 7558Requirements and Caveats 7559~~~~~~~~~~~~~~~~~~~~~~~~ 7560 7561You need to be aware of the following before using ``devtool`` to create 7562NPM packages: 7563 7564- Of the two methods that you can use ``devtool`` to create NPM 7565 packages, the registry approach is slightly simpler. However, you 7566 might consider the project approach because you do not have to 7567 publish your module in the NPM registry 7568 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which 7569 is NPM's public registry. 7570 7571- Be familiar with 7572 :doc:`devtool </ref-manual/devtool-reference>`. 7573 7574- The NPM host tools need the native ``nodejs-npm`` package, which is 7575 part of the OpenEmbedded environment. You need to get the package by 7576 cloning the https://github.com/openembedded/meta-openembedded 7577 repository out of GitHub. Be sure to add the path to your local copy 7578 to your ``bblayers.conf`` file. 7579 7580- ``devtool`` cannot detect native libraries in module dependencies. 7581 Consequently, you must manually add packages to your recipe. 7582 7583- While deploying NPM packages, ``devtool`` cannot determine which 7584 dependent packages are missing on the target (e.g. the node runtime 7585 ``nodejs``). Consequently, you need to find out what files are 7586 missing and be sure they are on the target. 7587 7588- Although you might not need NPM to run your node package, it is 7589 useful to have NPM on your target. The NPM package name is 7590 ``nodejs-npm``. 7591 7592Using the Registry Modules Method 7593~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7594 7595This section presents an example that uses the ``cute-files`` module, 7596which is a file browser web application. 7597 7598.. note:: 7599 7600 You must know the ``cute-files`` module version. 7601 7602The first thing you need to do is use ``devtool`` and the NPM fetcher to 7603create the recipe:: 7604 7605 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2" 7606 7607The 7608``devtool add`` command runs ``recipetool create`` and uses the same 7609fetch URI to download each dependency and capture license details where 7610possible. The result is a generated recipe. 7611 7612The recipe file is fairly simple and contains every license that 7613``recipetool`` finds and includes the licenses in the recipe's 7614:term:`LIC_FILES_CHKSUM` 7615variables. You need to examine the variables and look for those with 7616"unknown" in the :term:`LICENSE` 7617field. You need to track down the license information for "unknown" 7618modules and manually add the information to the recipe. 7619 7620``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap 7621files capture the version of all dependent modules. Many packages do not 7622provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it 7623runs. 7624 7625.. note:: 7626 7627 A package is created for each sub-module. This policy is the only 7628 practical way to have the licenses for all of the dependencies 7629 represented in the license manifest of the image. 7630 7631The ``devtool edit-recipe`` command lets you take a look at the recipe:: 7632 7633 $ devtool edit-recipe cute-files 7634 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network." 7635 LICENSE = "MIT & ISC & Unknown" 7636 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \ 7637 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \ 7638 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \ 7639 ... 7640 SRC_URI = " \ 7641 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 7642 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 7643 " 7644 S = "${WORKDIR}/npm" 7645 inherit npm 7646 LICENSE:${PN} = "MIT" 7647 LICENSE:${PN}-accepts = "MIT" 7648 LICENSE:${PN}-array-flatten = "MIT" 7649 ... 7650 LICENSE:${PN}-vary = "MIT" 7651 7652Here are three key points in the previous example: 7653 7654- :term:`SRC_URI` uses the NPM 7655 scheme so that the NPM fetcher is used. 7656 7657- ``recipetool`` collects all the license information. If a 7658 sub-module's license is unavailable, the sub-module's name appears in 7659 the comments. 7660 7661- The ``inherit npm`` statement causes the 7662 :ref:`npm <ref-classes-npm>` class to package 7663 up all the modules. 7664 7665You can run the following command to build the ``cute-files`` package:: 7666 7667 $ devtool build cute-files 7668 7669Remember that ``nodejs`` must be installed on 7670the target before your package. 7671 7672Assuming 192.168.7.2 for the target's IP address, use the following 7673command to deploy your package:: 7674 7675 $ devtool deploy-target -s cute-files root@192.168.7.2 7676 7677Once the package is installed on the target, you can 7678test the application: 7679 7680.. note:: 7681 7682 Because of a known issue, you cannot simply run ``cute-files`` as you would 7683 if you had run ``npm install``. 7684 7685:: 7686 7687 $ cd /usr/lib/node_modules/cute-files 7688 $ node cute-files.js 7689 7690On a browser, 7691go to ``http://192.168.7.2:3000`` and you see the following: 7692 7693.. image:: figures/cute-files-npm-example.png 7694 :align: center 7695 7696You can find the recipe in ``workspace/recipes/cute-files``. You can use 7697the recipe in any layer you choose. 7698 7699Using the NPM Projects Code Method 7700~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7701 7702Although it is useful to package modules already in the NPM registry, 7703adding ``node.js`` projects under development is a more common developer 7704use case. 7705 7706This section covers the NPM projects code method, which is very similar 7707to the "registry" approach described in the previous section. In the NPM 7708projects method, you provide ``devtool`` with an URL that points to the 7709source files. 7710 7711Replicating the same example, (i.e. ``cute-files``) use the following 7712command:: 7713 7714 $ devtool add https://github.com/martinaglv/cute-files.git 7715 7716The 7717recipe this command generates is very similar to the recipe created in 7718the previous section. However, the :term:`SRC_URI` looks like the following:: 7719 7720 SRC_URI = " \ 7721 git://github.com/martinaglv/cute-files.git;protocol=https \ 7722 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 7723 " 7724 7725In this example, 7726the main module is taken from the Git repository and dependencies are 7727taken from the NPM registry. Other than those differences, the recipe is 7728basically the same between the two methods. You can build and deploy the 7729package exactly as described in the previous section that uses the 7730registry modules method. 7731 7732Adding custom metadata to packages 7733---------------------------------- 7734 7735The variable 7736:term:`PACKAGE_ADD_METADATA` 7737can be used to add additional metadata to packages. This is reflected in 7738the package control/spec file. To take the ipk format for example, the 7739CONTROL file stored inside would contain the additional metadata as 7740additional lines. 7741 7742The variable can be used in multiple ways, including using suffixes to 7743set it for a specific package type and/or package. Note that the order 7744of precedence is the same as this list: 7745 7746- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>`` 7747 7748- ``PACKAGE_ADD_METADATA_<PKGTYPE>`` 7749 7750- ``PACKAGE_ADD_METADATA:<PN>`` 7751 7752- :term:`PACKAGE_ADD_METADATA` 7753 7754`<PKGTYPE>` is a parameter and expected to be a distinct name of specific 7755package type: 7756 7757- IPK for .ipk packages 7758 7759- DEB for .deb packages 7760 7761- RPM for .rpm packages 7762 7763`<PN>` is a parameter and expected to be a package name. 7764 7765The variable can contain multiple [one-line] metadata fields separated 7766by the literal sequence '\\n'. The separator can be redefined using the 7767variable flag ``separator``. 7768 7769Here is an example that adds two custom fields for ipk 7770packages:: 7771 7772 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets" 7773 7774Efficiently Fetching Source Files During a Build 7775================================================ 7776 7777The OpenEmbedded build system works with source files located through 7778the :term:`SRC_URI` variable. When 7779you build something using BitBake, a big part of the operation is 7780locating and downloading all the source tarballs. For images, 7781downloading all the source for various packages can take a significant 7782amount of time. 7783 7784This section shows you how you can use mirrors to speed up fetching 7785source files and how you can pre-fetch files all of which leads to more 7786efficient use of resources and time. 7787 7788Setting up Effective Mirrors 7789---------------------------- 7790 7791A good deal that goes into a Yocto Project build is simply downloading 7792all of the source tarballs. Maybe you have been working with another 7793build system for which you have built up a 7794sizable directory of source tarballs. Or, perhaps someone else has such 7795a directory for which you have read access. If so, you can save time by 7796adding statements to your configuration file so that the build process 7797checks local directories first for existing tarballs before checking the 7798Internet. 7799 7800Here is an efficient way to set it up in your ``local.conf`` file:: 7801 7802 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/" 7803 INHERIT += "own-mirrors" 7804 BB_GENERATE_MIRROR_TARBALLS = "1" 7805 # BB_NO_NETWORK = "1" 7806 7807In the previous example, the 7808:term:`BB_GENERATE_MIRROR_TARBALLS` 7809variable causes the OpenEmbedded build system to generate tarballs of 7810the Git repositories and store them in the 7811:term:`DL_DIR` directory. Due to 7812performance reasons, generating and storing these tarballs is not the 7813build system's default behavior. 7814 7815You can also use the 7816:term:`PREMIRRORS` variable. For 7817an example, see the variable's glossary entry in the Yocto Project 7818Reference Manual. 7819 7820Getting Source Files and Suppressing the Build 7821---------------------------------------------- 7822 7823Another technique you can use to ready yourself for a successive string 7824of build operations, is to pre-fetch all the source files without 7825actually starting a build. This technique lets you work through any 7826download issues and ultimately gathers all the source files into your 7827download directory :ref:`structure-build-downloads`, 7828which is located with :term:`DL_DIR`. 7829 7830Use the following BitBake command form to fetch all the necessary 7831sources without starting the build:: 7832 7833 $ bitbake target --runall=fetch 7834 7835This 7836variation of the BitBake command guarantees that you have all the 7837sources for that BitBake target should you disconnect from the Internet 7838and want to do the build later offline. 7839 7840Selecting an Initialization Manager 7841=================================== 7842 7843By default, the Yocto Project uses SysVinit as the initialization 7844manager. However, there is also support for systemd, which is a full 7845replacement for init with parallel starting of services, reduced shell 7846overhead and other features that are used by many distributions. 7847 7848Within the system, SysVinit treats system components as services. These 7849services are maintained as shell scripts stored in the ``/etc/init.d/`` 7850directory. Services organize into different run levels. This 7851organization is maintained by putting links to the services in the 7852``/etc/rcN.d/`` directories, where `N/` is one of the following options: 7853"S", "0", "1", "2", "3", "4", "5", or "6". 7854 7855.. note:: 7856 7857 Each runlevel has a dependency on the previous runlevel. This 7858 dependency allows the services to work properly. 7859 7860In comparison, systemd treats components as units. Using units is a 7861broader concept as compared to using a service. A unit includes several 7862different types of entities. Service is one of the types of entities. 7863The runlevel concept in SysVinit corresponds to the concept of a target 7864in systemd, where target is also a type of supported unit. 7865 7866In a SysVinit-based system, services load sequentially (i.e. one by one) 7867during init and parallelization is not supported. With systemd, services 7868start in parallel. Needless to say, the method can have an impact on 7869system startup performance. 7870 7871If you want to use SysVinit, you do not have to do anything. But, if you 7872want to use systemd, you must take some steps as described in the 7873following sections. 7874 7875Using systemd Exclusively 7876------------------------- 7877 7878Set these variables in your distribution configuration file as follows:: 7879 7880 DISTRO_FEATURES:append = " systemd" 7881 VIRTUAL-RUNTIME_init_manager = "systemd" 7882 7883You can also prevent the SysVinit distribution feature from 7884being automatically enabled as follows:: 7885 7886 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit" 7887 7888Doing so removes any 7889redundant SysVinit scripts. 7890 7891To remove initscripts from your image altogether, set this variable 7892also:: 7893 7894 VIRTUAL-RUNTIME_initscripts = "" 7895 7896For information on the backfill variable, see 7897:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`. 7898 7899Using systemd for the Main Image and Using SysVinit for the Rescue Image 7900------------------------------------------------------------------------ 7901 7902Set these variables in your distribution configuration file as follows:: 7903 7904 DISTRO_FEATURES:append = " systemd" 7905 VIRTUAL-RUNTIME_init_manager = "systemd" 7906 7907Doing so causes your main image to use the 7908``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal 7909image cannot use this package group. However, it can install SysVinit 7910and the appropriate packages will have support for both systemd and 7911SysVinit. 7912 7913Using systemd-journald without a traditional syslog daemon 7914---------------------------------------------------------- 7915 7916Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider, 7917and the proper way to use systemd-journald as your sole logging mechanism is to 7918effectively disable syslog entirely by setting these variables in your distribution 7919configuration file:: 7920 7921 VIRTUAL-RUNTIME_syslog = "" 7922 VIRTUAL-RUNTIME_base-utils-syslog = "" 7923 7924Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by 7925default, leaving only ``journald``. 7926 7927Selecting a Device Manager 7928========================== 7929 7930The Yocto Project provides multiple ways to manage the device manager 7931(``/dev``): 7932 7933- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev`` 7934 directory is persistent and the required device nodes are created 7935 during the build. 7936 7937- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev`` 7938 directory is provided by the kernel as an in-memory file system and 7939 is automatically populated by the kernel at runtime. Additional 7940 configuration of device nodes is done in user space by a device 7941 manager like ``udev`` or ``busybox-mdev``. 7942 7943Using Persistent and Pre-Populated ``/dev`` 7944-------------------------------------------- 7945 7946To use the static method for device population, you need to set the 7947:term:`USE_DEVFS` variable to "0" 7948as follows:: 7949 7950 USE_DEVFS = "0" 7951 7952The content of the resulting ``/dev`` directory is defined in a Device 7953Table file. The 7954:term:`IMAGE_DEVICE_TABLES` 7955variable defines the Device Table to use and should be set in the 7956machine or distro configuration file. Alternatively, you can set this 7957variable in your ``local.conf`` configuration file. 7958 7959If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default 7960``device_table-minimal.txt`` is used:: 7961 7962 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt" 7963 7964The population is handled by the ``makedevs`` utility during image 7965creation: 7966 7967Using ``devtmpfs`` and a Device Manager 7968--------------------------------------- 7969 7970To use the dynamic method for device population, you need to use (or be 7971sure to set) the :term:`USE_DEVFS` 7972variable to "1", which is the default:: 7973 7974 USE_DEVFS = "1" 7975 7976With this 7977setting, the resulting ``/dev`` directory is populated by the kernel 7978using ``devtmpfs``. Make sure the corresponding kernel configuration 7979variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux 7980kernel. 7981 7982All devices created by ``devtmpfs`` will be owned by ``root`` and have 7983permissions ``0600``. 7984 7985To have more control over the device nodes, you can use a device manager 7986like ``udev`` or ``busybox-mdev``. You choose the device manager by 7987defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or 7988distro configuration file. Alternatively, you can set this variable in 7989your ``local.conf`` configuration file:: 7990 7991 VIRTUAL-RUNTIME_dev_manager = "udev" 7992 7993 # Some alternative values 7994 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev" 7995 # VIRTUAL-RUNTIME_dev_manager = "systemd" 7996 7997Using an External SCM 7998===================== 7999 8000If you're working on a recipe that pulls from an external Source Code 8001Manager (SCM), it is possible to have the OpenEmbedded build system 8002notice new recipe changes added to the SCM and then build the resulting 8003packages that depend on the new recipes by using the latest versions. 8004This only works for SCMs from which it is possible to get a sensible 8005revision number for changes. Currently, you can do this with Apache 8006Subversion (SVN), Git, and Bazaar (BZR) repositories. 8007 8008To enable this behavior, the :term:`PV` of 8009the recipe needs to reference 8010:term:`SRCPV`. Here is an example:: 8011 8012 PV = "1.2.3+git${SRCPV}" 8013 8014Then, you can add the following to your 8015``local.conf``:: 8016 8017 SRCREV:pn-PN = "${AUTOREV}" 8018 8019:term:`PN` is the name of the recipe for 8020which you want to enable automatic source revision updating. 8021 8022If you do not want to update your local configuration file, you can add 8023the following directly to the recipe to finish enabling the feature:: 8024 8025 SRCREV = "${AUTOREV}" 8026 8027The Yocto Project provides a distribution named ``poky-bleeding``, whose 8028configuration file contains the line:: 8029 8030 require conf/distro/include/poky-floating-revisions.inc 8031 8032This line pulls in the 8033listed include file that contains numerous lines of exactly that form:: 8034 8035 #SRCREV:pn-opkg-native ?= "${AUTOREV}" 8036 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}" 8037 #SRCREV:pn-opkg ?= "${AUTOREV}" 8038 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}" 8039 #SRCREV:pn-opkg-utils ?= "${AUTOREV}" 8040 SRCREV:pn-gconf-dbus ?= "${AUTOREV}" 8041 SRCREV:pn-matchbox-common ?= "${AUTOREV}" 8042 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}" 8043 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}" 8044 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}" 8045 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}" 8046 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}" 8047 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}" 8048 SRCREV:pn-matchbox-wm ?= "${AUTOREV}" 8049 SRCREV:pn-settings-daemon ?= "${AUTOREV}" 8050 SRCREV:pn-screenshot ?= "${AUTOREV}" 8051 . . . 8052 8053These lines allow you to 8054experiment with building a distribution that tracks the latest 8055development source for numerous packages. 8056 8057.. note:: 8058 8059 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep 8060 this in mind if you use it. 8061 8062Creating a Read-Only Root Filesystem 8063==================================== 8064 8065Suppose, for security reasons, you need to disable your target device's 8066root filesystem's write permissions (i.e. you need a read-only root 8067filesystem). Or, perhaps you are running the device's operating system 8068from a read-only storage device. For either case, you can customize your 8069image for that behavior. 8070 8071.. note:: 8072 8073 Supporting a read-only root filesystem requires that the system and 8074 applications do not try to write to the root filesystem. You must 8075 configure all parts of the target system to write elsewhere, or to 8076 gracefully fail in the event of attempting to write to the root 8077 filesystem. 8078 8079Creating the Root Filesystem 8080---------------------------- 8081 8082To create the read-only root filesystem, simply add the 8083"read-only-rootfs" feature to your image, normally in one of two ways. 8084The first way is to add the "read-only-rootfs" image feature in the 8085image's recipe file via the :term:`IMAGE_FEATURES` variable:: 8086 8087 IMAGE_FEATURES += "read-only-rootfs" 8088 8089As an alternative, you can add the same feature 8090from within your build directory's ``local.conf`` file with the 8091associated :term:`EXTRA_IMAGE_FEATURES` variable, as in:: 8092 8093 EXTRA_IMAGE_FEATURES = "read-only-rootfs" 8094 8095For more information on how to use these variables, see the 8096":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``" 8097section. For information on the variables, see 8098:term:`IMAGE_FEATURES` and 8099:term:`EXTRA_IMAGE_FEATURES`. 8100 8101Post-Installation Scripts and Read-Only Root Filesystem 8102------------------------------------------------------- 8103 8104It is very important that you make sure all post-Installation 8105(``pkg_postinst``) scripts for packages that are installed into the 8106image can be run at the time when the root filesystem is created during 8107the build on the host system. These scripts cannot attempt to run during 8108the first boot on the target device. With the "read-only-rootfs" feature 8109enabled, the build system makes sure that all post-installation scripts 8110succeed at file system creation time. If any of these scripts 8111still need to be run after the root filesystem is created, the build 8112immediately fails. These build-time checks ensure that the build fails 8113rather than the target device fails later during its initial boot 8114operation. 8115 8116Most of the common post-installation scripts generated by the build 8117system for the out-of-the-box Yocto Project are engineered so that they 8118can run during root filesystem creation (e.g. post-installation scripts 8119for caching fonts). However, if you create and add custom scripts, you 8120need to be sure they can be run during this file system creation. 8121 8122Here are some common problems that prevent post-installation scripts 8123from running during root filesystem creation: 8124 8125- *Not using $D in front of absolute paths:* The build system defines 8126 ``$``\ :term:`D` when the root 8127 filesystem is created. Furthermore, ``$D`` is blank when the script 8128 is run on the target device. This implies two purposes for ``$D``: 8129 ensuring paths are valid in both the host and target environments, 8130 and checking to determine which environment is being used as a method 8131 for taking appropriate actions. 8132 8133- *Attempting to run processes that are specific to or dependent on the 8134 target architecture:* You can work around these attempts by using 8135 native tools, which run on the host system, to accomplish the same 8136 tasks, or by alternatively running the processes under QEMU, which 8137 has the ``qemu_run_binary`` function. For more information, see the 8138 :ref:`qemu <ref-classes-qemu>` class. 8139 8140Areas With Write Access 8141----------------------- 8142 8143With the "read-only-rootfs" feature enabled, any attempt by the target 8144to write to the root filesystem at runtime fails. Consequently, you must 8145make sure that you configure processes and applications that attempt 8146these types of writes do so to directories with write access (e.g. 8147``/tmp`` or ``/var/run``). 8148 8149Maintaining Build Output Quality 8150================================ 8151 8152Many factors can influence the quality of a build. For example, if you 8153upgrade a recipe to use a new version of an upstream software package or 8154you experiment with some new configuration options, subtle changes can 8155occur that you might not detect until later. Consider the case where 8156your recipe is using a newer version of an upstream package. In this 8157case, a new version of a piece of software might introduce an optional 8158dependency on another library, which is auto-detected. If that library 8159has already been built when the software is building, the software will 8160link to the built library and that library will be pulled into your 8161image along with the new software even if you did not want the library. 8162 8163The :ref:`buildhistory <ref-classes-buildhistory>` 8164class helps you maintain the quality of your build output. You 8165can use the class to highlight unexpected and possibly unwanted changes 8166in the build output. When you enable build history, it records 8167information about the contents of each package and image and then 8168commits that information to a local Git repository where you can examine 8169the information. 8170 8171The remainder of this section describes the following: 8172 8173- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>` 8174 8175- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>` 8176 8177- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>` 8178 8179- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>` 8180 8181Enabling and Disabling Build History 8182------------------------------------ 8183 8184Build history is disabled by default. To enable it, add the following 8185:term:`INHERIT` statement and set the 8186:term:`BUILDHISTORY_COMMIT` 8187variable to "1" at the end of your ``conf/local.conf`` file found in the 8188:term:`Build Directory`:: 8189 8190 INHERIT += "buildhistory" 8191 BUILDHISTORY_COMMIT = "1" 8192 8193Enabling build history as 8194previously described causes the OpenEmbedded build system to collect 8195build output information and commit it as a single commit to a local 8196:ref:`overview-manual/development-environment:git` repository. 8197 8198.. note:: 8199 8200 Enabling build history increases your build times slightly, 8201 particularly for images, and increases the amount of disk space used 8202 during the build. 8203 8204You can disable build history by removing the previous statements from 8205your ``conf/local.conf`` file. 8206 8207Understanding What the Build History Contains 8208--------------------------------------------- 8209 8210Build history information is kept in 8211``${``\ :term:`TOPDIR`\ ``}/buildhistory`` 8212in the Build Directory as defined by the 8213:term:`BUILDHISTORY_DIR` 8214variable. Here is an example abbreviated listing: 8215 8216.. image:: figures/buildhistory.png 8217 :align: center 8218 8219At the top level, there is a ``metadata-revs`` file that lists the 8220revisions of the repositories for the enabled layers when the build was 8221produced. The rest of the data splits into separate ``packages``, 8222``images`` and ``sdk`` directories, the contents of which are described 8223as follows. 8224 8225Build History Package Information 8226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8227 8228The history for each package contains a text file that has name-value 8229pairs with information about the package. For example, 8230``buildhistory/packages/i586-poky-linux/busybox/busybox/latest`` 8231contains the following: 8232 8233.. code-block:: none 8234 8235 PV = 1.22.1 8236 PR = r32 8237 RPROVIDES = 8238 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg 8239 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d 8240 PKGSIZE = 540168 8241 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ 8242 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ 8243 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ 8244 /usr/share/pixmaps /usr/share/applications /usr/share/idl \ 8245 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers 8246 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ 8247 /etc/busybox.links.nosuid /etc/busybox.links.suid 8248 8249Most of these 8250name-value pairs correspond to variables used to produce the package. 8251The exceptions are ``FILELIST``, which is the actual list of files in 8252the package, and ``PKGSIZE``, which is the total size of files in the 8253package in bytes. 8254 8255There is also a file that corresponds to the recipe from which the package 8256came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``): 8257 8258.. code-block:: none 8259 8260 PV = 1.22.1 8261 PR = r32 8262 DEPENDS = initscripts kern-tools-native update-rc.d-native \ 8263 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ 8264 virtual/libc virtual/update-alternatives 8265 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ 8266 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ 8267 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox 8268 8269Finally, for those recipes fetched from a version control system (e.g., 8270Git), there is a file that lists source revisions that are specified in 8271the recipe and the actual revisions used during the build. Listed 8272and actual revisions might differ when 8273:term:`SRCREV` is set to 8274${:term:`AUTOREV`}. Here is an 8275example assuming 8276``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):: 8277 8278 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" 8279 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" 8280 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" 8281 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f" 8282 8283You can use the 8284``buildhistory-collect-srcrevs`` command with the ``-a`` option to 8285collect the stored :term:`SRCREV` values from build history and report them 8286in a format suitable for use in global configuration (e.g., 8287``local.conf`` or a distro include file) to override floating 8288:term:`AUTOREV` values to a fixed set of revisions. Here is some example 8289output from this command:: 8290 8291 $ buildhistory-collect-srcrevs -a 8292 # all-poky-linux 8293 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8" 8294 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac" 8295 # core2-64-poky-linux 8296 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66" 8297 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8" 8298 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0" 8299 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0" 8300 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1" 8301 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c" 8302 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7" 8303 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee" 8304 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1" 8305 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100" 8306 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3" 8307 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf" 8308 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced" 8309 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8" 8310 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f" 8311 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa" 8312 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70" 8313 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3" 8314 # qemux86_64-poky-linux 8315 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c" 8316 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0" 8317 # x86_64-linux 8318 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66" 8319 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c" 8320 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87" 8321 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57" 8322 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb" 8323 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b" 8324 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3" 8325 8326.. note:: 8327 8328 Here are some notes on using the ``buildhistory-collect-srcrevs`` command: 8329 8330 - By default, only values where the :term:`SRCREV` was not hardcoded 8331 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a`` 8332 option to see all :term:`SRCREV` values. 8333 8334 - The output statements might not have any effect if overrides are 8335 applied elsewhere in the build system configuration. Use the 8336 ``-f`` option to add the ``forcevariable`` override to each output 8337 line if you need to work around this restriction. 8338 8339 - The script does apply special handling when building for multiple 8340 machines. However, the script does place a comment before each set 8341 of values that specifies which triplet to which they belong as 8342 previously shown (e.g., ``i586-poky-linux``). 8343 8344Build History Image Information 8345~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8346 8347The files produced for each image are as follows: 8348 8349- ``image-files:`` A directory containing selected files from the root 8350 filesystem. The files are defined by 8351 :term:`BUILDHISTORY_IMAGE_FILES`. 8352 8353- ``build-id.txt:`` Human-readable information about the build 8354 configuration and metadata source revisions. This file contains the 8355 full build header as printed by BitBake. 8356 8357- ``*.dot:`` Dependency graphs for the image that are compatible with 8358 ``graphviz``. 8359 8360- ``files-in-image.txt:`` A list of files in the image with 8361 permissions, owner, group, size, and symlink information. 8362 8363- ``image-info.txt:`` A text file containing name-value pairs with 8364 information about the image. See the following listing example for 8365 more information. 8366 8367- ``installed-package-names.txt:`` A list of installed packages by name 8368 only. 8369 8370- ``installed-package-sizes.txt:`` A list of installed packages ordered 8371 by size. 8372 8373- ``installed-packages.txt:`` A list of installed packages with full 8374 package filenames. 8375 8376.. note:: 8377 8378 Installed package information is able to be gathered and produced 8379 even if package management is disabled for the final image. 8380 8381Here is an example of ``image-info.txt``: 8382 8383.. code-block:: none 8384 8385 DISTRO = poky 8386 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be 8387 USER_CLASSES = buildstats 8388 IMAGE_CLASSES = qemuboot qemuboot license_image 8389 IMAGE_FEATURES = debug-tweaks 8390 IMAGE_LINGUAS = 8391 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp 8392 BAD_RECOMMENDATIONS = 8393 NO_RECOMMENDATIONS = 8394 PACKAGE_EXCLUDE = 8395 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible; 8396 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; 8397 IMAGESIZE = 9265 8398 8399Other than ``IMAGESIZE``, 8400which is the total size of the files in the image in Kbytes, the 8401name-value pairs are variables that may have influenced the content of 8402the image. This information is often useful when you are trying to 8403determine why a change in the package or file listings has occurred. 8404 8405Using Build History to Gather Image Information Only 8406~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8407 8408As you can see, build history produces image information, including 8409dependency graphs, so you can see why something was pulled into the 8410image. If you are just interested in this information and not interested 8411in collecting specific package or SDK information, you can enable 8412writing only image information without any history by adding the 8413following to your ``conf/local.conf`` file found in the 8414:term:`Build Directory`:: 8415 8416 INHERIT += "buildhistory" 8417 BUILDHISTORY_COMMIT = "0" 8418 BUILDHISTORY_FEATURES = "image" 8419 8420Here, you set the 8421:term:`BUILDHISTORY_FEATURES` 8422variable to use the image feature only. 8423 8424Build History SDK Information 8425~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8426 8427Build history collects similar information on the contents of SDKs (e.g. 8428``bitbake -c populate_sdk imagename``) as compared to information it 8429collects for images. Furthermore, this information differs depending on 8430whether an extensible or standard SDK is being produced. 8431 8432The following list shows the files produced for SDKs: 8433 8434- ``files-in-sdk.txt:`` A list of files in the SDK with permissions, 8435 owner, group, size, and symlink information. This list includes both 8436 the host and target parts of the SDK. 8437 8438- ``sdk-info.txt:`` A text file containing name-value pairs with 8439 information about the SDK. See the following listing example for more 8440 information. 8441 8442- ``sstate-task-sizes.txt:`` A text file containing name-value pairs 8443 with information about task group sizes (e.g. ``do_populate_sysroot`` 8444 tasks have a total size). The ``sstate-task-sizes.txt`` file exists 8445 only when an extensible SDK is created. 8446 8447- ``sstate-package-sizes.txt:`` A text file containing name-value pairs 8448 with information for the shared-state packages and sizes in the SDK. 8449 The ``sstate-package-sizes.txt`` file exists only when an extensible 8450 SDK is created. 8451 8452- ``sdk-files:`` A folder that contains copies of the files mentioned 8453 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output. 8454 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is 8455 specific to the extensible SDK although you can set it differently if 8456 you would like to pull in specific files from the standard SDK. 8457 8458 The default files are ``conf/local.conf``, ``conf/bblayers.conf``, 8459 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and 8460 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get 8461 copied into the ``sdk-files`` directory. 8462 8463- The following information appears under each of the ``host`` and 8464 ``target`` directories for the portions of the SDK that run on the 8465 host and on the target, respectively: 8466 8467 .. note:: 8468 8469 The following files for the most part are empty when producing an 8470 extensible SDK because this type of SDK is not constructed from 8471 packages as is the standard SDK. 8472 8473 - ``depends.dot:`` Dependency graph for the SDK that is compatible 8474 with ``graphviz``. 8475 8476 - ``installed-package-names.txt:`` A list of installed packages by 8477 name only. 8478 8479 - ``installed-package-sizes.txt:`` A list of installed packages 8480 ordered by size. 8481 8482 - ``installed-packages.txt:`` A list of installed packages with full 8483 package filenames. 8484 8485Here is an example of ``sdk-info.txt``: 8486 8487.. code-block:: none 8488 8489 DISTRO = poky 8490 DISTRO_VERSION = 1.3+snapshot-20130327 8491 SDK_NAME = poky-glibc-i686-arm 8492 SDK_VERSION = 1.3+snapshot 8493 SDKMACHINE = 8494 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs 8495 BAD_RECOMMENDATIONS = 8496 SDKSIZE = 352712 8497 8498Other than ``SDKSIZE``, which is 8499the total size of the files in the SDK in Kbytes, the name-value pairs 8500are variables that might have influenced the content of the SDK. This 8501information is often useful when you are trying to determine why a 8502change in the package or file listings has occurred. 8503 8504Examining Build History Information 8505~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8506 8507You can examine build history output from the command line or from a web 8508interface. 8509 8510To see any changes that have occurred (assuming you have 8511:term:`BUILDHISTORY_COMMIT` = "1"), 8512you can simply use any Git command that allows you to view the history 8513of a repository. Here is one method:: 8514 8515 $ git log -p 8516 8517You need to realize, 8518however, that this method does show changes that are not significant 8519(e.g. a package's size changing by a few bytes). 8520 8521There is a command-line tool called ``buildhistory-diff``, though, 8522that queries the Git repository and prints just the differences that 8523might be significant in human-readable form. Here is an example:: 8524 8525 $ poky/poky/scripts/buildhistory-diff . HEAD^ 8526 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): 8527 /etc/anotherpkg.conf was added 8528 /sbin/anotherpkg was added 8529 * (installed-package-names.txt): 8530 * anotherpkg was added 8531 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): 8532 anotherpkg was added 8533 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" 8534 * PR changed from "r0" to "r1" 8535 * PV changed from "0.1.10" to "0.1.12" 8536 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) 8537 * PR changed from "r0" to "r1" 8538 * PV changed from "0.1.10" to "0.1.12" 8539 8540.. note:: 8541 8542 The ``buildhistory-diff`` tool requires the ``GitPython`` 8543 package. Be sure to install it using Pip3 as follows:: 8544 8545 $ pip3 install GitPython --user 8546 8547 8548 Alternatively, you can install ``python3-git`` using the appropriate 8549 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``). 8550 8551To see changes to the build history using a web interface, follow the 8552instruction in the ``README`` file 8553:yocto_git:`here </buildhistory-web/>`. 8554 8555Here is a sample screenshot of the interface: 8556 8557.. image:: figures/buildhistory-web.png 8558 :align: center 8559 8560Performing Automated Runtime Testing 8561==================================== 8562 8563The OpenEmbedded build system makes available a series of automated 8564tests for images to verify runtime functionality. You can run these 8565tests on either QEMU or actual target hardware. Tests are written in 8566Python making use of the ``unittest`` module, and the majority of them 8567run commands on the target system over SSH. This section describes how 8568you set up the environment to use these tests, run available tests, and 8569write and add your own tests. 8570 8571For information on the test and QA infrastructure available within the 8572Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`" 8573section in the Yocto Project Reference Manual. 8574 8575Enabling Tests 8576-------------- 8577 8578Depending on whether you are planning to run tests using QEMU or on the 8579hardware, you have to take different steps to enable the tests. See the 8580following subsections for information on how to enable both types of 8581tests. 8582 8583Enabling Runtime Tests on QEMU 8584~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8585 8586In order to run tests, you need to do the following: 8587 8588- *Set up to avoid interaction with sudo for networking:* To 8589 accomplish this, you must do one of the following: 8590 8591 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all 8592 commands or just for ``runqemu-ifup``. You must provide the full 8593 path as that can change if you are using multiple clones of the 8594 source repository. 8595 8596 .. note:: 8597 8598 On some distributions, you also need to comment out "Defaults 8599 requiretty" in ``/etc/sudoers``. 8600 8601 - Manually configure a tap interface for your system. 8602 8603 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which 8604 should generate a list of tap devices. This is the option 8605 typically chosen for Autobuilder-type environments. 8606 8607 .. note:: 8608 8609 - Be sure to use an absolute path when calling this script 8610 with sudo. 8611 8612 - The package recipe ``qemu-helper-native`` is required to run 8613 this script. Build the package using the following command:: 8614 8615 $ bitbake qemu-helper-native 8616 8617- *Set the DISPLAY variable:* You need to set this variable so that 8618 you have an X server available (e.g. start ``vncserver`` for a 8619 headless machine). 8620 8621- *Be sure your host's firewall accepts incoming connections from 8622 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an 8623 HTTP server on a random high number port, which is used to serve 8624 files to the target. The DNF module serves 8625 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands. 8626 That means your host's firewall must accept incoming connections from 8627 192.168.7.0/24, which is the default IP range used for tap devices by 8628 ``runqemu``. 8629 8630- *Be sure your host has the correct packages installed:* Depending 8631 your host's distribution, you need to have the following packages 8632 installed: 8633 8634 - Ubuntu and Debian: ``sysstat`` and ``iproute2`` 8635 8636 - openSUSE: ``sysstat`` and ``iproute2`` 8637 8638 - Fedora: ``sysstat`` and ``iproute`` 8639 8640 - CentOS: ``sysstat`` and ``iproute`` 8641 8642Once you start running the tests, the following happens: 8643 86441. A copy of the root filesystem is written to ``${WORKDIR}/testimage``. 8645 86462. The image is booted under QEMU using the standard ``runqemu`` script. 8647 86483. A default timeout of 500 seconds occurs to allow for the boot process 8649 to reach the login prompt. You can change the timeout period by 8650 setting 8651 :term:`TEST_QEMUBOOT_TIMEOUT` 8652 in the ``local.conf`` file. 8653 86544. Once the boot process is reached and the login prompt appears, the 8655 tests run. The full boot log is written to 8656 ``${WORKDIR}/testimage/qemu_boot_log``. 8657 86585. Each test module loads in the order found in :term:`TEST_SUITES`. You can 8659 find the full output of the commands run over SSH in 8660 ``${WORKDIR}/testimgage/ssh_target_log``. 8661 86626. If no failures occur, the task running the tests ends successfully. 8663 You can find the output from the ``unittest`` in the task log at 8664 ``${WORKDIR}/temp/log.do_testimage``. 8665 8666Enabling Runtime Tests on Hardware 8667~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8668 8669The OpenEmbedded build system can run tests on real hardware, and for 8670certain devices it can also deploy the image to be tested onto the 8671device beforehand. 8672 8673For automated deployment, a "controller image" is installed onto the 8674hardware once as part of setup. Then, each time tests are to be run, the 8675following occurs: 8676 86771. The controller image is booted into and used to write the image to be 8678 tested to a second partition. 8679 86802. The device is then rebooted using an external script that you need to 8681 provide. 8682 86833. The device boots into the image to be tested. 8684 8685When running tests (independent of whether the image has been deployed 8686automatically or not), the device is expected to be connected to a 8687network on a pre-determined IP address. You can either use static IP 8688addresses written into the image, or set the image to use DHCP and have 8689your DHCP server on the test network assign a known IP address based on 8690the MAC address of the device. 8691 8692In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an 8693appropriate value. For QEMU, you do not have to change anything, the 8694default value is "qemu". For running tests on hardware, the following 8695options are available: 8696 8697- *"simpleremote":* Choose "simpleremote" if you are going to run tests 8698 on a target system that is already running the image to be tested and 8699 is available on the network. You can use "simpleremote" in 8700 conjunction with either real hardware or an image running within a 8701 separately started QEMU or any other virtual machine manager. 8702 8703- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is 8704 an EFI-based machine with ``systemd-boot`` as bootloader and 8705 ``core-image-testmaster`` (or something similar) is installed. Also, 8706 your hardware under test must be in a DHCP-enabled network that gives 8707 it the same IP address for each reboot. 8708 8709 If you choose "SystemdbootTarget", there are additional requirements 8710 and considerations. See the 8711 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which 8712 follows, for more information. 8713 8714- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying 8715 images and running tests on the BeagleBone "Black" or original 8716 "White" hardware. For information on how to use these tests, see the 8717 comments at the top of the BeagleBoneTarget 8718 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file. 8719 8720- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying 8721 images and running tests on the Ubiquiti Networks EdgeRouter Lite. 8722 For information on how to use these tests, see the comments at the 8723 top of the EdgeRouterTarget 8724 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file. 8725 8726- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running 8727 tests on any generic PC that boots using GRUB. For information on how 8728 to use these tests, see the comments at the top of the GrubTarget 8729 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file. 8730 8731- *"your-target":* Create your own custom target if you want to run 8732 tests when you are deploying images and running tests on a custom 8733 machine within your BSP layer. To do this, you need to add a Python 8734 unit that defines the target class under ``lib/oeqa/controllers/`` 8735 within your layer. You must also provide an empty ``__init__.py``. 8736 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``. 8737 8738Selecting SystemdbootTarget 8739~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8740 8741If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do 8742not need any information in this section. You can skip down to the 8743":ref:`dev-manual/common-tasks:running tests`" section. 8744 8745If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to 8746perform a one-time setup of your controller image by doing the following: 8747 87481. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows:: 8749 8750 EFI_PROVIDER = "systemd-boot" 8751 87522. *Build the controller image:* Build the ``core-image-testmaster`` image. 8753 The ``core-image-testmaster`` recipe is provided as an example for a 8754 "controller" image and you can customize the image recipe as you would 8755 any other recipe. 8756 8757 Here are the image recipe requirements: 8758 8759 - Inherits ``core-image`` so that kernel modules are installed. 8760 8761 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``, 8762 ``coreutils``, ``tar``, ``gzip``, and ``kmod``). 8763 8764 - Uses a custom Initial RAM Disk (initramfs) image with a custom 8765 installer. A normal image that you can install usually creates a 8766 single root filesystem partition. This image uses another installer that 8767 creates a specific partition layout. Not all Board Support 8768 Packages (BSPs) can use an installer. For such cases, you need to 8769 manually create the following partition layout on the target: 8770 8771 - First partition mounted under ``/boot``, labeled "boot". 8772 8773 - The main root filesystem partition where this image gets installed, 8774 which is mounted under ``/``. 8775 8776 - Another partition labeled "testrootfs" where test images get 8777 deployed. 8778 87793. *Install image:* Install the image that you just built on the target 8780 system. 8781 8782The final thing you need to do when setting :term:`TEST_TARGET` to 8783"SystemdbootTarget" is to set up the test image: 8784 87851. *Set up your local.conf file:* Make sure you have the following 8786 statements in your ``local.conf`` file:: 8787 8788 IMAGE_FSTYPES += "tar.gz" 8789 INHERIT += "testimage" 8790 TEST_TARGET = "SystemdbootTarget" 8791 TEST_TARGET_IP = "192.168.2.3" 8792 87932. *Build your test image:* Use BitBake to build the image:: 8794 8795 $ bitbake core-image-sato 8796 8797Power Control 8798~~~~~~~~~~~~~ 8799 8800For most hardware targets other than "simpleremote", you can control 8801power: 8802 8803- You can use :term:`TEST_POWERCONTROL_CMD` together with 8804 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host 8805 and does power cycling. The test code passes one argument to that 8806 command: off, on or cycle (off then on). Here is an example that 8807 could appear in your ``local.conf`` file:: 8808 8809 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" 8810 8811 In this example, the expect 8812 script does the following: 8813 8814 .. code-block:: shell 8815 8816 ssh test@10.11.12.1 "pyctl nuc1 arg" 8817 8818 It then runs a Python script that controls power for a label called 8819 ``nuc1``. 8820 8821 .. note:: 8822 8823 You need to customize :term:`TEST_POWERCONTROL_CMD` and 8824 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement 8825 is that it accepts "on", "off", and "cycle" as the last argument. 8826 8827- When no command is defined, it connects to the device over SSH and 8828 uses the classic reboot command to reboot the device. Classic reboot 8829 is fine as long as the machine actually reboots (i.e. the SSH test 8830 has not failed). It is useful for scenarios where you have a simple 8831 setup, typically with a single board, and where some manual 8832 interaction is okay from time to time. 8833 8834If you have no hardware to automatically perform power control but still 8835wish to experiment with automated hardware testing, you can use the 8836``dialog-power-control`` script that shows a dialog prompting you to perform 8837the required power action. This script requires either KDialog or Zenity 8838to be installed. To use this script, set the 8839:term:`TEST_POWERCONTROL_CMD` 8840variable as follows:: 8841 8842 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" 8843 8844Serial Console Connection 8845~~~~~~~~~~~~~~~~~~~~~~~~~ 8846 8847For test target classes requiring a serial console to interact with the 8848bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget), 8849you need to specify a command to use to connect to the serial console of 8850the target machine by using the 8851:term:`TEST_SERIALCONTROL_CMD` 8852variable and optionally the 8853:term:`TEST_SERIALCONTROL_EXTRA_ARGS` 8854variable. 8855 8856These cases could be a serial terminal program if the machine is 8857connected to a local serial port, or a ``telnet`` or ``ssh`` command 8858connecting to a remote console server. Regardless of the case, the 8859command simply needs to connect to the serial console and forward that 8860connection to standard input and output as any normal terminal program 8861does. For example, to use the picocom terminal program on serial device 8862``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:: 8863 8864 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" 8865 8866For local 8867devices where the serial port device disappears when the device reboots, 8868an additional "serdevtry" wrapper script is provided. To use this 8869wrapper, simply prefix the terminal command with 8870``${COREBASE}/scripts/contrib/serdevtry``:: 8871 8872 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0" 8873 8874Running Tests 8875------------- 8876 8877You can start the tests automatically or manually: 8878 8879- *Automatically running tests:* To run the tests automatically after 8880 the OpenEmbedded build system successfully creates an image, first 8881 set the 8882 :term:`TESTIMAGE_AUTO` 8883 variable to "1" in your ``local.conf`` file in the 8884 :term:`Build Directory`:: 8885 8886 TESTIMAGE_AUTO = "1" 8887 8888 Next, build your image. If the image successfully builds, the 8889 tests run:: 8890 8891 bitbake core-image-sato 8892 8893- *Manually running tests:* To manually run the tests, first globally 8894 inherit the 8895 :ref:`testimage <ref-classes-testimage*>` class 8896 by editing your ``local.conf`` file:: 8897 8898 INHERIT += "testimage" 8899 8900 Next, use BitBake to run the tests:: 8901 8902 bitbake -c testimage image 8903 8904All test files reside in ``meta/lib/oeqa/runtime/cases`` in the 8905:term:`Source Directory`. A test name maps 8906directly to a Python module. Each test module may contain a number of 8907individual tests. Tests are usually grouped together by the area tested 8908(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``). 8909 8910You can add tests to any layer provided you place them in the proper 8911area and you extend :term:`BBPATH` in 8912the ``local.conf`` file as normal. Be sure that tests reside in 8913``layer/lib/oeqa/runtime/cases``. 8914 8915.. note:: 8916 8917 Be sure that module names do not collide with module names used in 8918 the default set of test modules in ``meta/lib/oeqa/runtime/cases``. 8919 8920You can change the set of tests run by appending or overriding 8921:term:`TEST_SUITES` variable in 8922``local.conf``. Each name in :term:`TEST_SUITES` represents a required test 8923for the image. Test modules named within :term:`TEST_SUITES` cannot be 8924skipped even if a test is not suitable for an image (e.g. running the 8925RPM tests on an image without ``rpm``). Appending "auto" to 8926:term:`TEST_SUITES` causes the build system to try to run all tests that are 8927suitable for the image (i.e. each test module may elect to skip itself). 8928 8929The order you list tests in :term:`TEST_SUITES` is important and influences 8930test dependencies. Consequently, tests that depend on other tests should 8931be added after the test on which they depend. For example, since the 8932``ssh`` test depends on the ``ping`` test, "ssh" needs to come after 8933"ping" in the list. The test class provides no re-ordering or dependency 8934handling. 8935 8936.. note:: 8937 8938 Each module can have multiple classes with multiple test methods. 8939 And, Python ``unittest`` rules apply. 8940 8941Here are some things to keep in mind when running tests: 8942 8943- The default tests for the image are defined as:: 8944 8945 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" 8946 8947- Add your own test to the list of the by using the following:: 8948 8949 TEST_SUITES:append = " mytest" 8950 8951- Run a specific list of tests as follows:: 8952 8953 TEST_SUITES = "test1 test2 test3" 8954 8955 Remember, order is important. Be sure to place a test that is 8956 dependent on another test later in the order. 8957 8958Exporting Tests 8959--------------- 8960 8961You can export tests so that they can run independently of the build 8962system. Exporting tests is required if you want to be able to hand the 8963test execution off to a scheduler. You can only export tests that are 8964defined in :term:`TEST_SUITES`. 8965 8966If your image is already built, make sure the following are set in your 8967``local.conf`` file:: 8968 8969 INHERIT += "testexport" 8970 TEST_TARGET_IP = "IP-address-for-the-test-target" 8971 TEST_SERVER_IP = "IP-address-for-the-test-server" 8972 8973You can then export the tests with the 8974following BitBake command form:: 8975 8976 $ bitbake image -c testexport 8977 8978Exporting the tests places them in the 8979:term:`Build Directory` in 8980``tmp/testexport/``\ image, which is controlled by the 8981:term:`TEST_EXPORT_DIR` variable. 8982 8983You can now run the tests outside of the build environment:: 8984 8985 $ cd tmp/testexport/image 8986 $ ./runexported.py testdata.json 8987 8988Here is a complete example that shows IP addresses and uses the 8989``core-image-sato`` image:: 8990 8991 INHERIT += "testexport" 8992 TEST_TARGET_IP = "192.168.7.2" 8993 TEST_SERVER_IP = "192.168.7.1" 8994 8995Use BitBake to export the tests:: 8996 8997 $ bitbake core-image-sato -c testexport 8998 8999Run the tests outside of 9000the build environment using the following:: 9001 9002 $ cd tmp/testexport/core-image-sato 9003 $ ./runexported.py testdata.json 9004 9005Writing New Tests 9006----------------- 9007 9008As mentioned previously, all new test files need to be in the proper 9009place for the build system to find them. New tests for additional 9010functionality outside of the core should be added to the layer that adds 9011the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as 9012:term:`BBPATH` is extended in the 9013layer's ``layer.conf`` file as normal). Just remember the following: 9014 9015- Filenames need to map directly to test (module) names. 9016 9017- Do not use module names that collide with existing core tests. 9018 9019- Minimally, an empty ``__init__.py`` file must be present in the runtime 9020 directory. 9021 9022To create a new test, start by copying an existing module (e.g. 9023``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use 9024code from ``meta/lib/oeqa/utils``, which are helper classes. 9025 9026.. note:: 9027 9028 Structure shell commands such that you rely on them and they return a 9029 single code for success. Be aware that sometimes you will need to 9030 parse the output. See the ``df.py`` and ``date.py`` modules for examples. 9031 9032You will notice that all test classes inherit ``oeRuntimeTest``, which 9033is found in ``meta/lib/oetest.py``. This base class offers some helper 9034attributes, which are described in the following sections: 9035 9036Class Methods 9037~~~~~~~~~~~~~ 9038 9039Class methods are as follows: 9040 9041- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed 9042 package list of the image, which is based on the manifest file that 9043 is generated during the ``do_rootfs`` task. 9044 9045- *hasFeature(feature):* Returns "True" if the feature is in 9046 :term:`IMAGE_FEATURES` or 9047 :term:`DISTRO_FEATURES`. 9048 9049Class Attributes 9050~~~~~~~~~~~~~~~~ 9051 9052Class attributes are as follows: 9053 9054- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image. 9055 Otherwise, ``pscmd`` equals "ps" (busybox). 9056 9057- *tc:* The called test context, which gives access to the 9058 following attributes: 9059 9060 - *d:* The BitBake datastore, which allows you to use stuff such 9061 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``. 9062 9063 - *testslist and testsrequired:* Used internally. The tests 9064 do not need these. 9065 9066 - *filesdir:* The absolute path to 9067 ``meta/lib/oeqa/runtime/files``, which contains helper files for 9068 tests meant for copying on the target such as small files written 9069 in C for compilation. 9070 9071 - *target:* The target controller object used to deploy and 9072 start an image on a particular target (e.g. Qemu, SimpleRemote, 9073 and SystemdbootTarget). Tests usually use the following: 9074 9075 - *ip:* The target's IP address. 9076 9077 - *server_ip:* The host's IP address, which is usually used 9078 by the DNF test suite. 9079 9080 - *run(cmd, timeout=None):* The single, most used method. 9081 This command is a wrapper for: ``ssh root@host "cmd"``. The 9082 command returns a tuple: (status, output), which are what their 9083 names imply - the return code of "cmd" and whatever output it 9084 produces. The optional timeout argument represents the number 9085 of seconds the test should wait for "cmd" to return. If the 9086 argument is "None", the test uses the default instance's 9087 timeout period, which is 300 seconds. If the argument is "0", 9088 the test runs until the command returns. 9089 9090 - *copy_to(localpath, remotepath):* 9091 ``scp localpath root@ip:remotepath``. 9092 9093 - *copy_from(remotepath, localpath):* 9094 ``scp root@host:remotepath localpath``. 9095 9096Instance Attributes 9097~~~~~~~~~~~~~~~~~~~ 9098 9099There is a single instance attribute, which is ``target``. The ``target`` 9100instance attribute is identical to the class attribute of the same name, 9101which is described in the previous section. This attribute exists as 9102both an instance and class attribute so tests can use 9103``self.target.run(cmd)`` in instance methods instead of 9104``oeRuntimeTest.tc.target.run(cmd)``. 9105 9106Installing Packages in the DUT Without the Package Manager 9107---------------------------------------------------------- 9108 9109When a test requires a package built by BitBake, it is possible to 9110install that package. Installing the package does not require a package 9111manager be installed in the device under test (DUT). It does, however, 9112require an SSH connection and the target must be using the 9113``sshcontrol`` class. 9114 9115.. note:: 9116 9117 This method uses ``scp`` to copy files from the host to the target, which 9118 causes permissions and special attributes to be lost. 9119 9120A JSON file is used to define the packages needed by a test. This file 9121must be in the same path as the file used to define the tests. 9122Furthermore, the filename must map directly to the test module name with 9123a ``.json`` extension. 9124 9125The JSON file must include an object with the test name as keys of an 9126object or an array. This object (or array of objects) uses the following 9127data: 9128 9129- "pkg" - A mandatory string that is the name of the package to be 9130 installed. 9131 9132- "rm" - An optional boolean, which defaults to "false", that specifies 9133 to remove the package after the test. 9134 9135- "extract" - An optional boolean, which defaults to "false", that 9136 specifies if the package must be extracted from the package format. 9137 When set to "true", the package is not automatically installed into 9138 the DUT. 9139 9140Following is an example JSON file that handles test "foo" installing 9141package "bar" and test "foobar" installing packages "foo" and "bar". 9142Once the test is complete, the packages are removed from the DUT. 9143:: 9144 9145 { 9146 "foo": { 9147 "pkg": "bar" 9148 }, 9149 "foobar": [ 9150 { 9151 "pkg": "foo", 9152 "rm": true 9153 }, 9154 { 9155 "pkg": "bar", 9156 "rm": true 9157 } 9158 ] 9159 } 9160 9161Debugging Tools and Techniques 9162============================== 9163 9164The exact method for debugging build failures depends on the nature of 9165the problem and on the system's area from which the bug originates. 9166Standard debugging practices such as comparison against the last known 9167working version with examination of the changes and the re-application 9168of steps to identify the one causing the problem are valid for the Yocto 9169Project just as they are for any other system. Even though it is 9170impossible to detail every possible potential failure, this section 9171provides some general tips to aid in debugging given a variety of 9172situations. 9173 9174.. note:: 9175 9176 A useful feature for debugging is the error reporting tool. 9177 Configuring the Yocto Project to use this tool causes the 9178 OpenEmbedded build system to produce error reporting commands as part 9179 of the console output. You can enter the commands after the build 9180 completes to log error information into a common database, that can 9181 help you figure out what might be going wrong. For information on how 9182 to enable and use this feature, see the 9183 ":ref:`dev-manual/common-tasks:using the error reporting tool`" 9184 section. 9185 9186The following list shows the debugging topics in the remainder of this 9187section: 9188 9189- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes 9190 how to find and view logs from tasks that failed during the build 9191 process. 9192 9193- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to 9194 use the BitBake ``-e`` option to examine variable values after a 9195 recipe has been parsed. 9196 9197- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``" 9198 describes how to use the ``oe-pkgdata-util`` utility to query 9199 :term:`PKGDATA_DIR` and 9200 display package-related information for built packages. 9201 9202- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`" 9203 describes how to use the BitBake ``-g`` option to display recipe 9204 dependency information used during the build. 9205 9206- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes 9207 how to use the ``bitbake-dumpsig`` command in conjunction with key 9208 subdirectories in the 9209 :term:`Build Directory` to determine 9210 variable dependencies. 9211 9212- ":ref:`dev-manual/common-tasks:running specific tasks`" describes 9213 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``) 9214 to run specific tasks in the build chain. It can be useful to run 9215 tasks "out-of-order" when trying isolate build issues. 9216 9217- ":ref:`dev-manual/common-tasks:general bitbake problems`" describes how 9218 to use BitBake's ``-D`` debug output option to reveal more about what 9219 BitBake is doing during the build. 9220 9221- ":ref:`dev-manual/common-tasks:building with no dependencies`" 9222 describes how to use the BitBake ``-b`` option to build a recipe 9223 while ignoring dependencies. 9224 9225- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`" 9226 describes how to use the many recipe logging functions to produce 9227 debugging output and report errors and warnings. 9228 9229- ":ref:`dev-manual/common-tasks:debugging parallel make races`" 9230 describes how to debug situations where the build consists of several 9231 parts that are run simultaneously and when the output or result of 9232 one part is not ready for use with a different part of the build that 9233 depends on that output. 9234 9235- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`" 9236 describes how to use GDB to allow you to examine running programs, which can 9237 help you fix problems. 9238 9239- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`" 9240 describes how to use GDB directly on target hardware for debugging. 9241 9242- ":ref:`dev-manual/common-tasks:other debugging tips`" describes 9243 miscellaneous debugging tips that can be useful. 9244 9245Viewing Logs from Failed Tasks 9246------------------------------ 9247 9248You can find the log for a task in the file 9249``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`. 9250For example, the log for the 9251:ref:`ref-tasks-compile` task of the 9252QEMU minimal image for the x86 machine (``qemux86``) might be in 9253``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``. 9254To see the commands :term:`BitBake` ran 9255to generate a log, look at the corresponding ``run.do_``\ `taskname` file 9256in the same directory. 9257 9258``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic 9259links to ``log.do_``\ `taskname`\ ``.``\ `pid` and 9260``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had 9261when it ran. The symlinks always point to the files corresponding to the 9262most recent run. 9263 9264Viewing Variable Values 9265----------------------- 9266 9267Sometimes you need to know the value of a variable as a result of 9268BitBake's parsing step. This could be because some unexpected behavior 9269occurred in your project. Perhaps an attempt to :ref:`modify a variable 9270<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing 9271variables>` did not work out as expected. 9272 9273BitBake's ``-e`` option is used to display variable values after 9274parsing. The following command displays the variable values after the 9275configuration files (i.e. ``local.conf``, ``bblayers.conf``, 9276``bitbake.conf`` and so forth) have been parsed:: 9277 9278 $ bitbake -e 9279 9280The following command displays variable values after a specific recipe has 9281been parsed. The variables include those from the configuration as well:: 9282 9283 $ bitbake -e recipename 9284 9285.. note:: 9286 9287 Each recipe has its own private set of variables (datastore). 9288 Internally, after parsing the configuration, a copy of the resulting 9289 datastore is made prior to parsing each recipe. This copying implies 9290 that variables set in one recipe will not be visible to other 9291 recipes. 9292 9293 Likewise, each task within a recipe gets a private datastore based on 9294 the recipe datastore, which means that variables set within one task 9295 will not be visible to other tasks. 9296 9297In the output of ``bitbake -e``, each variable is preceded by a 9298description of how the variable got its value, including temporary 9299values that were later overridden. This description also includes 9300variable flags (varflags) set on the variable. The output can be very 9301helpful during debugging. 9302 9303Variables that are exported to the environment are preceded by 9304``export`` in the output of ``bitbake -e``. See the following example:: 9305 9306 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" 9307 9308In addition to variable values, the output of the ``bitbake -e`` and 9309``bitbake -e`` recipe commands includes the following information: 9310 9311- The output starts with a tree listing all configuration files and 9312 classes included globally, recursively listing the files they include 9313 or inherit in turn. Much of the behavior of the OpenEmbedded build 9314 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is 9315 implemented in the 9316 :ref:`base <ref-classes-base>` class and the 9317 classes it inherits, rather than being built into BitBake itself. 9318 9319- After the variable values, all functions appear in the output. For 9320 shell functions, variables referenced within the function body are 9321 expanded. If a function has been modified using overrides or using 9322 override-style operators like ``:append`` and ``:prepend``, then the 9323 final assembled function body appears in the output. 9324 9325Viewing Package Information with ``oe-pkgdata-util`` 9326---------------------------------------------------- 9327 9328You can use the ``oe-pkgdata-util`` command-line utility to query 9329:term:`PKGDATA_DIR` and display 9330various package-related information. When you use the utility, you must 9331use it to view information on packages that have already been built. 9332 9333Following are a few of the available ``oe-pkgdata-util`` subcommands. 9334 9335.. note:: 9336 9337 You can use the standard \* and ? globbing wildcards as part of 9338 package names and paths. 9339 9340- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages 9341 that have been built, optionally limiting the match to packages that 9342 match pattern. 9343 9344- ``oe-pkgdata-util list-pkg-files package ...``: Lists the 9345 files and directories contained in the given packages. 9346 9347 .. note:: 9348 9349 A different way to view the contents of a package is to look at 9350 the 9351 ``${``\ :term:`WORKDIR`\ ``}/packages-split`` 9352 directory of the recipe that generates the package. This directory 9353 is created by the 9354 :ref:`ref-tasks-package` task 9355 and has one subdirectory for each package the recipe generates, 9356 which contains the files stored in that package. 9357 9358 If you want to inspect the ``${WORKDIR}/packages-split`` 9359 directory, make sure that 9360 :ref:`rm_work <ref-classes-rm-work>` is not 9361 enabled when you build the recipe. 9362 9363- ``oe-pkgdata-util find-path path ...``: Lists the names of 9364 the packages that contain the given paths. For example, the following 9365 tells us that ``/usr/share/man/man1/make.1`` is contained in the 9366 ``make-doc`` package:: 9367 9368 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 9369 make-doc: /usr/share/man/man1/make.1 9370 9371- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name 9372 of the recipes that produce the given packages. 9373 9374For more information on the ``oe-pkgdata-util`` command, use the help 9375facility:: 9376 9377 $ oe-pkgdata-util --help 9378 $ oe-pkgdata-util subcommand --help 9379 9380Viewing Dependencies Between Recipes and Tasks 9381---------------------------------------------- 9382 9383Sometimes it can be hard to see why BitBake wants to build other recipes 9384before the one you have specified. Dependency information can help you 9385understand why a recipe is built. 9386 9387To generate dependency information for a recipe, run the following 9388command:: 9389 9390 $ bitbake -g recipename 9391 9392This command writes the following files in the current directory: 9393 9394- ``pn-buildlist``: A list of recipes/targets involved in building 9395 `recipename`. "Involved" here means that at least one task from the 9396 recipe needs to run when building `recipename` from scratch. Targets 9397 that are in 9398 :term:`ASSUME_PROVIDED` 9399 are not listed. 9400 9401- ``task-depends.dot``: A graph showing dependencies between tasks. 9402 9403The graphs are in 9404`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__ 9405format and can be converted to images (e.g. using the ``dot`` tool from 9406`Graphviz <https://www.graphviz.org/>`__). 9407 9408.. note:: 9409 9410 - DOT files use a plain text format. The graphs generated using the 9411 ``bitbake -g`` command are often so large as to be difficult to 9412 read without special pruning (e.g. with Bitbake's ``-I`` option) 9413 and processing. Despite the form and size of the graphs, the 9414 corresponding ``.dot`` files can still be possible to read and 9415 provide useful information. 9416 9417 As an example, the ``task-depends.dot`` file contains lines such 9418 as the following:: 9419 9420 "libxslt.do_configure" -> "libxml2.do_populate_sysroot" 9421 9422 The above example line reveals that the 9423 :ref:`ref-tasks-configure` 9424 task in ``libxslt`` depends on the 9425 :ref:`ref-tasks-populate_sysroot` 9426 task in ``libxml2``, which is a normal 9427 :term:`DEPENDS` dependency 9428 between the two recipes. 9429 9430 - For an example of how ``.dot`` files can be processed, see the 9431 ``scripts/contrib/graph-tool`` Python script, which finds and 9432 displays paths between graph nodes. 9433 9434You can use a different method to view dependency information by using 9435the following command:: 9436 9437 $ bitbake -g -u taskexp recipename 9438 9439This command 9440displays a GUI window from which you can view build-time and runtime 9441dependencies for the recipes involved in building recipename. 9442 9443Viewing Task Variable Dependencies 9444---------------------------------- 9445 9446As mentioned in the 9447":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake 9448User Manual, BitBake tries to automatically determine what variables a 9449task depends on so that it can rerun the task if any values of the 9450variables change. This determination is usually reliable. However, if 9451you do things like construct variable names at runtime, then you might 9452have to manually declare dependencies on those variables using 9453``vardeps`` as described in the 9454":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake 9455User Manual. 9456 9457If you are unsure whether a variable dependency is being picked up 9458automatically for a given task, you can list the variable dependencies 9459BitBake has determined by doing the following: 9460 94611. Build the recipe containing the task:: 9462 9463 $ bitbake recipename 9464 94652. Inside the :term:`STAMPS_DIR` 9466 directory, find the signature data (``sigdata``) file that 9467 corresponds to the task. The ``sigdata`` files contain a pickled 9468 Python database of all the metadata that went into creating the input 9469 checksum for the task. As an example, for the 9470 :ref:`ref-tasks-fetch` task of the 9471 ``db`` recipe, the ``sigdata`` file might be found in the following 9472 location:: 9473 9474 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 9475 9476 For tasks that are accelerated through the shared state 9477 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an 9478 additional ``siginfo`` file is written into 9479 :term:`SSTATE_DIR` along with 9480 the cached task output. The ``siginfo`` files contain exactly the 9481 same information as ``sigdata`` files. 9482 94833. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here 9484 is an example:: 9485 9486 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 9487 9488 In the output of the above command, you will find a line like the 9489 following, which lists all the (inferred) variable dependencies for 9490 the task. This list also includes indirect dependencies from 9491 variables depending on other variables, recursively. 9492 :: 9493 9494 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch'] 9495 9496 .. note:: 9497 9498 Functions (e.g. ``base_do_fetch``) also count as variable dependencies. 9499 These functions in turn depend on the variables they reference. 9500 9501 The output of ``bitbake-dumpsig`` also includes the value each 9502 variable had, a list of dependencies for each variable, and 9503 :term:`BB_BASEHASH_IGNORE_VARS` 9504 information. 9505 9506There is also a ``bitbake-diffsigs`` command for comparing two 9507``siginfo`` or ``sigdata`` files. This command can be helpful when 9508trying to figure out what changed between two versions of a task. If you 9509call ``bitbake-diffsigs`` with just one file, the command behaves like 9510``bitbake-dumpsig``. 9511 9512You can also use BitBake to dump out the signature construction 9513information without executing tasks by using either of the following 9514BitBake command-line options:: 9515 9516 ‐‐dump-signatures=SIGNATURE_HANDLER 9517 -S SIGNATURE_HANDLER 9518 9519 9520.. note:: 9521 9522 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which 9523 dump only the signature or compare the dumped signature with the cached one, 9524 respectively. 9525 9526Using BitBake with either of these options causes BitBake to dump out 9527``sigdata`` files in the ``stamps`` directory for every task it would 9528have executed instead of building the specified target package. 9529 9530Viewing Metadata Used to Create the Input Signature of a Shared State Task 9531-------------------------------------------------------------------------- 9532 9533Seeing what metadata went into creating the input signature of a shared 9534state (sstate) task can be a useful debugging aid. This information is 9535available in signature information (``siginfo``) files in 9536:term:`SSTATE_DIR`. For 9537information on how to view and interpret information in ``siginfo`` 9538files, see the 9539":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section. 9540 9541For conceptual information on shared state, see the 9542":ref:`overview-manual/concepts:shared state`" 9543section in the Yocto Project Overview and Concepts Manual. 9544 9545Invalidating Shared State to Force a Task to Run 9546------------------------------------------------ 9547 9548The OpenEmbedded build system uses 9549:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and 9550:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily 9551rebuilding tasks. Collectively, this scheme is known as "shared state 9552code". 9553 9554As with all schemes, this one has some drawbacks. It is possible that 9555you could make implicit changes to your code that the checksum 9556calculations do not take into account. These implicit changes affect a 9557task's output but do not trigger the shared state code into rebuilding a 9558recipe. Consider an example during which a tool changes its output. 9559Assume that the output of ``rpmdeps`` changes. The result of the change 9560should be that all the ``package`` and ``package_write_rpm`` shared 9561state cache items become invalid. However, because the change to the 9562output is external to the code and therefore implicit, the associated 9563shared state cache items do not become invalidated. In this case, the 9564build process uses the cached items rather than running the task again. 9565Obviously, these types of implicit changes can cause problems. 9566 9567To avoid these problems during the build, you need to understand the 9568effects of any changes you make. Realize that changes you make directly 9569to a function are automatically factored into the checksum calculation. 9570Thus, these explicit changes invalidate the associated area of shared 9571state cache. However, you need to be aware of any implicit changes that 9572are not obvious changes to the code and could affect the output of a 9573given task. 9574 9575When you identify an implicit change, you can easily take steps to 9576invalidate the cache and force the tasks to run. The steps you can take 9577are as simple as changing a function's comments in the source code. For 9578example, to invalidate package shared state files, change the comment 9579statements of 9580:ref:`ref-tasks-package` or the 9581comments of one of the functions it calls. Even though the change is 9582purely cosmetic, it causes the checksum to be recalculated and forces 9583the build system to run the task again. 9584 9585.. note:: 9586 9587 For an example of a commit that makes a cosmetic change to invalidate 9588 shared state, see this 9589 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`. 9590 9591Running Specific Tasks 9592---------------------- 9593 9594Any given recipe consists of a set of tasks. The standard BitBake 9595behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``, 9596``do_configure``, ``do_compile``, ``do_install``, ``do_package``, 9597``do_package_write_*``, and ``do_build``. The default task is 9598``do_build`` and any tasks on which it depends build first. Some tasks, 9599such as ``do_devshell``, are not part of the default build chain. If you 9600wish to run a task that is not part of the default build chain, you can 9601use the ``-c`` option in BitBake. Here is an example:: 9602 9603 $ bitbake matchbox-desktop -c devshell 9604 9605The ``-c`` option respects task dependencies, which means that all other 9606tasks (including tasks from other recipes) that the specified task 9607depends on will be run before the task. Even when you manually specify a 9608task to run with ``-c``, BitBake will only run the task if it considers 9609it "out of date". See the 9610":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`" 9611section in the Yocto Project Overview and Concepts Manual for how 9612BitBake determines whether a task is "out of date". 9613 9614If you want to force an up-to-date task to be rerun (e.g. because you 9615made manual modifications to the recipe's 9616:term:`WORKDIR` that you want to try 9617out), then you can use the ``-f`` option. 9618 9619.. note:: 9620 9621 The reason ``-f`` is never required when running the 9622 :ref:`ref-tasks-devshell` task is because the 9623 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ] 9624 variable flag is already set for the task. 9625 9626The following example shows one way you can use the ``-f`` option:: 9627 9628 $ bitbake matchbox-desktop 9629 . 9630 . 9631 make some changes to the source code in the work directory 9632 . 9633 . 9634 $ bitbake matchbox-desktop -c compile -f 9635 $ bitbake matchbox-desktop 9636 9637This sequence first builds and then recompiles ``matchbox-desktop``. The 9638last command reruns all tasks (basically the packaging tasks) after the 9639compile. BitBake recognizes that the ``do_compile`` task was rerun and 9640therefore understands that the other tasks also need to be run again. 9641 9642Another, shorter way to rerun a task and all 9643:ref:`ref-manual/tasks:normal recipe build tasks` 9644that depend on it is to use the ``-C`` option. 9645 9646.. note:: 9647 9648 This option is upper-cased and is separate from the ``-c`` 9649 option, which is lower-cased. 9650 9651Using this option invalidates the given task and then runs the 9652:ref:`ref-tasks-build` task, which is 9653the default task if no task is given, and the tasks on which it depends. 9654You could replace the final two commands in the previous example with 9655the following single command:: 9656 9657 $ bitbake matchbox-desktop -C compile 9658 9659Internally, the ``-f`` and ``-C`` options work by tainting (modifying) 9660the input checksum of the specified task. This tainting indirectly 9661causes the task and its dependent tasks to be rerun through the normal 9662task dependency mechanisms. 9663 9664.. note:: 9665 9666 BitBake explicitly keeps track of which tasks have been tainted in 9667 this fashion, and will print warnings such as the following for 9668 builds involving such tasks: 9669 9670 .. code-block:: none 9671 9672 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run 9673 9674 9675 The purpose of the warning is to let you know that the work directory 9676 and build output might not be in the clean state they would be in for 9677 a "normal" build, depending on what actions you took. To get rid of 9678 such warnings, you can remove the work directory and rebuild the 9679 recipe, as follows:: 9680 9681 $ bitbake matchbox-desktop -c clean 9682 $ bitbake matchbox-desktop 9683 9684 9685You can view a list of tasks in a given package by running the 9686``do_listtasks`` task as follows:: 9687 9688 $ bitbake matchbox-desktop -c listtasks 9689 9690The results appear as output to the console and are also in 9691the file ``${WORKDIR}/temp/log.do_listtasks``. 9692 9693General BitBake Problems 9694------------------------ 9695 9696You can see debug output from BitBake by using the ``-D`` option. The 9697debug output gives more information about what BitBake is doing and the 9698reason behind it. Each ``-D`` option you use increases the logging 9699level. The most common usage is ``-DDD``. 9700 9701The output from ``bitbake -DDD -v targetname`` can reveal why BitBake 9702chose a certain version of a package or why BitBake picked a certain 9703provider. This command could also help you in a situation where you 9704think BitBake did something unexpected. 9705 9706Building with No Dependencies 9707----------------------------- 9708 9709To build a specific recipe (``.bb`` file), you can use the following 9710command form:: 9711 9712 $ bitbake -b somepath/somerecipe.bb 9713 9714This command form does 9715not check for dependencies. Consequently, you should use it only when 9716you know existing dependencies have been met. 9717 9718.. note:: 9719 9720 You can also specify fragments of the filename. In this case, BitBake 9721 checks for a unique match. 9722 9723Recipe Logging Mechanisms 9724------------------------- 9725 9726The Yocto Project provides several logging functions for producing 9727debugging output and reporting errors and warnings. For Python 9728functions, the following logging functions are available. All of these functions 9729log to ``${T}/log.do_``\ `task`, and can also log to standard output 9730(stdout) with the right settings: 9731 9732- ``bb.plain(msg)``: Writes msg as is to the log while also 9733 logging to stdout. 9734 9735- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to 9736 stdout if BitBake is called with "-v". 9737 9738- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the 9739 log. Also logs to stdout if the log level is greater than or equal to 9740 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option 9741 in the BitBake User Manual for more information. 9742 9743- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also 9744 logging to stdout. 9745 9746- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also 9747 logging to standard out (stdout). 9748 9749 .. note:: 9750 9751 Calling this function does not cause the task to fail. 9752 9753- ``bb.fatal(msg)``: This logging function is similar to 9754 ``bb.error(msg)`` but also causes the calling task to fail. 9755 9756 .. note:: 9757 9758 ``bb.fatal()`` raises an exception, which means you do not need to put a 9759 "return" statement after the function. 9760 9761The same logging functions are also available in shell functions, under 9762the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``, 9763and ``bbfatal``. The 9764:ref:`logging <ref-classes-logging>` class 9765implements these functions. See that class in the ``meta/classes`` 9766folder of the :term:`Source Directory` for information. 9767 9768Logging With Python 9769~~~~~~~~~~~~~~~~~~~ 9770 9771When creating recipes using Python and inserting code that handles build 9772logs, keep in mind the goal is to have informative logs while keeping 9773the console as "silent" as possible. Also, if you want status messages 9774in the log, use the "debug" loglevel. 9775 9776Following is an example written in Python. The code handles logging for 9777a function that determines the number of tasks needed to be run. See the 9778":ref:`ref-tasks-listtasks`" 9779section for additional information:: 9780 9781 python do_listtasks() { 9782 bb.debug(2, "Starting to figure out the task list") 9783 if noteworthy_condition: 9784 bb.note("There are 47 tasks to run") 9785 bb.debug(2, "Got to point xyz") 9786 if warning_trigger: 9787 bb.warn("Detected warning_trigger, this might be a problem later.") 9788 if recoverable_error: 9789 bb.error("Hit recoverable_error, you really need to fix this!") 9790 if fatal_error: 9791 bb.fatal("fatal_error detected, unable to print the task list") 9792 bb.plain("The tasks present are abc") 9793 bb.debug(2, "Finished figuring out the tasklist") 9794 } 9795 9796Logging With Bash 9797~~~~~~~~~~~~~~~~~ 9798 9799When creating recipes using Bash and inserting code that handles build 9800logs, you have the same goals - informative with minimal console output. 9801The syntax you use for recipes written in Bash is similar to that of 9802recipes written in Python described in the previous section. 9803 9804Following is an example written in Bash. The code logs the progress of 9805the ``do_my_function`` function. 9806:: 9807 9808 do_my_function() { 9809 bbdebug 2 "Running do_my_function" 9810 if [ exceptional_condition ]; then 9811 bbnote "Hit exceptional_condition" 9812 fi 9813 bbdebug 2 "Got to point xyz" 9814 if [ warning_trigger ]; then 9815 bbwarn "Detected warning_trigger, this might cause a problem later." 9816 fi 9817 if [ recoverable_error ]; then 9818 bberror "Hit recoverable_error, correcting" 9819 fi 9820 if [ fatal_error ]; then 9821 bbfatal "fatal_error detected" 9822 fi 9823 bbdebug 2 "Completed do_my_function" 9824 } 9825 9826 9827Debugging Parallel Make Races 9828----------------------------- 9829 9830A parallel ``make`` race occurs when the build consists of several parts 9831that are run simultaneously and a situation occurs when the output or 9832result of one part is not ready for use with a different part of the 9833build that depends on that output. Parallel make races are annoying and 9834can sometimes be difficult to reproduce and fix. However, there are some simple 9835tips and tricks that can help you debug and fix them. This section 9836presents a real-world example of an error encountered on the Yocto 9837Project autobuilder and the process used to fix it. 9838 9839.. note:: 9840 9841 If you cannot properly fix a ``make`` race condition, you can work around it 9842 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST` 9843 variables. 9844 9845The Failure 9846~~~~~~~~~~~ 9847 9848For this example, assume that you are building an image that depends on 9849the "neard" package. And, during the build, BitBake runs into problems 9850and creates the following output. 9851 9852.. note:: 9853 9854 This example log file has longer lines artificially broken to make 9855 the listing easier to read. 9856 9857If you examine the output or the log file, you see the failure during 9858``make``: 9859 9860.. code-block:: none 9861 9862 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] 9863 | DEBUG: Executing shell function do_compile 9864 | NOTE: make -j 16 9865 | make --no-print-directory all-am 9866 | /bin/mkdir -p include/near 9867 | /bin/mkdir -p include/near 9868 | /bin/mkdir -p include/near 9869 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9870 0.14-r0/neard-0.14/include/types.h include/near/types.h 9871 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9872 0.14-r0/neard-0.14/include/log.h include/near/log.h 9873 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9874 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h 9875 | /bin/mkdir -p include/near 9876 | /bin/mkdir -p include/near 9877 | /bin/mkdir -p include/near 9878 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9879 0.14-r0/neard-0.14/include/tag.h include/near/tag.h 9880 | /bin/mkdir -p include/near 9881 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9882 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h 9883 | /bin/mkdir -p include/near 9884 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9885 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h 9886 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9887 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h 9888 | /bin/mkdir -p include/near 9889 | /bin/mkdir -p include/near 9890 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9891 0.14-r0/neard-0.14/include/setting.h include/near/setting.h 9892 | /bin/mkdir -p include/near 9893 | /bin/mkdir -p include/near 9894 | /bin/mkdir -p include/near 9895 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9896 0.14-r0/neard-0.14/include/device.h include/near/device.h 9897 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9898 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h 9899 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9900 0.14-r0/neard-0.14/include/snep.h include/near/snep.h 9901 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9902 0.14-r0/neard-0.14/include/version.h include/near/version.h 9903 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 9904 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h 9905 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h 9906 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/ 9907 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/ 9908 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0 9909 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/ 9910 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/ 9911 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/ 9912 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/ 9913 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3 9914 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\" 9915 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c 9916 -o tools/snep-send.o tools/snep-send.c 9917 | In file included from tools/snep-send.c:16:0: 9918 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory 9919 | #include <near/dbus.h> 9920 | ^ 9921 | compilation terminated. 9922 | make[1]: *** [tools/snep-send.o] Error 1 9923 | make[1]: *** Waiting for unfinished jobs.... 9924 | make: *** [all] Error 2 9925 | ERROR: oe_runmake failed 9926 9927Reproducing the Error 9928~~~~~~~~~~~~~~~~~~~~~ 9929 9930Because race conditions are intermittent, they do not manifest 9931themselves every time you do the build. In fact, most times the build 9932will complete without problems even though the potential race condition 9933exists. Thus, once the error surfaces, you need a way to reproduce it. 9934 9935In this example, compiling the "neard" package is causing the problem. 9936So the first thing to do is build "neard" locally. Before you start the 9937build, set the 9938:term:`PARALLEL_MAKE` variable 9939in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a 9940high value for :term:`PARALLEL_MAKE` increases the chances of the race 9941condition showing up:: 9942 9943 $ bitbake neard 9944 9945Once the local build for "neard" completes, start a ``devshell`` build:: 9946 9947 $ bitbake neard -c devshell 9948 9949For information on how to use a ``devshell``, see the 9950":ref:`dev-manual/common-tasks:using a development shell`" section. 9951 9952In the ``devshell``, do the following:: 9953 9954 $ make clean 9955 $ make tools/snep-send.o 9956 9957The ``devshell`` commands cause the failure to clearly 9958be visible. In this case, there is a missing dependency for the ``neard`` 9959Makefile target. Here is some abbreviated, sample output with the 9960missing dependency clearly visible at the end:: 9961 9962 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... 9963 . 9964 . 9965 . 9966 tools/snep-send.c 9967 In file included from tools/snep-send.c:16:0: 9968 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory 9969 #include <near/dbus.h> 9970 ^ 9971 compilation terminated. 9972 make: *** [tools/snep-send.o] Error 1 9973 $ 9974 9975 9976Creating a Patch for the Fix 9977~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 9978 9979Because there is a missing dependency for the Makefile target, you need 9980to patch the ``Makefile.am`` file, which is generated from 9981``Makefile.in``. You can use Quilt to create the patch:: 9982 9983 $ quilt new parallelmake.patch 9984 Patch patches/parallelmake.patch is now on top 9985 $ quilt add Makefile.am 9986 File Makefile.am added to patch patches/parallelmake.patch 9987 9988For more information on using Quilt, see the 9989":ref:`dev-manual/common-tasks:using quilt in your workflow`" section. 9990 9991At this point you need to make the edits to ``Makefile.am`` to add the 9992missing dependency. For our example, you have to add the following line 9993to the file:: 9994 9995 tools/snep-send.$(OBJEXT): include/near/dbus.h 9996 9997Once you have edited the file, use the ``refresh`` command to create the 9998patch:: 9999 10000 $ quilt refresh 10001 Refreshed patch patches/parallelmake.patch 10002 10003Once the patch file is created, you need to add it back to the originating 10004recipe folder. Here is an example assuming a top-level 10005:term:`Source Directory` named ``poky``:: 10006 10007 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard 10008 10009The final thing you need to do to implement the fix in the build is to 10010update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the 10011:term:`SRC_URI` statement includes 10012the patch file. The recipe file is in the folder above the patch. Here 10013is what the edited :term:`SRC_URI` statement would look like:: 10014 10015 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ 10016 file://neard.in \ 10017 file://neard.service.in \ 10018 file://parallelmake.patch \ 10019 " 10020 10021With the patch complete and moved to the correct folder and the 10022:term:`SRC_URI` statement updated, you can exit the ``devshell``:: 10023 10024 $ exit 10025 10026Testing the Build 10027~~~~~~~~~~~~~~~~~ 10028 10029With everything in place, you can get back to trying the build again 10030locally:: 10031 10032 $ bitbake neard 10033 10034This build should succeed. 10035 10036Now you can open up a ``devshell`` again and repeat the clean and make 10037operations as follows:: 10038 10039 $ bitbake neard -c devshell 10040 $ make clean 10041 $ make tools/snep-send.o 10042 10043The build should work without issue. 10044 10045As with all solved problems, if they originated upstream, you need to 10046submit the fix for the recipe in OE-Core and upstream so that the 10047problem is taken care of at its source. See the 10048":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 10049section for more information. 10050 10051Debugging With the GNU Project Debugger (GDB) Remotely 10052------------------------------------------------------ 10053 10054GDB allows you to examine running programs, which in turn helps you to 10055understand and fix problems. It also allows you to perform post-mortem 10056style analysis of program crashes. GDB is available as a package within 10057the Yocto Project and is installed in SDK images by default. See the 10058":ref:`ref-manual/images:Images`" chapter in the Yocto 10059Project Reference Manual for a description of these images. You can find 10060information on GDB at https://sourceware.org/gdb/. 10061 10062.. note:: 10063 10064 For best results, install debug (``-dbg``) packages for the applications you 10065 are going to debug. Doing so makes extra debug symbols available that give 10066 you more meaningful output. 10067 10068Sometimes, due to memory or disk space constraints, it is not possible 10069to use GDB directly on the remote target to debug applications. These 10070constraints arise because GDB needs to load the debugging information 10071and the binaries of the process being debugged. Additionally, GDB needs 10072to perform many computations to locate information such as function 10073names, variable names and values, stack traces and so forth - even 10074before starting the debugging process. These extra computations place 10075more load on the target system and can alter the characteristics of the 10076program being debugged. 10077 10078To help get past the previously mentioned constraints, there are two 10079methods you can use: running a debuginfod server and using gdbserver. 10080 10081Using the debuginfod server method 10082~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10083 10084``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files. 10085Running a ``debuginfod`` server makes debug symbols readily available, 10086which means you don't need to download debugging information 10087and the binaries of the process being debugged. You can just fetch 10088debug symbols from the server. 10089 10090To run a ``debuginfod`` server, you need to do the following: 10091 10092- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES` 10093 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution). 10094 If not, set in your distro config file or in ``local.conf``:: 10095 10096 DISTRO_FEATURES:append = " debuginfod" 10097 10098 This distro feature enables the server and client library in ``elfutils``, 10099 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``). 10100 10101- Run the following commands to launch the ``debuginfod`` server on the host:: 10102 10103 $ oe-debuginfod 10104 10105- To use ``debuginfod`` on the target, you need to know the ip:port where 10106 ``debuginfod`` is listening on the host (port defaults to 8002), and export 10107 that into the shell environment, for example in ``qemu``:: 10108 10109 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/" 10110 10111- Then debug info fetching should simply work when running the target ``gdb``, 10112 ``readelf`` or ``objdump``, for example:: 10113 10114 root@qemux86-64:~# gdb /bin/cat 10115 ... 10116 Reading symbols from /bin/cat... 10117 Downloading separate debug info for /bin/cat... 10118 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo... 10119 10120- It's also possible to use ``debuginfod-find`` to just query the server:: 10121 10122 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls 10123 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo 10124 10125 10126Using the gdbserver method 10127~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10128 10129gdbserver, which runs on the remote target and does not load any 10130debugging information from the debugged process. Instead, a GDB instance 10131processes the debugging information that is run on a remote computer - 10132the host GDB. The host GDB then sends control commands to gdbserver to 10133make it stop or start the debugged program, as well as read or write 10134memory regions of that debugged program. All the debugging information 10135loaded and processed as well as all the heavy debugging is done by the 10136host GDB. Offloading these processes gives the gdbserver running on the 10137target a chance to remain small and fast. 10138 10139Because the host GDB is responsible for loading the debugging 10140information and for doing the necessary processing to make actual 10141debugging happen, you have to make sure the host can access the 10142unstripped binaries complete with their debugging information and also 10143be sure the target is compiled with no optimizations. The host GDB must 10144also have local access to all the libraries used by the debugged 10145program. Because gdbserver does not need any local debugging 10146information, the binaries on the remote target can remain stripped. 10147However, the binaries must also be compiled without optimization so they 10148match the host's binaries. 10149 10150To remain consistent with GDB documentation and terminology, the binary 10151being debugged on the remote target machine is referred to as the 10152"inferior" binary. For documentation on GDB see the `GDB 10153site <https://sourceware.org/gdb/documentation/>`__. 10154 10155The following steps show you how to debug using the GNU project 10156debugger. 10157 101581. *Configure your build system to construct the companion debug 10159 filesystem:* 10160 10161 In your ``local.conf`` file, set the following:: 10162 10163 IMAGE_GEN_DEBUGFS = "1" 10164 IMAGE_FSTYPES_DEBUGFS = "tar.bz2" 10165 10166 These options cause the 10167 OpenEmbedded build system to generate a special companion filesystem 10168 fragment, which contains the matching source and debug symbols to 10169 your deployable filesystem. The build system does this by looking at 10170 what is in the deployed filesystem, and pulling the corresponding 10171 ``-dbg`` packages. 10172 10173 The companion debug filesystem is not a complete filesystem, but only 10174 contains the debug fragments. This filesystem must be combined with 10175 the full filesystem for debugging. Subsequent steps in this procedure 10176 show how to combine the partial filesystem with the full filesystem. 10177 101782. *Configure the system to include gdbserver in the target filesystem:* 10179 10180 Make the following addition in either your ``local.conf`` file or in 10181 an image recipe:: 10182 10183 IMAGE_INSTALL:append = " gdbserver" 10184 10185 The change makes 10186 sure the ``gdbserver`` package is included. 10187 101883. *Build the environment:* 10189 10190 Use the following command to construct the image and the companion 10191 Debug Filesystem:: 10192 10193 $ bitbake image 10194 10195 Build the cross GDB component and 10196 make it available for debugging. Build the SDK that matches the 10197 image. Building the SDK is best for a production build that can be 10198 used later for debugging, especially during long term maintenance:: 10199 10200 $ bitbake -c populate_sdk image 10201 10202 Alternatively, you can build the minimal toolchain components that 10203 match the target. Doing so creates a smaller than typical SDK and 10204 only contains a minimal set of components with which to build simple 10205 test applications, as well as run the debugger:: 10206 10207 $ bitbake meta-toolchain 10208 10209 A final method is to build Gdb itself within the build system:: 10210 10211 $ bitbake gdb-cross-<architecture> 10212 10213 Doing so produces a temporary copy of 10214 ``cross-gdb`` you can use for debugging during development. While 10215 this is the quickest approach, the two previous methods in this step 10216 are better when considering long-term maintenance strategies. 10217 10218 .. note:: 10219 10220 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests 10221 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the 10222 actual name you want to use. 10223 102244. *Set up the* ``debugfs``\ *:* 10225 10226 Run the following commands to set up the ``debugfs``:: 10227 10228 $ mkdir debugfs 10229 $ cd debugfs 10230 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2 10231 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2 10232 102335. *Set up GDB:* 10234 10235 Install the SDK (if you built one) and then source the correct 10236 environment file. Sourcing the environment file puts the SDK in your 10237 ``PATH`` environment variable. 10238 10239 If you are using the build system, Gdb is located in 10240 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb`` 10241 102426. *Boot the target:* 10243 10244 For information on how to run QEMU, see the `QEMU 10245 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__. 10246 10247 .. note:: 10248 10249 Be sure to verify that your host can access the target via TCP. 10250 102517. *Debug a program:* 10252 10253 Debugging a program involves running gdbserver on the target and then 10254 running Gdb on the host. The example in this step debugs ``gzip``: 10255 10256 .. code-block:: shell 10257 10258 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help 10259 10260 For 10261 additional gdbserver options, see the `GDB Server 10262 Documentation <https://www.gnu.org/software/gdb/documentation/>`__. 10263 10264 After running gdbserver on the target, you need to run Gdb on the 10265 host and configure it and connect to the target. Use these commands:: 10266 10267 $ cd directory-holding-the-debugfs-directory 10268 $ arch-gdb 10269 (gdb) set sysroot debugfs 10270 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug 10271 (gdb) target remote IP-of-target:1234 10272 10273 At this 10274 point, everything should automatically load (i.e. matching binaries, 10275 symbols and headers). 10276 10277 .. note:: 10278 10279 The Gdb ``set`` commands in the previous example can be placed into the 10280 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever 10281 commands are in that file. 10282 102838. *Deploying without a full image rebuild:* 10284 10285 In many cases, during development you want a quick method to deploy a 10286 new binary to the target and debug it, without waiting for a full 10287 image build. 10288 10289 One approach to solving this situation is to just build the component 10290 you want to debug. Once you have built the component, copy the 10291 executable directly to both the target and the host ``debugfs``. 10292 10293 If the binary is processed through the debug splitting in 10294 OpenEmbedded, you should also copy the debug items (i.e. ``.debug`` 10295 contents and corresponding ``/usr/src/debug`` files) from the work 10296 directory. Here is an example:: 10297 10298 $ bitbake bash 10299 $ bitbake -c devshell bash 10300 $ cd .. 10301 $ scp packages-split/bash/bin/bash target:/bin/bash 10302 $ cp -a packages-split/bash-dbg/\* path/debugfs 10303 10304Debugging with the GNU Project Debugger (GDB) on the Target 10305----------------------------------------------------------- 10306 10307The previous section addressed using GDB remotely for debugging 10308purposes, which is the most usual case due to the inherent hardware 10309limitations on many embedded devices. However, debugging in the target 10310hardware itself is also possible with more powerful devices. This 10311section describes what you need to do in order to support using GDB to 10312debug on the target hardware. 10313 10314To support this kind of debugging, you need do the following: 10315 10316- Ensure that GDB is on the target. You can do this by adding "gdb" to 10317 :term:`IMAGE_INSTALL`:: 10318 10319 IMAGE_INSTALL:append = " gdb" 10320 10321 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`:: 10322 10323 IMAGE_FEATURES:append = " tools-debug" 10324 10325- Ensure that debug symbols are present. You can make sure these 10326 symbols are present by installing ``-dbg``:: 10327 10328 IMAGE_INSTALL:append = "packagename-dbg" 10329 10330 Alternatively, you can do the following to include 10331 all the debug symbols:: 10332 10333 IMAGE_FEATURES:append = " dbg-pkgs" 10334 10335.. note:: 10336 10337 To improve the debug information accuracy, you can reduce the level 10338 of optimization used by the compiler. For example, when adding the 10339 following line to your ``local.conf`` file, you will reduce optimization 10340 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION` 10341 of "-O -fno-omit-frame-pointer":: 10342 10343 DEBUG_BUILD = "1" 10344 10345 Consider that this will reduce the application's performance and is 10346 recommended only for debugging purposes. 10347 10348Other Debugging Tips 10349-------------------- 10350 10351Here are some other tips that you might find useful: 10352 10353- When adding new packages, it is worth watching for undesirable items 10354 making their way into compiler command lines. For example, you do not 10355 want references to local system files like ``/usr/lib/`` or 10356 ``/usr/include/``. 10357 10358- If you want to remove the ``psplash`` boot splashscreen, add 10359 ``psplash=false`` to the kernel command line. Doing so prevents 10360 ``psplash`` from loading and thus allows you to see the console. It 10361 is also possible to switch out of the splashscreen by switching the 10362 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus). 10363 10364- Removing :term:`TMPDIR` (usually 10365 ``tmp/``, within the 10366 :term:`Build Directory`) can often fix 10367 temporary build issues. Removing :term:`TMPDIR` is usually a relatively 10368 cheap operation, because task output will be cached in 10369 :term:`SSTATE_DIR` (usually 10370 ``sstate-cache/``, which is also in the Build Directory). 10371 10372 .. note:: 10373 10374 Removing :term:`TMPDIR` might be a workaround rather than a fix. 10375 Consequently, trying to determine the underlying cause of an issue before 10376 removing the directory is a good idea. 10377 10378- Understanding how a feature is used in practice within existing 10379 recipes can be very helpful. It is recommended that you configure 10380 some method that allows you to quickly search through files. 10381 10382 Using GNU Grep, you can use the following shell function to 10383 recursively search through common recipe-related files, skipping 10384 binary files, ``.git`` directories, and the Build Directory (assuming 10385 its name starts with "build"):: 10386 10387 g() { 10388 grep -Ir \ 10389 --exclude-dir=.git \ 10390 --exclude-dir='build*' \ 10391 --include='*.bb*' \ 10392 --include='*.inc*' \ 10393 --include='*.conf*' \ 10394 --include='*.py*' \ 10395 "$@" 10396 } 10397 10398 Following are some usage examples:: 10399 10400 $ g FOO # Search recursively for "FOO" 10401 $ g -i foo # Search recursively for "foo", ignoring case 10402 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" 10403 10404 If figuring 10405 out how some feature works requires a lot of searching, it might 10406 indicate that the documentation should be extended or improved. In 10407 such cases, consider filing a documentation bug using the Yocto 10408 Project implementation of 10409 :yocto_bugs:`Bugzilla <>`. For information on 10410 how to submit a bug against the Yocto Project, see the Yocto Project 10411 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>` 10412 and the 10413 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`" 10414 section. 10415 10416 .. note:: 10417 10418 The manuals might not be the right place to document variables 10419 that are purely internal and have a limited scope (e.g. internal 10420 variables used to implement a single ``.bbclass`` file). 10421 10422Making Changes to the Yocto Project 10423=================================== 10424 10425Because the Yocto Project is an open-source, community-based project, 10426you can effect changes to the project. This section presents procedures 10427that show you how to submit a defect against the project and how to 10428submit a change. 10429 10430Submitting a Defect Against the Yocto Project 10431--------------------------------------------- 10432 10433Use the Yocto Project implementation of 10434`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug) 10435against the Yocto Project. For additional information on this 10436implementation of Bugzilla see the ":ref:`Yocto Project 10437Bugzilla <resources-bugtracker>`" section in the 10438Yocto Project Reference Manual. For more detail on any of the following 10439steps, see the Yocto Project 10440:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`. 10441 10442Use the following general steps to submit a bug: 10443 104441. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`. 10445 104462. Click "File a Bug" to enter a new bug. 10447 104483. Choose the appropriate "Classification", "Product", and "Component" 10449 for which the bug was found. Bugs for the Yocto Project fall into 10450 one of several classifications, which in turn break down into 10451 several products and components. For example, for a bug against the 10452 ``meta-intel`` layer, you would choose "Build System, Metadata & 10453 Runtime", "BSPs", and "bsps-meta-intel", respectively. 10454 104554. Choose the "Version" of the Yocto Project for which you found the 10456 bug (e.g. &DISTRO;). 10457 104585. Determine and select the "Severity" of the bug. The severity 10459 indicates how the bug impacted your work. 10460 104616. Choose the "Hardware" that the bug impacts. 10462 104637. Choose the "Architecture" that the bug impacts. 10464 104658. Choose a "Documentation change" item for the bug. Fixing a bug might 10466 or might not affect the Yocto Project documentation. If you are 10467 unsure of the impact to the documentation, select "Don't Know". 10468 104699. Provide a brief "Summary" of the bug. Try to limit your summary to 10470 just a line or two and be sure to capture the essence of the bug. 10471 1047210. Provide a detailed "Description" of the bug. You should provide as 10473 much detail as you can about the context, behavior, output, and so 10474 forth that surrounds the bug. You can even attach supporting files 10475 for output from logs by using the "Add an attachment" button. 10476 1047711. Click the "Submit Bug" button submit the bug. A new Bugzilla number 10478 is assigned to the bug and the defect is logged in the bug tracking 10479 system. 10480 10481Once you file a bug, the bug is processed by the Yocto Project Bug 10482Triage Team and further details concerning the bug are assigned (e.g. 10483priority and owner). You are the "Submitter" of the bug and any further 10484categorization, progress, or comments on the bug result in Bugzilla 10485sending you an automated email concerning the particular change or 10486progress to the bug. 10487 10488Submitting a Change to the Yocto Project 10489---------------------------------------- 10490 10491Contributions to the Yocto Project and OpenEmbedded are very welcome. 10492Because the system is extremely configurable and flexible, we recognize 10493that developers will want to extend, configure or optimize it for their 10494specific uses. 10495 10496The Yocto Project uses a mailing list and a patch-based workflow that is 10497similar to the Linux kernel but contains important differences. In 10498general, there is a mailing list through which you can submit patches. You 10499should send patches to the appropriate mailing list so that they can be 10500reviewed and merged by the appropriate maintainer. The specific mailing 10501list you need to use depends on the location of the code you are 10502changing. Each component (e.g. layer) should have a ``README`` file that 10503indicates where to send the changes and which process to follow. 10504 10505You can send the patch to the mailing list using whichever approach you 10506feel comfortable with to generate the patch. Once sent, the patch is 10507usually reviewed by the community at large. If somebody has concerns 10508with the patch, they will usually voice their concern over the mailing 10509list. If a patch does not receive any negative reviews, the maintainer 10510of the affected layer typically takes the patch, tests it, and then 10511based on successful testing, merges the patch. 10512 10513The "poky" repository, which is the Yocto Project's reference build 10514environment, is a hybrid repository that contains several individual 10515pieces (e.g. BitBake, Metadata, documentation, and so forth) built using 10516the combo-layer tool. The upstream location used for submitting changes 10517varies by component: 10518 10519- *Core Metadata:* Send your patch to the 10520 :oe_lists:`openembedded-core </g/openembedded-core>` 10521 mailing list. For example, a change to anything under the ``meta`` or 10522 ``scripts`` directories should be sent to this mailing list. 10523 10524- *BitBake:* For changes to BitBake (i.e. anything under the 10525 ``bitbake`` directory), send your patch to the 10526 :oe_lists:`bitbake-devel </g/bitbake-devel>` 10527 mailing list. 10528 10529- *"meta-\*" trees:* These trees contain Metadata. Use the 10530 :yocto_lists:`poky </g/poky>` mailing list. 10531 10532- *Documentation*: For changes to the Yocto Project documentation, use the 10533 :yocto_lists:`docs </g/docs>` mailing list. 10534 10535For changes to other layers hosted in the Yocto Project source 10536repositories (i.e. ``yoctoproject.org``) and tools use the 10537:yocto_lists:`Yocto Project </g/yocto/>` general mailing list. 10538 10539.. note:: 10540 10541 Sometimes a layer's documentation specifies to use a particular 10542 mailing list. If so, use that list. 10543 10544For additional recipes that do not fit into the core Metadata, you 10545should determine which layer the recipe should go into and submit the 10546change in the manner recommended by the documentation (e.g. the 10547``README`` file) supplied with the layer. If in doubt, please ask on the 10548Yocto general mailing list or on the openembedded-devel mailing list. 10549 10550You can also push a change upstream and request a maintainer to pull the 10551change into the component's upstream repository. You do this by pushing 10552to a contribution repository that is upstream. See the 10553":ref:`overview-manual/development-environment:git workflows and the yocto project`" 10554section in the Yocto Project Overview and Concepts Manual for additional 10555concepts on working in the Yocto Project development environment. 10556 10557Maintainers commonly use ``-next`` branches to test submissions prior to 10558merging patches. Thus, you can get an idea of the status of a patch based on 10559whether the patch has been merged into one of these branches. The commonly 10560used testing branches for OpenEmbedded-Core are as follows: 10561 10562- *openembedded-core "master-next" branch:* This branch is part of the 10563 :oe_git:`openembedded-core </openembedded-core/>` repository and contains 10564 proposed changes to the core metadata. 10565 10566- *poky "master-next" branch:* This branch is part of the 10567 :yocto_git:`poky </poky/>` repository and combines proposed 10568 changes to bitbake, the core metadata and the poky distro. 10569 10570Similarly, stable branches maintained by the project may have corresponding 10571``-next`` branches which collect proposed changes. For example, 10572``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next`` 10573branches in both the "openembdedded-core" and "poky" repositories. 10574 10575Other layers may have similar testing branches but there is no formal 10576requirement or standard for these so please check the documentation for the 10577layers you are contributing to. 10578 10579The following sections provide procedures for submitting a change. 10580 10581Preparing Changes for Submission 10582~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10583 105841. *Make Your Changes Locally:* Make your changes in your local Git 10585 repository. You should make small, controlled, isolated changes. 10586 Keeping changes small and isolated aids review, makes 10587 merging/rebasing easier and keeps the change history clean should 10588 anyone need to refer to it in future. 10589 105902. *Stage Your Changes:* Stage your changes by using the ``git add`` 10591 command on each file you changed. 10592 105933. *Commit Your Changes:* Commit the change by using the ``git commit`` 10594 command. Make sure your commit information follows standards by 10595 following these accepted conventions: 10596 10597 - Be sure to include a "Signed-off-by:" line in the same style as 10598 required by the Linux kernel. This can be done by using the 10599 ``git commit -s`` command. Adding this line signifies that you, 10600 the submitter, have agreed to the Developer's Certificate of 10601 Origin 1.1 as follows: 10602 10603 .. code-block:: none 10604 10605 Developer's Certificate of Origin 1.1 10606 10607 By making a contribution to this project, I certify that: 10608 10609 (a) The contribution was created in whole or in part by me and I 10610 have the right to submit it under the open source license 10611 indicated in the file; or 10612 10613 (b) The contribution is based upon previous work that, to the best 10614 of my knowledge, is covered under an appropriate open source 10615 license and I have the right under that license to submit that 10616 work with modifications, whether created in whole or in part 10617 by me, under the same open source license (unless I am 10618 permitted to submit under a different license), as indicated 10619 in the file; or 10620 10621 (c) The contribution was provided directly to me by some other 10622 person who certified (a), (b) or (c) and I have not modified 10623 it. 10624 10625 (d) I understand and agree that this project and the contribution 10626 are public and that a record of the contribution (including all 10627 personal information I submit with it, including my sign-off) is 10628 maintained indefinitely and may be redistributed consistent with 10629 this project or the open source license(s) involved. 10630 10631 - Provide a single-line summary of the change and, if more 10632 explanation is needed, provide more detail in the body of the 10633 commit. This summary is typically viewable in the "shortlist" of 10634 changes. Thus, providing something short and descriptive that 10635 gives the reader a summary of the change is useful when viewing a 10636 list of many commits. You should prefix this short description 10637 with the recipe name (if changing a recipe), or else with the 10638 short form path to the file being changed. 10639 10640 - For the body of the commit message, provide detailed information 10641 that describes what you changed, why you made the change, and the 10642 approach you used. It might also be helpful if you mention how you 10643 tested the change. Provide as much detail as you can in the body 10644 of the commit message. 10645 10646 .. note:: 10647 10648 You do not need to provide a more detailed explanation of a 10649 change if the change is minor to the point of the single line 10650 summary providing all the information. 10651 10652 - If the change addresses a specific bug or issue that is associated 10653 with a bug-tracking ID, include a reference to that ID in your 10654 detailed description. For example, the Yocto Project uses a 10655 specific convention for bug references - any commit that addresses 10656 a specific bug should use the following form for the detailed 10657 description. Be sure to use the actual bug-tracking ID from 10658 Bugzilla for bug-id:: 10659 10660 Fixes [YOCTO #bug-id] 10661 10662 detailed description of change 10663 10664Using Email to Submit a Patch 10665~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10666 10667Depending on the components changed, you need to submit the email to a 10668specific mailing list. For some guidance on which mailing list to use, 10669see the 10670:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>` 10671at the beginning of this section. For a description of all the available 10672mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the 10673Yocto Project Reference Manual. 10674 10675Here is the general procedure on how to submit a patch through email 10676without using the scripts once the steps in 10677:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed: 10678 106791. *Format the Commit:* Format the commit into an email message. To 10680 format commits, use the ``git format-patch`` command. When you 10681 provide the command, you must include a revision list or a number of 10682 patches as part of the command. For example, either of these two 10683 commands takes your most recent single commit and formats it as an 10684 email message in the current directory:: 10685 10686 $ git format-patch -1 10687 10688 or :: 10689 10690 $ git format-patch HEAD~ 10691 10692 After the command is run, the current directory contains a numbered 10693 ``.patch`` file for the commit. 10694 10695 If you provide several commits as part of the command, the 10696 ``git format-patch`` command produces a series of numbered files in 10697 the current directory – one for each commit. If you have more than 10698 one patch, you should also use the ``--cover`` option with the 10699 command, which generates a cover letter as the first "patch" in the 10700 series. You can then edit the cover letter to provide a description 10701 for the series of patches. For information on the 10702 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed 10703 using the ``man git-format-patch`` command. 10704 10705 .. note:: 10706 10707 If you are or will be a frequent contributor to the Yocto Project 10708 or to OpenEmbedded, you might consider requesting a contrib area 10709 and the necessary associated rights. 10710 107112. *Send the patches via email:* Send the patches to the recipients and 10712 relevant mailing lists by using the ``git send-email`` command. 10713 10714 .. note:: 10715 10716 In order to use ``git send-email``, you must have the proper Git packages 10717 installed on your host. 10718 For Ubuntu, Debian, and Fedora the package is ``git-email``. 10719 10720 The ``git send-email`` command sends email by using a local or remote 10721 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or 10722 through a direct ``smtp`` configuration in your Git ``~/.gitconfig`` 10723 file. If you are submitting patches through email only, it is very 10724 important that you submit them without any whitespace or HTML 10725 formatting that either you or your mailer introduces. The maintainer 10726 that receives your patches needs to be able to save and apply them 10727 directly from your emails. A good way to verify that what you are 10728 sending will be applicable by the maintainer is to do a dry run and 10729 send them to yourself and then save and apply them as the maintainer 10730 would. 10731 10732 The ``git send-email`` command is the preferred method for sending 10733 your patches using email since there is no risk of compromising 10734 whitespace in the body of the message, which can occur when you use 10735 your own mail client. The command also has several options that let 10736 you specify recipients and perform further editing of the email 10737 message. For information on how to use the ``git send-email`` 10738 command, see ``GIT-SEND-EMAIL(1)`` displayed using the 10739 ``man git-send-email`` command. 10740 10741The Yocto Project uses a `Patchwork instance <https://patchwork.yoctoproject.org/>`__ 10742to track the status of patches submitted to the various mailing lists and to 10743support automated patch testing. Each submitted patch is checked for common 10744mistakes and deviations from the expected patch format and submitters are 10745notified by patchtest if such mistakes are found. This process helps to 10746reduce the burden of patch review on maintainers. 10747 10748.. note:: 10749 10750 This system is imperfect and changes can sometimes get lost in the flow. 10751 Asking about the status of a patch or change is reasonable if the change 10752 has been idle for a while with no feedback. 10753 10754Using Scripts to Push a Change Upstream and Request a Pull 10755~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10756 10757For larger patch series it is preferable to send a pull request which not 10758only includes the patch but also a pointer to a branch that can be pulled 10759from. This involves making a local branch for your changes, pushing this 10760branch to an accessible repository and then using the ``create-pull-request`` 10761and ``send-pull-request`` scripts from openembedded-core to create and send a 10762patch series with a link to the branch for review. 10763 10764Follow this procedure to push a change to an upstream "contrib" Git 10765repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have 10766been followed: 10767 10768.. note:: 10769 10770 You can find general Git information on how to push a change upstream 10771 in the 10772 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__. 10773 107741. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for 10775 permissions to push to an upstream contrib repository, push the 10776 change to that repository:: 10777 10778 $ git push upstream_remote_repo local_branch_name 10779 10780 For example, suppose you have permissions to push 10781 into the upstream ``meta-intel-contrib`` repository and you are 10782 working in a local branch named `your_name`\ ``/README``. The following 10783 command pushes your local commits to the ``meta-intel-contrib`` 10784 upstream repository and puts the commit in a branch named 10785 `your_name`\ ``/README``:: 10786 10787 $ git push meta-intel-contrib your_name/README 10788 107892. *Determine Who to Notify:* Determine the maintainer or the mailing 10790 list that you need to notify for the change. 10791 10792 Before submitting any change, you need to be sure who the maintainer 10793 is or what mailing list that you need to notify. Use either these 10794 methods to find out: 10795 10796 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is 10797 located in the :term:`Source Directory` at 10798 ``meta/conf/distro/include``, to see who is responsible for code. 10799 10800 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can 10801 enter the following command to bring up a short list of all 10802 commits against a specific file:: 10803 10804 git shortlog -- filename 10805 10806 Just provide the name of the file for which you are interested. The 10807 information returned is not ordered by history but does include a 10808 list of everyone who has committed grouped by name. From the list, 10809 you can see who is responsible for the bulk of the changes against 10810 the file. 10811 10812 - *Examine the List of Mailing Lists:* For a list of the Yocto 10813 Project and related mailing lists, see the ":ref:`Mailing 10814 lists <resources-mailinglist>`" section in 10815 the Yocto Project Reference Manual. 10816 108173. *Make a Pull Request:* Notify the maintainer or the mailing list that 10818 you have pushed a change by making a pull request. 10819 10820 The Yocto Project provides two scripts that conveniently let you 10821 generate and send pull requests to the Yocto Project. These scripts 10822 are ``create-pull-request`` and ``send-pull-request``. You can find 10823 these scripts in the ``scripts`` directory within the 10824 :term:`Source Directory` (e.g. 10825 ``poky/scripts``). 10826 10827 Using these scripts correctly formats the requests without 10828 introducing any whitespace or HTML formatting. The maintainer that 10829 receives your patches either directly or through the mailing list 10830 needs to be able to save and apply them directly from your emails. 10831 Using these scripts is the preferred method for sending patches. 10832 10833 First, create the pull request. For example, the following command 10834 runs the script, specifies the upstream repository in the contrib 10835 directory into which you pushed the change, and provides a subject 10836 line in the created patch files:: 10837 10838 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README" 10839 10840 Running this script forms ``*.patch`` files in a folder named 10841 ``pull-``\ `PID` in the current directory. One of the patch files is a 10842 cover letter. 10843 10844 Before running the ``send-pull-request`` script, you must edit the 10845 cover letter patch to insert information about your change. After 10846 editing the cover letter, send the pull request. For example, the 10847 following command runs the script and specifies the patch directory 10848 and email address. In this example, the email address is a mailing 10849 list:: 10850 10851 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org 10852 10853 You need to follow the prompts as the script is interactive. 10854 10855 .. note:: 10856 10857 For help on using these scripts, simply provide the ``-h`` 10858 argument as follows:: 10859 10860 $ poky/scripts/create-pull-request -h 10861 $ poky/scripts/send-pull-request -h 10862 10863Responding to Patch Review 10864~~~~~~~~~~~~~~~~~~~~~~~~~~ 10865 10866You may get feedback on your submitted patches from other community members 10867or from the automated patchtest service. If issues are identified in your 10868patch then it is usually necessary to address these before the patch will be 10869accepted into the project. In this case you should amend the patch according 10870to the feedback and submit an updated version to the relevant mailing list, 10871copying in the reviewers who provided feedback to the previous version of the 10872patch. 10873 10874The patch should be amended using ``git commit --amend`` or perhaps ``git 10875rebase`` for more expert git users. You should also modify the ``[PATCH]`` 10876tag in the email subject line when sending the revised patch to mark the new 10877iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be 10878done by passing the ``-v`` argument to ``git format-patch`` with a version 10879number. 10880 10881Lastly please ensure that you also test your revised changes. In particular 10882please don't just edit the patch file written out by ``git format-patch`` and 10883resend it. 10884 10885Submitting Changes to Stable Release Branches 10886~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10887 10888The process for proposing changes to a Yocto Project stable branch differs 10889from the steps described above. Changes to a stable branch must address 10890identified bugs or CVEs and should be made carefully in order to avoid the 10891risk of introducing new bugs or breaking backwards compatibility. Typically 10892bug fixes must already be accepted into the master branch before they can be 10893backported to a stable branch unless the bug in question does not affect the 10894master branch or the fix on the master branch is unsuitable for backporting. 10895 10896The list of stable branches along with the status and maintainer for each 10897branch can be obtained from the 10898:yocto_wiki:`Releases wiki page </Releases>`. 10899 10900.. note:: 10901 10902 Changes will not typically be accepted for branches which are marked as 10903 End-Of-Life (EOL). 10904 10905With this in mind, the steps to submit a change for a stable branch are as 10906follows: 10907 109081. *Identify the bug or CVE to be fixed:* This information should be 10909 collected so that it can be included in your submission. 10910 10911 See :ref:`dev-manual/common-tasks:checking for vulnerabilities` 10912 for details about CVE tracking. 10913 109142. *Check if the fix is already present in the master branch:* This will 10915 result in the most straightforward path into the stable branch for the 10916 fix. 10917 10918 a. *If the fix is present in the master branch - Submit a backport request 10919 by email:* You should send an email to the relevant stable branch 10920 maintainer and the mailing list with details of the bug or CVE to be 10921 fixed, the commit hash on the master branch that fixes the issue and 10922 the stable branches which you would like this fix to be backported to. 10923 10924 b. *If the fix is not present in the master branch - Submit the fix to the 10925 master branch first:* This will ensure that the fix passes through the 10926 project's usual patch review and test processes before being accepted. 10927 It will also ensure that bugs are not left unresolved in the master 10928 branch itself. Once the fix is accepted in the master branch a backport 10929 request can be submitted as above. 10930 10931 c. *If the fix is unsuitable for the master branch - Submit a patch 10932 directly for the stable branch:* This method should be considered as a 10933 last resort. It is typically necessary when the master branch is using 10934 a newer version of the software which includes an upstream fix for the 10935 issue or when the issue has been fixed on the master branch in a way 10936 that introduces backwards incompatible changes. In this case follow the 10937 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and 10938 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch 10939 email to include the name of the stable branch which you are 10940 targetting. This can be done using the ``--subject-prefix`` argument to 10941 ``git format-patch``, for example to submit a patch to the dunfell 10942 branch use 10943 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``. 10944 10945Working With Licenses 10946===================== 10947 10948As mentioned in the ":ref:`overview-manual/development-environment:licensing`" 10949section in the Yocto Project Overview and Concepts Manual, open source 10950projects are open to the public and they consequently have different 10951licensing structures in place. This section describes the mechanism by 10952which the :term:`OpenEmbedded Build System` 10953tracks changes to 10954licensing text and covers how to maintain open source license compliance 10955during your project's lifecycle. The section also describes how to 10956enable commercially licensed recipes, which by default are disabled. 10957 10958Tracking License Changes 10959------------------------ 10960 10961The license of an upstream project might change in the future. In order 10962to prevent these changes going unnoticed, the 10963:term:`LIC_FILES_CHKSUM` 10964variable tracks changes to the license text. The checksums are validated 10965at the end of the configure step, and if the checksums do not match, the 10966build will fail. 10967 10968Specifying the ``LIC_FILES_CHKSUM`` Variable 10969~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10970 10971The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text 10972in the source code for the recipe. Following is an example of how to 10973specify :term:`LIC_FILES_CHKSUM`:: 10974 10975 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \ 10976 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \ 10977 file://licfile2.txt;endline=50;md5=zzzz \ 10978 ..." 10979 10980.. note:: 10981 10982 - When using "beginline" and "endline", realize that line numbering 10983 begins with one and not zero. Also, the included lines are 10984 inclusive (i.e. lines five through and including 29 in the 10985 previous example for ``licfile1.txt``). 10986 10987 - When a license check fails, the selected license text is included 10988 as part of the QA message. Using this output, you can determine 10989 the exact start and finish for the needed license text. 10990 10991The build system uses the :term:`S` 10992variable as the default directory when searching files listed in 10993:term:`LIC_FILES_CHKSUM`. The previous example employs the default 10994directory. 10995 10996Consider this next example:: 10997 10998 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\ 10999 md5=bb14ed3c4cda583abc85401304b5cd4e" 11000 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6" 11001 11002The first line locates a file in ``${S}/src/ls.c`` and isolates lines 11003five through 16 as license text. The second line refers to a file in 11004:term:`WORKDIR`. 11005 11006Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes, 11007unless the :term:`LICENSE` variable is set to "CLOSED". 11008 11009Explanation of Syntax 11010~~~~~~~~~~~~~~~~~~~~~ 11011 11012As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable 11013lists all the important files that contain the license text for the 11014source code. It is possible to specify a checksum for an entire file, or 11015a specific section of a file (specified by beginning and ending line 11016numbers with the "beginline" and "endline" parameters, respectively). 11017The latter is useful for source files with a license notice header, 11018README documents, and so forth. If you do not use the "beginline" 11019parameter, then it is assumed that the text begins on the first line of 11020the file. Similarly, if you do not use the "endline" parameter, it is 11021assumed that the license text ends with the last line of the file. 11022 11023The "md5" parameter stores the md5 checksum of the license text. If the 11024license text changes in any way as compared to this parameter then a 11025mismatch occurs. This mismatch triggers a build failure and notifies the 11026developer. Notification allows the developer to review and address the 11027license text changes. Also note that if a mismatch occurs during the 11028build, the correct md5 checksum is placed in the build log and can be 11029easily copied to the recipe. 11030 11031There is no limit to how many files you can specify using the 11032:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project 11033requires a few specifications for license tracking. Many projects have a 11034"COPYING" file that stores the license information for all the source 11035code files. This practice allows you to just track the "COPYING" file as 11036long as it is kept up to date. 11037 11038.. note:: 11039 11040 - If you specify an empty or invalid "md5" parameter, 11041 :term:`BitBake` returns an md5 11042 mis-match error and displays the correct "md5" parameter value 11043 during the build. The correct parameter is also captured in the 11044 build log. 11045 11046 - If the whole file contains only license text, you do not need to 11047 use the "beginline" and "endline" parameters. 11048 11049Enabling Commercially Licensed Recipes 11050-------------------------------------- 11051 11052By default, the OpenEmbedded build system disables components that have 11053commercial or other special licensing requirements. Such requirements 11054are defined on a recipe-by-recipe basis through the 11055:term:`LICENSE_FLAGS` variable 11056definition in the affected recipe. For instance, the 11057``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe 11058contains the following statement:: 11059 11060 LICENSE_FLAGS = "commercial" 11061 11062Here is a 11063slightly more complicated example that contains both an explicit recipe 11064name and version (after variable expansion):: 11065 11066 LICENSE_FLAGS = "license_${PN}_${PV}" 11067 11068In order for a component restricted by a 11069:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it 11070needs to have a matching entry in the global 11071:term:`LICENSE_FLAGS_ACCEPTED` 11072variable, which is a variable typically defined in your ``local.conf`` 11073file. For example, to enable the 11074``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you 11075could add either the string "commercial_gst-plugins-ugly" or the more 11076general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the 11077":ref:`dev-manual/common-tasks:license flag matching`" section for a full 11078explanation of how :term:`LICENSE_FLAGS` matching works. Here is the 11079example:: 11080 11081 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly" 11082 11083Likewise, to additionally enable the package built from the recipe 11084containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that 11085the actual recipe name was ``emgd_1.10.bb``, the following string would 11086enable that package as well as the original ``gst-plugins-ugly`` 11087package:: 11088 11089 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10" 11090 11091As a convenience, you do not need to specify the 11092complete license string for every package. You can use 11093an abbreviated form, which consists of just the first portion or 11094portions of the license string before the initial underscore character 11095or characters. A partial string will match any license that contains the 11096given string as the first portion of its license. For example, the 11097following value will also match both of the packages 11098previously mentioned as well as any other packages that have licenses 11099starting with "commercial" or "license". 11100:: 11101 11102 LICENSE_FLAGS_ACCEPTED = "commercial license" 11103 11104License Flag Matching 11105~~~~~~~~~~~~~~~~~~~~~ 11106 11107License flag matching allows you to control what recipes the 11108OpenEmbedded build system includes in the build. Fundamentally, the 11109build system attempts to match :term:`LICENSE_FLAGS` strings found in 11110recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`. 11111A match causes the build system to include a recipe in the 11112build, while failure to find a match causes the build system to exclude 11113a recipe. 11114 11115In general, license flag matching is simple. However, understanding some 11116concepts will help you correctly and effectively use matching. 11117 11118Before a flag defined by a particular recipe is tested against the 11119entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded 11120string ``_${PN}`` is appended to the flag. This expansion makes each 11121:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the 11122string is then matched against the entries. Thus, specifying 11123``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results 11124in the string ``"commercial_foo"``. And, to create a match, that string 11125must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`. 11126 11127Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the 11128:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for 11129including or excluding recipes based on licensing. For example, you can 11130broaden the matching capabilities by using license flags string subsets 11131in :term:`LICENSE_FLAGS_ACCEPTED`. 11132 11133.. note:: 11134 11135 When using a string subset, be sure to use the part of the expanded 11136 string that precedes the appended underscore character (e.g. 11137 ``usethispart_1.3``, ``usethispart_1.4``, and so forth). 11138 11139For example, simply specifying the string "commercial" in the 11140:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded 11141:term:`LICENSE_FLAGS` definition that starts with the string 11142"commercial" such as "commercial_foo" and "commercial_bar", which 11143are the strings the build system automatically generates for 11144hypothetical recipes named "foo" and "bar" assuming those recipes simply 11145specify the following:: 11146 11147 LICENSE_FLAGS = "commercial" 11148 11149Thus, you can choose to exhaustively enumerate each license flag in the 11150list and allow only specific recipes into the image, or you can use a 11151string subset that causes a broader range of matches to allow a range of 11152recipes into the image. 11153 11154This scheme works even if the :term:`LICENSE_FLAGS` string already has 11155``_${PN}`` appended. For example, the build system turns the license 11156flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match 11157both the general "commercial" and the specific "commercial_1.2_foo" 11158strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected. 11159 11160Here are some other scenarios: 11161 11162- You can specify a versioned string in the recipe such as 11163 "commercial_foo_1.2" in a "foo" recipe. The build system expands this 11164 string to "commercial_foo_1.2_foo". Combine this license flag with a 11165 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string 11166 "commercial" and you match the flag along with any other flag that 11167 starts with the string "commercial". 11168 11169- Under the same circumstances, you can add "commercial_foo" in the 11170 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only 11171 matches "commercial_foo_1.2" but also matches any license flag with 11172 the string "commercial_foo", regardless of the version. 11173 11174- You can be very specific and use both the package and version parts 11175 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g. 11176 "commercial_foo_1.2") to specifically match a versioned recipe. 11177 11178Other Variables Related to Commercial Licenses 11179~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11180 11181There are other helpful variables related to commercial license handling, 11182defined in the 11183``poky/meta/conf/distro/include/default-distrovars.inc`` file:: 11184 11185 COMMERCIAL_AUDIO_PLUGINS ?= "" 11186 COMMERCIAL_VIDEO_PLUGINS ?= "" 11187 11188If you 11189want to enable these components, you can do so by making sure you have 11190statements similar to the following in your ``local.conf`` configuration 11191file:: 11192 11193 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \ 11194 gst-plugins-ugly-mpegaudioparse" 11195 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \ 11196 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse" 11197 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp" 11198 11199 11200Of course, you could also create a matching list for those 11201components using the more general "commercial" in the 11202:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all 11203the other packages with :term:`LICENSE_FLAGS` 11204containing "commercial", which you may or may not want:: 11205 11206 LICENSE_FLAGS_ACCEPTED = "commercial" 11207 11208Specifying audio and video plugins as part of the 11209``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements 11210(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the 11211plugins or components into built images, thus adding support for media 11212formats or components. 11213 11214Maintaining Open Source License Compliance During Your Product's Lifecycle 11215-------------------------------------------------------------------------- 11216 11217One of the concerns for a development organization using open source 11218software is how to maintain compliance with various open source 11219licensing during the lifecycle of the product. While this section does 11220not provide legal advice or comprehensively cover all scenarios, it does 11221present methods that you can use to assist you in meeting the compliance 11222requirements during a software release. 11223 11224With hundreds of different open source licenses that the Yocto Project 11225tracks, it is difficult to know the requirements of each and every 11226license. However, the requirements of the major FLOSS licenses can begin 11227to be covered by assuming that there are three main areas of concern: 11228 11229- Source code must be provided. 11230 11231- License text for the software must be provided. 11232 11233- Compilation scripts and modifications to the source code must be 11234 provided. 11235 11236- spdx files can be provided. 11237 11238There are other requirements beyond the scope of these three and the 11239methods described in this section (e.g. the mechanism through which 11240source code is distributed). 11241 11242As different organizations have different methods of complying with open 11243source licensing, this section is not meant to imply that there is only 11244one single way to meet your compliance obligations, but rather to 11245describe one method of achieving compliance. The remainder of this 11246section describes methods supported to meet the previously mentioned 11247three requirements. Once you take steps to meet these requirements, and 11248prior to releasing images, sources, and the build system, you should 11249audit all artifacts to ensure completeness. 11250 11251.. note:: 11252 11253 The Yocto Project generates a license manifest during image creation 11254 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp` 11255 to assist with any audits. 11256 11257Providing the Source Code 11258~~~~~~~~~~~~~~~~~~~~~~~~~ 11259 11260Compliance activities should begin before you generate the final image. 11261The first thing you should look at is the requirement that tops the list 11262for most compliance groups - providing the source. The Yocto Project has 11263a few ways of meeting this requirement. 11264 11265One of the easiest ways to meet this requirement is to provide the 11266entire :term:`DL_DIR` used by the 11267build. This method, however, has a few issues. The most obvious is the 11268size of the directory since it includes all sources used in the build 11269and not just the source used in the released image. It will include 11270toolchain source, and other artifacts, which you would not generally 11271release. However, the more serious issue for most companies is 11272accidental release of proprietary software. The Yocto Project provides 11273an :ref:`archiver <ref-classes-archiver>` class to 11274help avoid some of these concerns. 11275 11276Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to 11277decide how you choose to provide source. The source ``archiver`` class 11278can generate tarballs and SRPMs and can create them with various levels 11279of compliance in mind. 11280 11281One way of doing this (but certainly not the only way) is to release 11282just the source as a tarball. You can do this by adding the following to 11283the ``local.conf`` file found in the 11284:term:`Build Directory`:: 11285 11286 INHERIT += "archiver" 11287 ARCHIVER_MODE[src] = "original" 11288 11289During the creation of your 11290image, the source from all recipes that deploy packages to the image is 11291placed within subdirectories of ``DEPLOY_DIR/sources`` based on the 11292:term:`LICENSE` for each recipe. 11293Releasing the entire directory enables you to comply with requirements 11294concerning providing the unmodified source. It is important to note that 11295the size of the directory can get large. 11296 11297A way to help mitigate the size issue is to only release tarballs for 11298licenses that require the release of source. Let us assume you are only 11299concerned with GPL code as identified by running the following script: 11300 11301.. code-block:: shell 11302 11303 # Script to archive a subset of packages matching specific license(s) 11304 # Source and license files are copied into sub folders of package folder 11305 # Must be run from build folder 11306 #!/bin/bash 11307 src_release_dir="source-release" 11308 mkdir -p $src_release_dir 11309 for a in tmp/deploy/sources/*; do 11310 for d in $a/*; do 11311 # Get package name from path 11312 p=`basename $d` 11313 p=${p%-*} 11314 p=${p%-*} 11315 # Only archive GPL packages (update *GPL* regex for your license check) 11316 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l` 11317 if [ $numfiles -ge 1 ]; then 11318 echo Archiving $p 11319 mkdir -p $src_release_dir/$p/source 11320 cp $d/* $src_release_dir/$p/source 2> /dev/null 11321 mkdir -p $src_release_dir/$p/license 11322 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null 11323 fi 11324 done 11325 done 11326 11327At this point, you 11328could create a tarball from the ``gpl_source_release`` directory and 11329provide that to the end user. This method would be a step toward 11330achieving compliance with section 3a of GPLv2 and with section 6 of 11331GPLv3. 11332 11333Providing License Text 11334~~~~~~~~~~~~~~~~~~~~~~ 11335 11336One requirement that is often overlooked is inclusion of license text. 11337This requirement also needs to be dealt with prior to generating the 11338final image. Some licenses require the license text to accompany the 11339binary. You can achieve this by adding the following to your 11340``local.conf`` file:: 11341 11342 COPY_LIC_MANIFEST = "1" 11343 COPY_LIC_DIRS = "1" 11344 LICENSE_CREATE_PACKAGE = "1" 11345 11346Adding these statements to the 11347configuration file ensures that the licenses collected during package 11348generation are included on your image. 11349 11350.. note:: 11351 11352 Setting all three variables to "1" results in the image having two 11353 copies of the same license file. One copy resides in 11354 ``/usr/share/common-licenses`` and the other resides in 11355 ``/usr/share/license``. 11356 11357 The reason for this behavior is because 11358 :term:`COPY_LIC_DIRS` and 11359 :term:`COPY_LIC_MANIFEST` 11360 add a copy of the license when the image is built but do not offer a 11361 path for adding licenses for newly installed packages to an image. 11362 :term:`LICENSE_CREATE_PACKAGE` 11363 adds a separate package and an upgrade path for adding licenses to an 11364 image. 11365 11366As the source ``archiver`` class has already archived the original 11367unmodified source that contains the license files, you would have 11368already met the requirements for inclusion of the license information 11369with source as defined by the GPL and other open source licenses. 11370 11371Providing Compilation Scripts and Source Code Modifications 11372~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11373 11374At this point, we have addressed all we need to prior to generating the 11375image. The next two requirements are addressed during the final 11376packaging of the release. 11377 11378By releasing the version of the OpenEmbedded build system and the layers 11379used during the build, you will be providing both compilation scripts 11380and the source code modifications in one step. 11381 11382If the deployment team has a :ref:`overview-manual/concepts:bsp layer` 11383and a distro layer, and those 11384those layers are used to patch, compile, package, or modify (in any way) 11385any open source software included in your released images, you might be 11386required to release those layers under section 3 of GPLv2 or section 1 11387of GPLv3. One way of doing that is with a clean checkout of the version 11388of the Yocto Project and layers used during your build. Here is an 11389example: 11390 11391.. code-block:: shell 11392 11393 # We built using the dunfell branch of the poky repo 11394 $ git clone -b dunfell git://git.yoctoproject.org/poky 11395 $ cd poky 11396 # We built using the release_branch for our layers 11397 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer 11398 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer 11399 # clean up the .git repos 11400 $ find . -name ".git" -type d -exec rm -rf {} \; 11401 11402One 11403thing a development organization might want to consider for end-user 11404convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to 11405ensure that when the end user utilizes the released build system to 11406build an image, the development organization's layers are included in 11407the ``bblayers.conf`` file automatically:: 11408 11409 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf 11410 # changes incompatibly 11411 POKY_BBLAYERS_CONF_VERSION = "2" 11412 11413 BBPATH = "${TOPDIR}" 11414 BBFILES ?= "" 11415 11416 BBLAYERS ?= " \ 11417 ##OEROOT##/meta \ 11418 ##OEROOT##/meta-poky \ 11419 ##OEROOT##/meta-yocto-bsp \ 11420 ##OEROOT##/meta-mylayer \ 11421 " 11422 11423Creating and 11424providing an archive of the :term:`Metadata` 11425layers (recipes, configuration files, and so forth) enables you to meet 11426your requirements to include the scripts to control compilation as well 11427as any modifications to the original source. 11428 11429Compliance Limitations with Executables Built from Static Libraries 11430~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11431 11432When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS` 11433mechanisms as well as explicitly included in the image recipe with 11434:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B 11435(``DEPENDS += "B"``), package B will neither appear in the generated license 11436manifest nor in the generated source tarballs. This occurs as the 11437:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>` 11438classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS` 11439end up in the image. 11440 11441As a result, potential obligations regarding license compliance for package B 11442may not be met. 11443 11444The Yocto Project doesn't enable static libraries by default, in part because 11445of this issue. Before a solution to this limitation is found, you need to 11446keep in mind that if your root filesystem is built from static libraries, 11447you will need to manually ensure that your deliveries are compliant 11448with the licenses of these libraries. 11449 11450Copying Non Standard Licenses 11451----------------------------- 11452 11453Some packages, such as the linux-firmware package, have many licenses 11454that are not in any way common. You can avoid adding a lot of these 11455types of common license files, which are only applicable to a specific 11456package, by using the 11457:term:`NO_GENERIC_LICENSE` 11458variable. Using this variable also avoids QA errors when you use a 11459non-common, non-CLOSED license in a recipe. 11460 11461Here is an example that uses the ``LICENSE.Abilis.txt`` file as 11462the license from the fetched source:: 11463 11464 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt" 11465 11466Checking for Vulnerabilities 11467============================ 11468 11469Vulnerabilities in Poky and OE-Core 11470----------------------------------- 11471 11472The Yocto Project has an infrastructure to track and address unfixed 11473known security vulnerabilities, as tracked by the public 11474:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>` 11475database. 11476 11477The Yocto Project maintains a `list of known vulnerabilities 11478<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__ 11479for packages in Poky and OE-Core, tracking the evolution of the number of 11480unpatched CVEs and the status of patches. Such information is available for 11481the current development version and for each supported release. 11482 11483Security is a process, not a product, and thus at any time, a number of security 11484issues may be impacting Poky and OE-Core. It is up to the maintainers, users, 11485contributors and anyone interested in the issues to investigate and possibly fix them by 11486updating software components to newer versions or by applying patches to address them. 11487It is recommended to work with Poky and OE-Core upstream maintainers and submit 11488patches to fix them, see ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" for details. 11489 11490Vulnerability check at build time 11491--------------------------------- 11492 11493To enable a check for CVE security vulnerabilities using :ref:`cve-check <ref-classes-cve-check>` in the specific image 11494or target you are building, add the following setting to your configuration:: 11495 11496 INHERIT += "cve-check" 11497 11498The CVE database contains some old incomplete entries which have been 11499deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the 11500check using build configuration:: 11501 11502 include conf/distro/include/cve-extra-exclusions.inc 11503 11504With this CVE check enabled, BitBake build will try to map each compiled software component 11505recipe name and version information to the CVE database and generate recipe and 11506image specific reports. These reports will contain: 11507 11508- metadata about the software component like names and versions 11509 11510- metadata about the CVE issue such as description and NVD link 11511 11512- for each software component, a list of CVEs which are possibly impacting this version 11513 11514- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored`` 11515 11516The status ``Patched`` means that a patch file to address the security issue has been 11517applied. ``Unpatched`` status means that no patches to address the issue have been 11518applied and that the issue needs to be investigated. ``Ignored`` means that after 11519analysis, it has been deemed to ignore the issue as it for example affects 11520the software component on a different operating system platform. 11521 11522After a build with CVE check enabled, reports for each compiled source recipe will be 11523found in ``build/tmp/deploy/cve``. 11524 11525For example the CVE check report for the ``flex-native`` recipe looks like:: 11526 11527 $ cat poky/build/tmp/deploy/cve/flex-native 11528 LAYER: meta 11529 PACKAGE NAME: flex-native 11530 PACKAGE VERSION: 2.6.4 11531 CVE: CVE-2016-6354 11532 CVE STATUS: Patched 11533 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read. 11534 CVSS v2 BASE SCORE: 7.5 11535 CVSS v3 BASE SCORE: 9.8 11536 VECTOR: NETWORK 11537 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354 11538 11539 LAYER: meta 11540 PACKAGE NAME: flex-native 11541 PACKAGE VERSION: 2.6.4 11542 CVE: CVE-2019-6293 11543 CVE STATUS: Ignored 11544 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service. 11545 CVSS v2 BASE SCORE: 4.3 11546 CVSS v3 BASE SCORE: 5.5 11547 VECTOR: NETWORK 11548 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293 11549 11550For images, a summary of all recipes included in the image and their CVEs is also 11551generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found 11552in the ``tmp/deploy/images`` directory for each compiled image. 11553 11554At build time CVE check will also throw warnings about ``Unpatched`` CVEs:: 11555 11556 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log 11557 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log 11558 11559It is also possible to check the CVE status of individual packages as follows:: 11560 11561 bitbake -c cve_check flex libarchive 11562 11563Fixing CVE product name and version mappings 11564-------------------------------------------- 11565 11566By default, :ref:`cve-check <ref-classes-cve-check>` uses the recipe name :term:`BPN` as CVE 11567product name when querying the CVE database. If this mapping contains false positives, e.g. 11568some reported CVEs are not for the software component in question, or false negatives like 11569some CVEs are not found to impact the recipe when they should, then the problems can be 11570in the recipe name to CVE product mapping. These mapping issues can be fixed by setting 11571the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the 11572upstream `NIST CVE database <https://nvd.nist.gov/>`__. 11573 11574The variable supports using vendor and product names like this:: 11575 11576 CVE_PRODUCT = "flex_project:flex" 11577 11578In this example the vendor name used in the CVE database is ``flex_project`` and the 11579product is ``flex``. With this setting the ``flex`` recipe only maps to this specific 11580product and not products from other vendors with same name ``flex``. 11581 11582Similarly, when the recipe version :term:`PV` is not compatible with software versions used by 11583the upstream software component releases and the CVE database, these can be fixed using 11584the :term:`CVE_VERSION` variable. 11585 11586Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete 11587information, it is recommended to fix the information there directly instead of working 11588around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to 11589NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__. 11590 11591Fixing vulnerabilities in recipes 11592--------------------------------- 11593 11594If a CVE security issue impacts a software component, it can be fixed by updating to a newer 11595version of the software component or by applying a patch. For Poky and OE-Core master branches, updating 11596to a newer software component release with fixes is the best option, but patches can be applied 11597if releases are not yet available. 11598 11599For stable branches, it is preferred to apply patches for the issues. For some software 11600components minor version updates can also be applied if they are backwards compatible. 11601 11602Here is an example of fixing CVE security issues with patch files, 11603an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`:: 11604 11605 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \ 11606 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \ 11607 file://fix-CVE-2020-20446.patch \ 11608 file://fix-CVE-2020-20453.patch \ 11609 file://fix-CVE-2020-22015.patch \ 11610 file://fix-CVE-2020-22021.patch \ 11611 file://fix-CVE-2020-22033-CVE-2020-22019.patch \ 11612 file://fix-CVE-2021-33815.patch \ 11613 11614A good practice is to include the CVE identifier in both the patch file name 11615and inside the patch file commit message using the format:: 11616 11617 CVE: CVE-2020-22033 11618 11619CVE checker will then capture this information and change the CVE status to ``Patched`` 11620in the generated reports. 11621 11622If analysis shows that the CVE issue does not impact the recipe due to configuration, platform, 11623version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable. 11624As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those 11625issues in the CVE database directly. 11626 11627Recipes can be completely skipped by CVE check by including the recipe name in 11628the :term:`CVE_CHECK_SKIP_RECIPE` variable. 11629 11630Implementation details 11631---------------------- 11632 11633Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to 11634find unpatched CVE IDs. 11635 11636First the code goes through each patch file provided by a recipe. If a valid CVE ID 11637is found in the name of the file, the corresponding CVE is considered as patched. 11638Don't forget that if multiple CVE IDs are found in the filename, only the last 11639one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch 11640file. The found CVE IDs are also considered as patched. 11641 11642Then, the code looks up all the CVE IDs in the NIST database for all the 11643products defined in :term:`CVE_PRODUCT`. Then, for each found CVE: 11644 11645- If the package name (:term:`PN`) is part of 11646 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``. 11647 11648- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is 11649 set as ``Ignored``. 11650 11651- If the CVE ID is part of the patched CVE for the recipe, it is 11652 already considered as ``Patched``. 11653 11654- Otherwise, the code checks whether the recipe version (:term:`PV`) 11655 is within the range of versions impacted by the CVE. If so, the CVE 11656 is considered as ``Unpatched``. 11657 11658The CVE database is stored in :term:`DL_DIR` and can be inspected using 11659``sqlite3`` command as follows:: 11660 11661 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462 11662 11663When analyzing CVEs, it is recommended to: 11664 11665- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__. 11666 11667- check how upstream developers of the software component addressed the issue, e.g. 11668 what patch was applied, which upstream release contains the fix. 11669 11670- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__ 11671 did to analyze and address the issue. 11672 11673- follow security notices from other Linux distributions. 11674 11675- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for 11676 discussions and advance notifications of CVE bugs and software releases with fixes. 11677 11678Creating a Software Bill of Materials 11679===================================== 11680 11681Once you are able to build an image for your project, once the licenses for 11682each software component are all identified (see 11683":ref:`dev-manual/common-tasks:working with licenses`") and once vulnerability 11684fixes are applied (see ":ref:`dev-manual/common-tasks:checking 11685for vulnerabilities`"), the OpenEmbedded build system can generate 11686a description of all the components you used, their licenses, their dependencies, 11687the changes that were applied and the known vulnerabilities that were fixed. 11688 11689This description is generated in the form of a *Software Bill of Materials* 11690(:term:`SBOM`), using the :term:`SPDX` standard. 11691 11692When you release software, this is the most standard way to provide information 11693about the Software Supply Chain of your software image and SDK. The 11694:term:`SBOM` tooling is often used to ensure open source license compliance by 11695providing the license texts used in the product which legal departments and end 11696users can read in standardized format. 11697 11698:term:`SBOM` information is also critical to performing vulnerability exposure 11699assessments, as all the components used in the Software Supply Chain are listed. 11700 11701The OpenEmbedded build system doesn't generate such information by default. 11702To make this happen, you must inherit the 11703:ref:`create-spdx <ref-classes-create-spdx>` class from a configuration file:: 11704 11705 INHERIT += "create-spdx" 11706 11707You then get :term:`SPDX` output in JSON format as an 11708``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the 11709:term:`Build Directory`. 11710 11711This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json`` 11712containing an index of JSON :term:`SPDX` files for individual recipes, together 11713with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such 11714files. 11715 11716The :ref:`ref-classes-create-spdx` class offers options to include 11717more information in the output :term:`SPDX` data, such as making the generated 11718files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of 11719the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`), 11720adding a description of the source files used to generate host tools and target 11721packages (:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source 11722files themselves (:term:`SPDX_ARCHIVE_SOURCES`). 11723 11724Though the toplevel :term:`SPDX` output is available in 11725``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary 11726generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as: 11727 11728- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst`` 11729 archive. 11730 11731- Compressed archives of the files in the generated target packages, 11732 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED` 11733 is set). 11734 11735- Compressed archives of the source files used to build the host tools 11736 and the target packages in ``recipes/recipe-packagename.tar.zst`` 11737 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill 11738 "source code access" license requirements. 11739 11740See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX` 11741project website for a list of tools to consume and transform the :term:`SPDX` 11742data generated by the OpenEmbedded build system. 11743 11744See also Joshua Watt's 11745`Automated SBoM generation with OpenEmbedded and the Yocto Project <https://youtu.be/Q5UQUM6zxVU>`__ 11746presentation at FOSDEM 2023. 11747 11748 11749Using the Error Reporting Tool 11750============================== 11751 11752The error reporting tool allows you to submit errors encountered during 11753builds to a central database. Outside of the build environment, you can 11754use a web interface to browse errors, view statistics, and query for 11755errors. The tool works using a client-server system where the client 11756portion is integrated with the installed Yocto Project 11757:term:`Source Directory` (e.g. ``poky``). 11758The server receives the information collected and saves it in a 11759database. 11760 11761There is a live instance of the error reporting server at 11762https://errors.yoctoproject.org. 11763When you want to get help with build failures, you can submit all of the 11764information on the failure easily and then point to the URL in your bug 11765report or send an email to the mailing list. 11766 11767.. note:: 11768 11769 If you send error reports to this server, the reports become publicly 11770 visible. 11771 11772Enabling and Using the Tool 11773--------------------------- 11774 11775By default, the error reporting tool is disabled. You can enable it by 11776inheriting the 11777:ref:`report-error <ref-classes-report-error>` 11778class by adding the following statement to the end of your 11779``local.conf`` file in your 11780:term:`Build Directory`. 11781:: 11782 11783 INHERIT += "report-error" 11784 11785By default, the error reporting feature stores information in 11786``${``\ :term:`LOG_DIR`\ ``}/error-report``. 11787However, you can specify a directory to use by adding the following to 11788your ``local.conf`` file:: 11789 11790 ERR_REPORT_DIR = "path" 11791 11792Enabling error 11793reporting causes the build process to collect the errors and store them 11794in a file as previously described. When the build system encounters an 11795error, it includes a command as part of the console output. You can run 11796the command to send the error file to the server. For example, the 11797following command sends the errors to an upstream server:: 11798 11799 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt 11800 11801In the previous example, the errors are sent to a public database 11802available at https://errors.yoctoproject.org, which is used by the 11803entire community. If you specify a particular server, you can send the 11804errors to a different database. Use the following command for more 11805information on available options:: 11806 11807 $ send-error-report --help 11808 11809When sending the error file, you are prompted to review the data being 11810sent as well as to provide a name and optional email address. Once you 11811satisfy these prompts, the command returns a link from the server that 11812corresponds to your entry in the database. For example, here is a 11813typical link: https://errors.yoctoproject.org/Errors/Details/9522/ 11814 11815Following the link takes you to a web interface where you can browse, 11816query the errors, and view statistics. 11817 11818Disabling the Tool 11819------------------ 11820 11821To disable the error reporting feature, simply remove or comment out the 11822following statement from the end of your ``local.conf`` file in your 11823:term:`Build Directory`. 11824:: 11825 11826 INHERIT += "report-error" 11827 11828Setting Up Your Own Error Reporting Server 11829------------------------------------------ 11830 11831If you want to set up your own error reporting server, you can obtain 11832the code from the Git repository at :yocto_git:`/error-report-web/`. 11833Instructions on how to set it up are in the README document. 11834 11835Using Wayland and Weston 11836======================== 11837 11838`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__ 11839is a computer display server protocol that provides a method for 11840compositing window managers to communicate directly with applications 11841and video hardware and expects them to communicate with input hardware 11842using other libraries. Using Wayland with supporting targets can result 11843in better control over graphics frame rendering than an application 11844might otherwise achieve. 11845 11846The Yocto Project provides the Wayland protocol libraries and the 11847reference 11848`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__ 11849compositor as part of its release. You can find the integrated packages 11850in the ``meta`` layer of the :term:`Source Directory`. 11851Specifically, you 11852can find the recipes that build both Wayland and Weston at 11853``meta/recipes-graphics/wayland``. 11854 11855You can build both the Wayland and Weston packages for use only with 11856targets that accept the `Mesa 3D and Direct Rendering 11857Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__, 11858which is also known as Mesa DRI. This implies that you cannot build and 11859use the packages if your target uses, for example, the Intel Embedded 11860Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI. 11861 11862.. note:: 11863 11864 Due to lack of EGL support, Weston 1.0.3 will not run directly on the 11865 emulated QEMU hardware. However, this version of Weston will run 11866 under X emulation without issues. 11867 11868This section describes what you need to do to implement Wayland and use 11869the Weston compositor when building an image for a supporting target. 11870 11871Enabling Wayland in an Image 11872---------------------------- 11873 11874To enable Wayland, you need to enable it to be built and enable it to be 11875included (installed) in the image. 11876 11877Building Wayland 11878~~~~~~~~~~~~~~~~ 11879 11880To cause Mesa to build the ``wayland-egl`` platform and Weston to build 11881Wayland with Kernel Mode Setting 11882(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__) 11883support, include the "wayland" flag in the 11884:term:`DISTRO_FEATURES` 11885statement in your ``local.conf`` file:: 11886 11887 DISTRO_FEATURES:append = " wayland" 11888 11889.. note:: 11890 11891 If X11 has been enabled elsewhere, Weston will build Wayland with X11 11892 support 11893 11894Installing Wayland and Weston 11895~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11896 11897To install the Wayland feature into an image, you must include the 11898following 11899:term:`CORE_IMAGE_EXTRA_INSTALL` 11900statement in your ``local.conf`` file:: 11901 11902 CORE_IMAGE_EXTRA_INSTALL += "wayland weston" 11903 11904Running Weston 11905-------------- 11906 11907To run Weston inside X11, enabling it as described earlier and building 11908a Sato image is sufficient. If you are running your image under Sato, a 11909Weston Launcher appears in the "Utility" category. 11910 11911Alternatively, you can run Weston through the command-line interpretor 11912(CLI), which is better suited for development work. To run Weston under 11913the CLI, you need to do the following after your image is built: 11914 119151. Run these commands to export ``XDG_RUNTIME_DIR``:: 11916 11917 mkdir -p /tmp/$USER-weston 11918 chmod 0700 /tmp/$USER-weston 11919 export XDG_RUNTIME_DIR=/tmp/$USER-weston 11920 119212. Launch Weston in the shell:: 11922 11923 weston 11924
