1.. SPDX-License-Identifier: CC-BY-2.5 2 3===================== 4File Download Support 5===================== 6 7| 8 9BitBake's fetch module is a standalone piece of library code that deals 10with the intricacies of downloading source code and files from remote 11systems. Fetching source code is one of the cornerstones of building 12software. As such, this module forms an important part of BitBake. 13 14The current fetch module is called "fetch2" and refers to the fact that 15it is the second major version of the API. The original version is 16obsolete and has been removed from the codebase. Thus, in all cases, 17"fetch" refers to "fetch2" in this manual. 18 19The Download (Fetch) 20==================== 21 22BitBake takes several steps when fetching source code or files. The 23fetcher codebase deals with two distinct processes in order: obtaining 24the files from somewhere (cached or otherwise) and then unpacking those 25files into a specific location and perhaps in a specific way. Getting 26and unpacking the files is often optionally followed by patching. 27Patching, however, is not covered by this module. 28 29The code to execute the first part of this process, a fetch, looks 30something like the following:: 31 32 src_uri = (d.getVar('SRC_URI') or "").split() 33 fetcher = bb.fetch2.Fetch(src_uri, d) 34 fetcher.download() 35 36This code sets up an instance of the fetch class. The instance uses a 37space-separated list of URLs from the :term:`SRC_URI` 38variable and then calls the ``download`` method to download the files. 39 40The instantiation of the fetch class is usually followed by:: 41 42 rootdir = l.getVar('WORKDIR') 43 fetcher.unpack(rootdir) 44 45This code unpacks the downloaded files to the specified by ``WORKDIR``. 46 47.. note:: 48 49 For convenience, the naming in these examples matches the variables 50 used by OpenEmbedded. If you want to see the above code in action, 51 examine the OpenEmbedded class file ``base.bbclass`` 52 . 53 54The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the 55fetcher, since those fetcher methods can be (and are) called with 56different variable names. In OpenEmbedded for example, the shared state 57(sstate) code uses the fetch module to fetch the sstate files. 58 59When the ``download()`` method is called, BitBake tries to resolve the 60URLs by looking for source files in a specific search order: 61 62- *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find 63 source files. These locations are defined using the 64 :term:`PREMIRRORS` variable. 65 66- *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g 67 from :term:`SRC_URI`). 68 69- *Mirror Sites:* If fetch failures occur, BitBake next uses mirror 70 locations as defined by the :term:`MIRRORS` variable. 71 72For each URL passed to the fetcher, the fetcher calls the submodule that 73handles that particular URL type. This behavior can be the source of 74some confusion when you are providing URLs for the :term:`SRC_URI` variable. 75Consider the following two URLs:: 76 77 https://git.yoctoproject.org/git/poky;protocol=git 78 git://git.yoctoproject.org/git/poky;protocol=http 79 80In the former case, the URL is passed to the ``wget`` fetcher, which does not 81understand "git". Therefore, the latter case is the correct form since the Git 82fetcher does know how to use HTTP as a transport. 83 84Here are some examples that show commonly used mirror definitions:: 85 86 PREMIRRORS ?= "\ 87 bzr://.*/.\* http://somemirror.org/sources/ \ 88 cvs://.*/.\* http://somemirror.org/sources/ \ 89 git://.*/.\* http://somemirror.org/sources/ \ 90 hg://.*/.\* http://somemirror.org/sources/ \ 91 osc://.*/.\* http://somemirror.org/sources/ \ 92 p4://.*/.\* http://somemirror.org/sources/ \ 93 svn://.*/.\* http://somemirror.org/sources/" 94 95 MIRRORS =+ "\ 96 ftp://.*/.\* http://somemirror.org/sources/ \ 97 http://.*/.\* http://somemirror.org/sources/ \ 98 https://.*/.\* http://somemirror.org/sources/" 99 100It is useful to note that BitBake 101supports cross-URLs. It is possible to mirror a Git repository on an 102HTTP server as a tarball. This is what the ``git://`` mapping in the 103previous example does. 104 105Since network accesses are slow, BitBake maintains a cache of files 106downloaded from the network. Any source files that are not local (i.e. 107downloaded from the Internet) are placed into the download directory, 108which is specified by the :term:`DL_DIR` variable. 109 110File integrity is of key importance for reproducing builds. For 111non-local archive downloads, the fetcher code can verify SHA-256 and MD5 112checksums to ensure the archives have been downloaded correctly. You can 113specify these checksums by using the :term:`SRC_URI` variable with the 114appropriate varflags as follows:: 115 116 SRC_URI[md5sum] = "value" 117 SRC_URI[sha256sum] = "value" 118 119You can also specify the checksums as 120parameters on the :term:`SRC_URI` as shown below:: 121 122 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" 123 124If multiple URIs exist, you can specify the checksums either directly as 125in the previous example, or you can name the URLs. The following syntax 126shows how you name the URIs:: 127 128 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" 129 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d 130 131After a file has been downloaded and 132has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`. 133BitBake uses this stamp during subsequent builds to avoid downloading or 134comparing a checksum for the file again. 135 136.. note:: 137 138 It is assumed that local storage is safe from data corruption. If 139 this were not the case, there would be bigger issues to worry about. 140 141If :term:`BB_STRICT_CHECKSUM` is set, any 142download without a checksum triggers an error message. The 143:term:`BB_NO_NETWORK` variable can be used to 144make any attempted network access a fatal error, which is useful for 145checking that mirrors are complete as well as other things. 146 147If :term:`BB_CHECK_SSL_CERTS` is set to ``0`` then SSL certificate checking will 148be disabled. This variable defaults to ``1`` so SSL certificates are normally 149checked. 150 151.. _bb-the-unpack: 152 153The Unpack 154========== 155 156The unpack process usually immediately follows the download. For all 157URLs except Git URLs, BitBake uses the common ``unpack`` method. 158 159A number of parameters exist that you can specify within the URL to 160govern the behavior of the unpack stage: 161 162- *unpack:* Controls whether the URL components are unpacked. If set to 163 "1", which is the default, the components are unpacked. If set to 164 "0", the unpack stage leaves the file alone. This parameter is useful 165 when you want an archive to be copied in and not be unpacked. 166 167- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether 168 to use DOS line ending conversion on text files. 169 170- *striplevel:* Strip specified number of leading components (levels) 171 from file names on extraction 172 173- *subdir:* Unpacks the specific URL to the specified subdirectory 174 within the root directory. 175 176The unpack call automatically decompresses and extracts files with ".Z", 177".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and 178".bz2" extensions as well as various combinations of tarball extensions. 179 180As mentioned, the Git fetcher has its own unpack method that is 181optimized to work with Git trees. Basically, this method works by 182cloning the tree into the final directory. The process is completed 183using references so that there is only one central copy of the Git 184metadata needed. 185 186.. _bb-fetchers: 187 188Fetchers 189======== 190 191As mentioned earlier, the URL prefix determines which fetcher submodule 192BitBake uses. Each submodule can support different URL parameters, which 193are described in the following sections. 194 195.. _local-file-fetcher: 196 197Local file fetcher (``file://``) 198-------------------------------- 199 200This submodule handles URLs that begin with ``file://``. The filename 201you specify within the URL can be either an absolute or relative path to 202a file. If the filename is relative, the contents of the 203:term:`FILESPATH` variable is used in the same way 204``PATH`` is used to find executables. If the file cannot be found, it is 205assumed that it is available in :term:`DL_DIR` by the 206time the ``download()`` method is called. 207 208If you specify a directory, the entire directory is unpacked. 209 210Here are a couple of example URLs, the first relative and the second 211absolute:: 212 213 SRC_URI = "file://relativefile.patch" 214 SRC_URI = "file:///Users/ich/very_important_software" 215 216.. _http-ftp-fetcher: 217 218HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``) 219------------------------------------------------------------- 220 221This fetcher obtains files from web and FTP servers. Internally, the 222fetcher uses the wget utility. 223 224The executable and parameters used are specified by the 225``FETCHCMD_wget`` variable, which defaults to sensible values. The 226fetcher supports a parameter "downloadfilename" that allows the name of 227the downloaded file to be specified. Specifying the name of the 228downloaded file is useful for avoiding collisions in 229:term:`DL_DIR` when dealing with multiple files that 230have the same name. 231 232If a username and password are specified in the ``SRC_URI``, a Basic 233Authorization header will be added to each request, including across redirects. 234To instead limit the Authorization header to the first request, add 235"redirectauth=0" to the list of parameters. 236 237Some example URLs are as follows:: 238 239 SRC_URI = "http://oe.handhelds.org/not_there.aac" 240 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" 241 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan" 242 243.. note:: 244 245 Because URL parameters are delimited by semi-colons, this can 246 introduce ambiguity when parsing URLs that also contain semi-colons, 247 for example:: 248 249 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" 250 251 252 Such URLs should should be modified by replacing semi-colons with '&' 253 characters:: 254 255 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" 256 257 258 In most cases this should work. Treating semi-colons and '&' in 259 queries identically is recommended by the World Wide Web Consortium 260 (W3C). Note that due to the nature of the URL, you may have to 261 specify the name of the downloaded file as well:: 262 263 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" 264 265 266.. _cvs-fetcher: 267 268CVS fetcher (``(cvs://``) 269------------------------- 270 271This submodule handles checking out files from the CVS version control 272system. You can configure it using a number of different variables: 273 274- :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running 275 the ``cvs`` command. This name is usually "cvs". 276 277- :term:`SRCDATE`: The date to use when fetching the CVS source code. A 278 special value of "now" causes the checkout to be updated on every 279 build. 280 281- :term:`CVSDIR`: Specifies where a temporary 282 checkout is saved. The location is often ``DL_DIR/cvs``. 283 284- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the 285 ``cvs`` command. 286 287- CVS_PROXY_PORT: The port number to use as a "proxyport=" 288 parameter to the ``cvs`` command. 289 290As well as the standard username and password URL syntax, you can also 291configure the fetcher with various URL parameters: 292 293The supported parameters are as follows: 294 295- *"method":* The protocol over which to communicate with the CVS 296 server. By default, this protocol is "pserver". If "method" is set to 297 "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You 298 can use "dir" for local directories. 299 300- *"module":* Specifies the module to check out. You must supply this 301 parameter. 302 303- *"tag":* Describes which CVS TAG should be used for the checkout. By 304 default, the TAG is empty. 305 306- *"date":* Specifies a date. If no "date" is specified, the 307 :term:`SRCDATE` of the configuration is used to 308 checkout a specific date. The special value of "now" causes the 309 checkout to be updated on every build. 310 311- *"localdir":* Used to rename the module. Effectively, you are 312 renaming the output directory to which the module is unpacked. You 313 are forcing the module into a special directory relative to 314 :term:`CVSDIR`. 315 316- *"rsh":* Used in conjunction with the "method" parameter. 317 318- *"scmdata":* Causes the CVS metadata to be maintained in the tarball 319 the fetcher creates when set to "keep". The tarball is expanded into 320 the work directory. By default, the CVS metadata is removed. 321 322- *"fullpath":* Controls whether the resulting checkout is at the 323 module level, which is the default, or is at deeper paths. 324 325- *"norecurse":* Causes the fetcher to only checkout the specified 326 directory with no recurse into any subdirectories. 327 328- *"port":* The port to which the CVS server connects. 329 330Some example URLs are as follows:: 331 332 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" 333 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" 334 335.. _svn-fetcher: 336 337Subversion (SVN) Fetcher (``svn://``) 338------------------------------------- 339 340This fetcher submodule fetches code from the Subversion source control 341system. The executable used is specified by ``FETCHCMD_svn``, which 342defaults to "svn". The fetcher's temporary working directory is set by 343:term:`SVNDIR`, which is usually ``DL_DIR/svn``. 344 345The supported parameters are as follows: 346 347- *"module":* The name of the svn module to checkout. You must provide 348 this parameter. You can think of this parameter as the top-level 349 directory of the repository data you want. 350 351- *"path_spec":* A specific directory in which to checkout the 352 specified svn module. 353 354- *"protocol":* The protocol to use, which defaults to "svn". If 355 "protocol" is set to "svn+ssh", the "ssh" parameter is also used. 356 357- *"rev":* The revision of the source code to checkout. 358 359- *"scmdata":* Causes the ".svn" directories to be available during 360 compile-time when set to "keep". By default, these directories are 361 removed. 362 363- *"ssh":* An optional parameter used when "protocol" is set to 364 "svn+ssh". You can use this parameter to specify the ssh program used 365 by svn. 366 367- *"transportuser":* When required, sets the username for the 368 transport. By default, this parameter is empty. The transport 369 username is different than the username used in the main URL, which 370 is passed to the subversion command. 371 372Following are three examples using svn:: 373 374 SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667" 375 SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh" 376 SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1" 377 378.. _git-fetcher: 379 380Git Fetcher (``git://``) 381------------------------ 382 383This fetcher submodule fetches code from the Git source control system. 384The fetcher works by creating a bare clone of the remote into 385:term:`GITDIR`, which is usually ``DL_DIR/git2``. This 386bare clone is then cloned into the work directory during the unpack 387stage when a specific tree is checked out. This is done using alternates 388and by reference to minimize the amount of duplicate data on the disk 389and make the unpack process fast. The executable used can be set with 390``FETCHCMD_git``. 391 392This fetcher supports the following parameters: 393 394- *"protocol":* The protocol used to fetch the files. The default is 395 "git" when a hostname is set. If a hostname is not set, the Git 396 protocol is "file". You can also use "http", "https", "ssh" and 397 "rsync". 398 399 .. note:: 400 401 When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs 402 from the one that is typically passed to ``git clone`` command and provided 403 by the Git server to fetch from. For example, the URL returned by GitLab 404 server for ``mesa`` when cloning over SSH is 405 ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in 406 :term:`SRC_URI` is the following:: 407 408 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." 409 410 Note the ``:`` character changed for a ``/`` before the path to the project. 411 412- *"nocheckout":* Tells the fetcher to not checkout source code when 413 unpacking when set to "1". Set this option for the URL where there is 414 a custom routine to checkout code. The default is "0". 415 416- *"rebaseable":* Indicates that the upstream Git repository can be 417 rebased. You should set this parameter to "1" if revisions can become 418 detached from branches. In this case, the source mirror tarball is 419 done per revision, which has a loss of efficiency. Rebasing the 420 upstream Git repository could cause the current revision to disappear 421 from the upstream repository. This option reminds the fetcher to 422 preserve the local cache carefully for future use. The default value 423 for this parameter is "0". 424 425- *"nobranch":* Tells the fetcher to not check the SHA validation for 426 the branch when set to "1". The default is "0". Set this option for 427 the recipe that refers to the commit that is valid for any namespace 428 (branch, tag, ...) instead of the branch. 429 430- *"bareclone":* Tells the fetcher to clone a bare clone into the 431 destination directory without checking out a working tree. Only the 432 raw Git metadata is provided. This parameter implies the "nocheckout" 433 parameter as well. 434 435- *"branch":* The branch(es) of the Git tree to clone. Unless 436 "nobranch" is set to "1", this is a mandatory parameter. The number of 437 branch parameters must match the number of name parameters. 438 439- *"rev":* The revision to use for the checkout. The default is 440 "master". 441 442- *"tag":* Specifies a tag to use for the checkout. To correctly 443 resolve tags, BitBake must access the network. For that reason, tags 444 are often not used. As far as Git is concerned, the "tag" parameter 445 behaves effectively the same as the "rev" parameter. 446 447- *"subpath":* Limits the checkout to a specific subpath of the tree. 448 By default, the whole tree is checked out. 449 450- *"destsuffix":* The name of the path in which to place the checkout. 451 By default, the path is ``git/``. 452 453- *"usehead":* Enables local ``git://`` URLs to use the current branch 454 HEAD as the revision for use with ``AUTOREV``. The "usehead" 455 parameter implies no branch and only works when the transfer protocol 456 is ``file://``. 457 458Here are some example URLs:: 459 460 SRC_URI = "git://github.com/fronteed/icheck.git;protocol=https;branch=${PV};tag=${PV}" 461 SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main" 462 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." 463 464.. note:: 465 466 When using ``git`` as the fetcher of the main source code of your software, 467 ``S`` should be set accordingly:: 468 469 S = "${WORKDIR}/git" 470 471.. note:: 472 473 Specifying passwords directly in ``git://`` urls is not supported. 474 There are several reasons: :term:`SRC_URI` is often written out to logs and 475 other places, and that could easily leak passwords; it is also all too 476 easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` 477 and ``~/.ssh/config`` files can be used as alternatives. 478 479 480.. _gitsm-fetcher: 481 482Git Submodule Fetcher (``gitsm://``) 483------------------------------------ 484 485This fetcher submodule inherits from the :ref:`Git 486fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher 487(\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a 488repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as 489described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git 490fetcher (\`\`git://\`\`)` section. 491 492.. note:: 493 494 You must clean a recipe when switching between '``git://``' and 495 '``gitsm://``' URLs. 496 497 The Git Submodules fetcher is not a complete fetcher implementation. 498 The fetcher has known issues where it does not use the normal source 499 mirroring infrastructure properly. Further, the submodule sources it 500 fetches are not visible to the licensing and source archiving 501 infrastructures. 502 503.. _clearcase-fetcher: 504 505ClearCase Fetcher (``ccrc://``) 506------------------------------- 507 508This fetcher submodule fetches code from a 509`ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__ 510repository. 511 512To use this fetcher, make sure your recipe has proper 513:term:`SRC_URI`, :term:`SRCREV`, and 514:term:`PV` settings. Here is an example:: 515 516 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" 517 SRCREV = "EXAMPLE_CLEARCASE_TAG" 518 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}" 519 520The fetcher uses the ``rcleartool`` or 521``cleartool`` remote client, depending on which one is available. 522 523Following are options for the :term:`SRC_URI` statement: 524 525- *vob*: The name, which must include the prepending "/" character, 526 of the ClearCase VOB. This option is required. 527 528- *module*: The module, which must include the prepending "/" 529 character, in the selected VOB. 530 531 .. note:: 532 533 The module and vob options are combined to create the load rule in the 534 view config spec. As an example, consider the vob and module values from 535 the SRC_URI statement at the start of this section. Combining those values 536 results in the following:: 537 538 load /example_vob/example_module 539 540- *proto*: The protocol, which can be either ``http`` or ``https``. 541 542By default, the fetcher creates a configuration specification. If you 543want this specification written to an area other than the default, use 544the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where 545the specification is written. 546 547.. note:: 548 549 the SRCREV loses its functionality if you specify this variable. However, 550 SRCREV is still used to label the archive after a fetch even though it does 551 not define what is fetched. 552 553Here are a couple of other behaviors worth mentioning: 554 555- When using ``cleartool``, the login of ``cleartool`` is handled by 556 the system. The login require no special steps. 557 558- In order to use ``rcleartool`` with authenticated users, an 559 "rcleartool login" is necessary before using the fetcher. 560 561.. _perforce-fetcher: 562 563Perforce Fetcher (``p4://``) 564---------------------------- 565 566This fetcher submodule fetches code from the 567`Perforce <https://www.perforce.com/>`__ source control system. The 568executable used is specified by ``FETCHCMD_p4``, which defaults to "p4". 569The fetcher's temporary working directory is set by 570:term:`P4DIR`, which defaults to "DL_DIR/p4". 571The fetcher does not make use of a perforce client, instead it 572relies on ``p4 files`` to retrieve a list of 573files and ``p4 print`` to transfer the content 574of those files locally. 575 576To use this fetcher, make sure your recipe has proper 577:term:`SRC_URI`, :term:`SRCREV`, and 578:term:`PV` values. The p4 executable is able to use the 579config file defined by your system's ``P4CONFIG`` environment variable 580in order to define the Perforce server URL and port, username, and 581password if you do not wish to keep those values in a recipe itself. If 582you choose not to use ``P4CONFIG``, or to explicitly set variables that 583``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is 584the server's URL and port number, and you can specify a username and 585password directly in your recipe within :term:`SRC_URI`. 586 587Here is an example that relies on ``P4CONFIG`` to specify the server URL 588and port, username, and password, and fetches the Head Revision:: 589 590 SRC_URI = "p4://example-depot/main/source/..." 591 SRCREV = "${AUTOREV}" 592 PV = "p4-${SRCPV}" 593 S = "${WORKDIR}/p4" 594 595Here is an example that specifies the server URL and port, username, and 596password, and fetches a Revision based on a Label:: 597 598 P4PORT = "tcp:p4server.example.net:1666" 599 SRC_URI = "p4://user:passwd@example-depot/main/source/..." 600 SRCREV = "release-1.0" 601 PV = "p4-${SRCPV}" 602 S = "${WORKDIR}/p4" 603 604.. note:: 605 606 You should always set S to "${WORKDIR}/p4" in your recipe. 607 608By default, the fetcher strips the depot location from the local file paths. In 609the above example, the content of ``example-depot/main/source/`` will be placed 610in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot 611paths locally is desirable, the fetcher supports two parameters: 612 613- *"module":* 614 The top-level depot location or directory to fetch. The value of this 615 parameter can also point to a single file within the depot, in which case 616 the local file path will include the module path. 617- *"remotepath":* 618 When used with the value "``keep``", the fetcher will mirror the full depot 619 paths locally for the specified location, even in combination with the 620 ``module`` parameter. 621 622Here is an example use of the the ``module`` parameter:: 623 624 SRC_URI = "p4://user:passwd@example-depot/main;module=source/..." 625 626In this case, the content of the top-level directory ``source/`` will be fetched 627to ``${P4DIR}``, including the directory itself. The top-level directory will 628be accesible at ``${P4DIR}/source/``. 629 630Here is an example use of the the ``remotepath`` parameter:: 631 632 SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep" 633 634In this case, the content of the top-level directory ``source/`` will be fetched 635to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The 636top-level directory will be accessible at 637``${P4DIR}/example-depot/main/source/``. 638 639.. _repo-fetcher: 640 641Repo Fetcher (``repo://``) 642-------------------------- 643 644This fetcher submodule fetches code from ``google-repo`` source control 645system. The fetcher works by initiating and syncing sources of the 646repository into :term:`REPODIR`, which is usually 647``${DL_DIR}/repo``. 648 649This fetcher supports the following parameters: 650 651- *"protocol":* Protocol to fetch the repository manifest (default: 652 git). 653 654- *"branch":* Branch or tag of repository to get (default: master). 655 656- *"manifest":* Name of the manifest file (default: ``default.xml``). 657 658Here are some example URLs:: 659 660 SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml" 661 SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml" 662 663.. _az-fetcher: 664 665Az Fetcher (``az://``) 666-------------------------- 667 668This submodule fetches data from an 669`Azure Storage account <https://docs.microsoft.com/en-us/azure/storage/>`__ , 670it inherits its functionality from the HTTP wget fetcher, but modifies its 671behavior to accomodate the usage of a 672`Shared Access Signature (SAS) <https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview>`__ 673for non-public data. 674 675Such functionality is set by the variable: 676 677- :term:`AZ_SAS`: The Azure Storage Shared Access Signature provides secure 678 delegate access to resources, if this variable is set, the Az Fetcher will 679 use it when fetching artifacts from the cloud. 680 681You can specify the AZ_SAS variable as shown below:: 682 683 AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>" 684 685Here is an example URL:: 686 687 SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<foo_container>/<bar_file>" 688 689It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable. 690 691.. _crate-fetcher: 692 693Crate Fetcher (``crate://``) 694---------------------------- 695 696This submodule fetches code for 697`Rust language "crates" <https://doc.rust-lang.org/reference/glossary.html?highlight=crate#crate>`__ 698corresponding to Rust libraries and programs to compile. Such crates are typically shared 699on https://crates.io/ but this fetcher supports other crate registries too. 700 701The format for the :term:`SRC_URI` setting must be:: 702 703 SRC_URI = "crate://REGISTRY/NAME/VERSION" 704 705Here is an example URL:: 706 707 SRC_URI = "crate://crates.io/glob/0.2.11" 708 709.. _npm-fetcher: 710 711NPM Fetcher (``npm://``) 712------------------------ 713 714This submodule fetches source code from an 715`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ 716Javascript package registry. 717 718The format for the :term:`SRC_URI` setting must be:: 719 720 SRC_URI = "npm://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." 721 722This fetcher supports the following parameters: 723 724- *"package":* The NPM package name. This is a mandatory parameter. 725 726- *"version":* The NPM package version. This is a mandatory parameter. 727 728- *"downloadfilename":* Specifies the filename used when storing the downloaded file. 729 730- *"destsuffix":* Specifies the directory to use to unpack the package (default: ``npm``). 731 732Note that NPM fetcher only fetches the package source itself. The dependencies 733can be fetched through the `npmsw-fetcher`_. 734 735Here is an example URL with both fetchers:: 736 737 SRC_URI = " \ 738 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 739 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 740 " 741 742See :yocto_docs:`Creating Node Package Manager (NPM) Packages 743</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>` 744in the Yocto Project manual for details about using 745:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` 746to automatically create a recipe from an NPM URL. 747 748.. _npmsw-fetcher: 749 750NPM shrinkwrap Fetcher (``npmsw://``) 751------------------------------------- 752 753This submodule fetches source code from an 754`NPM shrinkwrap <https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap>`__ 755description file, which lists the dependencies 756of an NPM package while locking their versions. 757 758The format for the :term:`SRC_URI` setting must be:: 759 760 SRC_URI = "npmsw://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." 761 762This fetcher supports the following parameters: 763 764- *"dev":* Set this parameter to ``1`` to install "devDependencies". 765 766- *"destsuffix":* Specifies the directory to use to unpack the dependencies 767 (``${S}`` by default). 768 769Note that the shrinkwrap file can also be provided by the recipe for 770the package which has such dependencies, for example:: 771 772 SRC_URI = " \ 773 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 774 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 775 " 776 777Such a file can automatically be generated using 778:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` 779as described in the :yocto_docs:`Creating Node Package Manager (NPM) Packages 780</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>` 781section of the Yocto Project. 782 783Other Fetchers 784-------------- 785 786Fetch submodules also exist for the following: 787 788- Bazaar (``bzr://``) 789 790- Mercurial (``hg://``) 791 792- OSC (``osc://``) 793 794- Secure FTP (``sftp://``) 795 796- Secure Shell (``ssh://``) 797 798- Trees using Git Annex (``gitannex://``) 799 800No documentation currently exists for these lesser used fetcher 801submodules. However, you might find the code helpful and readable. 802 803Auto Revisions 804============== 805 806We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. 807