1*4882a593Smuzhiyun.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2*4882a593Smuzhiyun 3*4882a593Smuzhiyun*********************************** 4*4882a593SmuzhiyunProject Testing and Release Process 5*4882a593Smuzhiyun*********************************** 6*4882a593Smuzhiyun 7*4882a593SmuzhiyunDay to Day Development 8*4882a593Smuzhiyun====================== 9*4882a593Smuzhiyun 10*4882a593SmuzhiyunThis section details how the project tests changes, through automation 11*4882a593Smuzhiyunon the Autobuilder or with the assistance of QA teams, through to making 12*4882a593Smuzhiyunreleases. 13*4882a593Smuzhiyun 14*4882a593SmuzhiyunThe project aims to test changes against our test matrix before those 15*4882a593Smuzhiyunchanges are merged into the master branch. As such, changes are queued 16*4882a593Smuzhiyunup in batches either in the ``master-next`` branch in the main trees, or 17*4882a593Smuzhiyunin user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton 18*4882a593Smuzhiyunhelps review and test patches and this is his testing tree). 19*4882a593Smuzhiyun 20*4882a593SmuzhiyunWe have two broad categories of test builds, including "full" and 21*4882a593Smuzhiyun"quick". On the Autobuilder, these can be seen as "a-quick" and 22*4882a593Smuzhiyun"a-full", simply for ease of sorting in the UI. Use our Autobuilder 23*4882a593Smuzhiyunconsole view to see where me manage most test-related items, available 24*4882a593Smuzhiyunat: :yocto_ab:`/typhoon/#/console`. 25*4882a593Smuzhiyun 26*4882a593SmuzhiyunBuilds are triggered manually when the test branches are ready. The 27*4882a593Smuzhiyunbuilds are monitored by the SWAT team. For additional information, see 28*4882a593Smuzhiyun:yocto_wiki:`/Yocto_Build_Failure_Swat_Team`. 29*4882a593SmuzhiyunIf successful, the changes would usually be merged to the ``master`` 30*4882a593Smuzhiyunbranch. If not successful, someone would respond to the changes on the 31*4882a593Smuzhiyunmailing list explaining that there was a failure in testing. The choice 32*4882a593Smuzhiyunof quick or full would depend on the type of changes and the speed with 33*4882a593Smuzhiyunwhich the result was required. 34*4882a593Smuzhiyun 35*4882a593SmuzhiyunThe Autobuilder does build the ``master`` branch once daily for several 36*4882a593Smuzhiyunreasons, in particular, to ensure the current ``master`` branch does 37*4882a593Smuzhiyunbuild, but also to keep ``yocto-testresults`` 38*4882a593Smuzhiyun(:yocto_git:`/yocto-testresults/`), 39*4882a593Smuzhiyunbuildhistory 40*4882a593Smuzhiyun(:yocto_git:`/poky-buildhistory/`), and 41*4882a593Smuzhiyunour sstate up to date. On the weekend, there is a master-next build 42*4882a593Smuzhiyuninstead to ensure the test results are updated for the less frequently 43*4882a593Smuzhiyunrun targets. 44*4882a593Smuzhiyun 45*4882a593SmuzhiyunPerformance builds (buildperf-\* targets in the console) are triggered 46*4882a593Smuzhiyunseparately every six hours and automatically push their results to the 47*4882a593Smuzhiyunbuildstats repository at: 48*4882a593Smuzhiyun:yocto_git:`/yocto-buildstats/`. 49*4882a593Smuzhiyun 50*4882a593SmuzhiyunThe 'quick' targets have been selected to be the ones which catch the 51*4882a593Smuzhiyunmost failures or give the most valuable data. We run 'fast' ptests in 52*4882a593Smuzhiyunthis case for example but not the ones which take a long time. The quick 53*4882a593Smuzhiyuntarget doesn't include \*-lsb builds for all architectures, some world 54*4882a593Smuzhiyunbuilds and doesn't trigger performance tests or ltp testing. The full 55*4882a593Smuzhiyunbuild includes all these things and is slower but more comprehensive. 56*4882a593Smuzhiyun 57*4882a593SmuzhiyunRelease Builds 58*4882a593Smuzhiyun============== 59*4882a593Smuzhiyun 60*4882a593SmuzhiyunThe project typically has two major releases a year with a six month 61*4882a593Smuzhiyuncadence in April and October. Between these there would be a number of 62*4882a593Smuzhiyunmilestone releases (usually four) with the final one being stabilization 63*4882a593Smuzhiyunonly along with point releases of our stable branches. 64*4882a593Smuzhiyun 65*4882a593SmuzhiyunThe build and release process for these project releases is similar to 66*4882a593Smuzhiyunthat in :ref:`test-manual/test-process:day to day development`, in that the 67*4882a593Smuzhiyuna-full target of the Autobuilder is used but in addition the form is 68*4882a593Smuzhiyunconfigured to generate and publish artifacts and the milestone number, 69*4882a593Smuzhiyunversion, release candidate number and other information is entered. The 70*4882a593Smuzhiyunbox to "generate an email to QA"is also checked. 71*4882a593Smuzhiyun 72*4882a593SmuzhiyunWhen the build completes, an email is sent out using the send-qa-email 73*4882a593Smuzhiyunscript in the ``yocto-autobuilder-helper`` repository to the list of 74*4882a593Smuzhiyunpeople configured for that release. Release builds are placed into a 75*4882a593Smuzhiyundirectory in https://autobuilder.yocto.io/pub/releases on the 76*4882a593SmuzhiyunAutobuilder which is included in the email. The process from here is 77*4882a593Smuzhiyunmore manual and control is effectively passed to release engineering. 78*4882a593SmuzhiyunThe next steps include: 79*4882a593Smuzhiyun 80*4882a593Smuzhiyun- QA teams respond to the email saying which tests they plan to run and 81*4882a593Smuzhiyun when the results will be available. 82*4882a593Smuzhiyun 83*4882a593Smuzhiyun- QA teams run their tests and share their results in the yocto- 84*4882a593Smuzhiyun testresults-contrib repository, along with a summary of their 85*4882a593Smuzhiyun findings. 86*4882a593Smuzhiyun 87*4882a593Smuzhiyun- Release engineering prepare the release as per their process. 88*4882a593Smuzhiyun 89*4882a593Smuzhiyun- Test results from the QA teams are included into the release in 90*4882a593Smuzhiyun separate directories and also uploaded to the yocto-testresults 91*4882a593Smuzhiyun repository alongside the other test results for the given revision. 92*4882a593Smuzhiyun 93*4882a593Smuzhiyun- The QA report in the final release is regenerated using resulttool to 94*4882a593Smuzhiyun include the new test results and the test summaries from the teams 95*4882a593Smuzhiyun (as headers to the generated report). 96*4882a593Smuzhiyun 97*4882a593Smuzhiyun- The release is checked against the release checklist and release 98*4882a593Smuzhiyun readiness criteria. 99*4882a593Smuzhiyun 100*4882a593Smuzhiyun- A final decision on whether to release is made by the YP TSC who have 101*4882a593Smuzhiyun final oversight on release readiness. 102