1LTS - Long-Term Support 2======================= 3 4.. table:: Table 1: Document History 5 6 +-------------+--------------------+-------------------------------------------------------+ 7 | Date | Author | Description | 8 +=============+====================+=======================================================+ 9 | 2022-07-20 | Okash Khawaja, | Initial draft. | 10 | | Varun Wadekar | | 11 +-------------+--------------------+-------------------------------------------------------+ 12 | 2022-07-21 | Varun Wadekar | Refine the Maintainership guidelines and planning | 13 | | | sections. Introduce a new section documenting a day | 14 | | | in the life of a LTS branch maintainer | 15 +-------------+--------------------+-------------------------------------------------------+ 16 | 2022-08-05 | Okash Khawaja, | Merge two drafts (draft 1 and 2), address comments | 17 | | Varun Wadekar | made by both authors, cosmetic changes to the content | 18 | | | all over the document | 19 +-------------+--------------------+-------------------------------------------------------+ 20 | 2022-08-05 | Okash Khawaja | Add note about testing support available from TF.org | 21 +-------------+--------------------+-------------------------------------------------------+ 22 | 2022-08-05 | Varun Wadekar | Changed the “Future plans” section to “FAQ” and | 23 | | | answered some of the questions with feedback from | 24 | | | the community. | 25 +-------------+--------------------+-------------------------------------------------------+ 26 |2025-01-07 | Govindraj Raja | Convert from pdf to rst. | 27 +-------------+--------------------+-------------------------------------------------------+ 28 29This document proposes a plan for long-term support (LTS) of the |TF-A| project. 30 31Why is LTS required? 32-------------------- 33LTS is needed for commercial reasons. More specifically, on the device side, 34when a product is released, the companies have to support that in-market product 35such that the amount of changes to the firmware are kept to a minimum to avoid 36the risk of regression. At the same time the companies don't want to exclude 37critical patches such as those for security advisories. Similarly on the server side, 38companies want to minimize the churn when deploying fixes during incident 39response, e.g. due to critical security bugs. 40 41This means that those companies have to maintain and backport critical updates to 42old branches internally. As this effort is duplicated across different companies 43using TF-A, it makes sense to factor out this effort into a community-wide LTS. 44 45What does LTS mean for TF-A? 46---------------------------- 47In this section we will define exactly what constitutes LTS for TF-A. 48Specifically, we will define the following characteristics: 49 50- criteria for selecting patches which will be backported to LTS branches 51- lifetime and frequency of LTS branches 52 53**Criteria** 54 55We must have an objective criterion for selecting patches to be backported to 56LTS branches. This will make maintenance easy because: 57 58a. there will be less -- ideally no -- discussion when selecting patches to backport 59b. large parts of the process can be automated 60 61Below is the criteria 62 63#. No features will be backported. 64#. Security advisories: Any patch that makes it into :ref:`Security Advisories` 65 is automatically selected for back porting. This includes patches to external 66 components too, e.g. libfdt. 67#. Workarounds for CPU and other ARM IP errata 68#. Workarounds for non-ARM IP errata, e.g. TI UART 69#. Fixes for platform bugs. These patches must not modify any code outside of 70 the specific platform that the fix applies to. 71#. Patches can only be backported from the master branch. In other words, the 72 master branch will be a superset of all the changes in any LTS branch. 73 74**Lifetime and frequency** 75 76This section approaches three questions: for how long should an LTS release be 77supported, how frequently should LTS releases be made and at which time(s) of 78the year should the releases be made. 79 801. For how long should an LTS release be supported? 81 82Linux kernel supports an LTS branch for 5 years. Since firmware tends to 83have less churn and longer lifetime than HLOS, TF-A should support at least 845 years for its LTS. We should leave the room open for discussions about 85extending it to 7 years. 86 872. How frequently should LTS releases be made? 88 89Given that many products that have a release cycle, have a yearly release 90cycle, it would make sense to have yearly TF-A releases. 91 923. Which time(s) of the year should the releases be made? 93 94TF-A releases are cut twice a year: May and November. Basing LTS release 95on the November TF-A release has a few benefits. First, it aligns with Linux 96LTS releases which happen towards the end of each year. Second, it aligns 97with Android releases which tend to fall in Q3 each year. Since product 98releases are timed with Android release, this gives enough time to harden 99the TF-A LTS release during development so that it's ready for launch in 100Q3 following year. On the other hand, if the May release of TF-A is chosen as 101the basis for LTS then developers will have little time -- about a month, 102taking into account the test-and-debug phase before LTS is cut (see below) -- 103before Android release. 104 105To summarize, there will be one LTS release per year. It will be supported for 1065 years and we can discuss extending it to 7 years later on. The LTS release 107will be based on the November release of TF-A. 108 109**Testing Criteria** 110 111Every patch merged to the LTS branch will complete the following tests before 112getting approved. 113 114#. TFTF tests currently running in the testing farm 115#. CI/CD static analysis scans 116#. Coverity scans 117#. Platform tests 118 119Platforms that are not maintained upstream will undergo testing downstream in a 120pre-defined window. The platform maintainer will complete the testing and provide 121a verified score on the patch once testing is completed. 122 123** A note about test coverage from TF.org ** 124 125Currently TF.org maintains a CI system to run TF-A automated tests on a 126selection of HW boards donated by TF.org members (a benefit reserved to project 127members, see the project charter for more details). This automated test coverage 128will be extended to cover testing for LTS as well for boards that are part of 129the CI system. 130 131**TFTF branching** 132 133A note about testing here. After a patch is backported to an LTS branch, that 134branch will need to be regression tested. Since TFTF moves forward with latest 135TF-A changes, newer TFTF tests may not apply to old LTS branches. Therefore 136TFTF will also need to be branched, in-sync with TF-A LTS branches. In other 137words, there will be one TFTF LTS branch corresponding to each TF-A LTS branch. 138The TFTF LTS branch will be used to regression test the corresponding TF-A LTS 139branch. 140 141As we work with the LTS branch of TFTF, we might also need fixes for TFTF 142itself to be ported to LTS. However, decision-making about those patches need 143not be as stringent as for TF-A. 144 145Release details 146--------------- 147This section goes into details of what the LTS release process will look like. 148 149 150**Test-and-debug period** 151 152Since the LTS branch will be used in product releases, it is expected that more 153testing and debugging will be done on the November release of TF-A. Therefore 154it would make sense to leave at least a month after the November release and 155then cut the LTS branch. We recommend two months, given that one of the months 156is December which tends to be slower due to holidays. So, an end-of-November 157TF-A release would result in a beginning-of-February LTS release. Note that 158the LTS branch will be created at the same time as the TF-A November release, 159but it will be officially released at the end of January or early February. 160Going forward we should strive to make the period smaller and smaller until 161ideally it coincides with TF-A November release which means that our test 162and CI/CD infra is good enough to allow that to happen. 163 164**Example timeline** 165 166Below is an example timeline starting from the November 2022 release of TF-A. 167 168.. image:: ../resources/diagrams/lts-timeline-example.png 169 170- Nov 2022: TF-A 2.8 is released towards the end of Nov, 2022. Not shown in the 171 diagram, at the same time LTS release candidate branch is made which is based 172 on TF-A 2.8. This means new features going in 2.8 won’t go in the LTS branch. 173 We can call it `LTS 2.8-rc`. 174- Feb 2023: After testing and debugging LTS 2.8-rc for a couple of months, 175 LTS 2.8.0 is officially released in early Feb 2023. 176- May 2023: TF-A 2.9 is released but since this is not an LTS branch it doesn’t 177 affect LTS. 178- Somewhere between May and Nov of 2023: A security advisory comes up and the 179 related patches go into TF-A master branch. Since these patches fall under 180 LTS criteria, they are backported to LTS 2.8.0 which results in LTS 2.8.1 181 being released. Note that here we don’t allow the extra testing and debugging 182 time that we had between Nov 2022 and early Feb 2023. This is because there 183 isn’t as much to test and debug as an annual LTS release has. Also companies 184 might want to deploy critical patches soon. 185- Nov 2023: TF-A 2.10 is released. Not shown in the diagram, at the same time 186 LTS 2.10-rc is made. It’s tested by partners for a couple of months. 187- Feb 2024: LTS 2.10.1 is released in early Feb. Now there are two LTS 188 branches: 2.8.1 and 2.10.1. 189 190Note that TFTF will follow similar branching model as TF-A LTS, i.e. there will 191be TFTF LTS 2.8.0 in Feb 2023, 2.8.1 (if new TFTF tests need to be added for 192the security advisory) when there is TF-A LTS 2.8.1 and so on. 193 194Maintainership 195-------------- 196 197**Guidelines & Responsibilities** 198 199#. Maintainers shall be impartial and strive to work for the benefit of 200 the community 201#. Objective and well-defined merge criteria to avoid confusion and discussions 202 at random points in time when there is a "candidate" patch 203#. The maintainers shall explain the lifecycle of a patch to the community, 204 with a detailed description of the maximum time spent in each step 205#. Automate, automate, automate 206#. Reviewers should not focus too much on "what" and instead focus on "how" 207#. Constantly refine the merge criteria to include more partner use cases 208#. Ensure that all candidate patches flow from the main branch to all LTS branches 209 210**Options** 211 212These are some options in the order of preference. 213 214#. Current set of :ref:`lts maintainers` from tf.org(or hired contractor) take care of the LTS 215#. From the community, create a set of maintainers focused solely on the LTS branches 216 217A day in the life of a maintainer 218********************************* 219This section documents the daily tasks that a maintainer might perform to 220support the LTS program. It is expected that a maintainer follows clearly laid 221down steps and does not have to make policy level decisions for merge, testing, 222or candidate patch selection. 223 224#. Monitor the main branch to identify candidate patches for the LTS branches 225#. Inform the LTS maintainers mailing list of a new candidate patch for LTS and solicit feedback 226#. Start the review process and CI/CD cycle for the patch 227#. Review the CI/CD output to ensure that the quality bar is met 228#. After reviews are complete, merge the patch and bump the minor version, if required 229#. Monitor the mailing list for any LTS related issues 230#. Propose or solicit patches to the main branch and tag them as candidates for LTS 231 232Execution Plan 233************** 234This section lists the steps needed to put the LTS system in place. However, 235to kick start LTS in Nov ‘22, only a few steps are needed. The rest can follow 236in the background. 237 238Initial release steps 239********************* 240 241The following steps are necessary to kickstart the project and potentially 242create the first LTS from the Nov’22 release. 243 244#. Create a TF-A LTS release-candidate branch and a TFTF LTS branch immediately 245 after the Nov’22 release 246#. Request all platform-owners to test and debug the RC branch 247#. Gather feedback from the test and debug cycle 248#. Mark the TF-A LTS branch ready by the end of January 249#. Announce the official LTS release availability on the mailing lists 250 251Long term release plan 252********************** 253Above will buy us time to then work on the rest of the execution plan which 254is given below. 255 256#. The review criteria for LTS patches must be the same as TF-A patches 257#. The maintainers shall publish the well-defined merge criteria to allow 258 the community to choose candidate patches 259#. The maintainers shall publish a well-defined test specification for any 260 patch entering the LTS branch 261 262 a. Tests required to pass in the CI/CD flow 263 b. Static analysis scans 264 c. Coverity scans 265 266#. The maintainers shall publish a mechanism to choose candidate patches for 267 the LTS branch 268#. The maintainers shall publish a mechanism to report bugs `[1]`_ seen with 269 an LTS branch 270#. The maintainers shall publish a versioning mechanism for the LTS branch 271 272 a. Bump minor version for every “logical” `[2]`_ fix that gets merged 273 274#. The CI/CD infrastructure shall provide test support for all “live” LTS 275 branches at any given point in time 276#. The CI/CD infrastructure shall provide means to 277 278 a. notify all maintainers that a patch is ready for review 279 b. automatically cherry-pick a patch to a given LTS branch 280 c. get it through the CI/CD testing flow 281 d. send nag emails to maintainers at regular intervals to ensure reviews keep moving 282 283FAQ 284*** 285 286In our discussions, in addition to the above points we also considered some 287questions. They have been discussed on the mailing list too. 288 289| Q. What happens when a bug fix applies just to a LTS branch and not to the 290 master branch? 291| A. This will be treated as a special case and the bug, and the fix will be 292 discussed 293 294| Q. When testing a backported patch, what if one of the partners needs more 295 time while the patch fix is time-critical and, hence slowing other 296 partners? 297| A. The maintainers will add more detail to the review and merge process to 298 handle this scenario. 299 300| Q. How do we handle the increasing version numbers for errata fixes? 301| A. Too many CPU errata workarounds resulting in too many LTS releases. 302 We propose bumping the version number for each logical fix as 303 described in the section “Long term release plan” above because 304 that will help accurately track what changes have been deployed in-field. 305 306| Q. What if LTS support duration needs to be extended to longer than 5 years? 307| A. Still under discussion. 308 309These are uncharted waters, and we will face some unseen problems. When they 310become real problems, then we will have concrete data and be better able to 311address them. This means that our LTS definition as presented in this document 312is not the final one. We will constantly be discussing it and deciding how to 313adapt it as we see practical problems. 314 315.. _[1]: 316 317[1] The plan is to create a system where reviewers can tag a patch on mainline which 318gets automatically rebased on LTS and pushed to Gerrit. On seeing this patch, 319the CI/CD starts tests and provides a score. In parallel, the system also sends 320an email to the maintainers announcing the arrival of a candidate patch for the 321LTS branch. 322 323.. _[2]: 324 325[2] Logical will be a patch or patches implementing a certain fix. For example, if a 326security mitigation is fixed with the help of three patches, then all of them are 327considered as one "logical" fix. The version is incremented only after all these 328patches are merged. with the maintainers. If agreed unanimously, the bug fix 329will be merged to the affected LTS branches after completing the review process. 330
