1% Xen Release Management 2% Wei Liu <<wei.liu2@citrix.com>> 3% Revision 1 4 5# Motivation 6 7Over the years we have had different people signing up as the Release Manager 8of Xen. It would be rather wasteful if every new Release Manager has to go over 9everything and tripped over by the same mistakes again and again. 10 11This file intends to document the process of managing a Xen release. It is 12mainly written for Release Manager, but other roles (contributors, 13maintainers and committers) are also encouraged to read this document, so 14that they can have an idea what to expect from the Release Manager. 15 16# Xen release cycle 17 18The Xen hypervisor project now releases every 8 months. We aim to 19release in the first half of March/July/November. These dates have 20been chosen to avoid major holidays and cultural events; if one 21release slips, ideally the subsequent release cycle would be shortened. 22The reasons for this schedule have been discussed on 23[xen-devel](https://lists.xen.org/archives/html/xen-devel/2018-07/msg02240.html). 24 25We can roughly divide one release into two periods. The development period 26and the freeze period. The former is 6 months long and the latter is about 2 27months long. 28 29During development period, contributors submit patches to be reviewed and 30committed into xen.git. All feature patches must be committed before a date, 31which is normally called the "cut-off date", after which the freeze period 32starts. There will be a date before which all patches that wish to be merged 33for the release should be posted -- it is normally called the "last posting 34date" and it is normally two weeks before the "cut-off date". 35 36During freeze period, the tree is closed for new features. Only bug fixes are 37accepted. This period can be shorter or longer than 2 months. If it ends up 38longer than 2 months, it eats into the next development period. 39 40The precise release schedule depends on a lot of factors and needs to 41be set afresh by the Release Manager in each release cycle. When the 42release is in March, particular consideration should be given to the 43Chinese New Year holiday which will then typically occur during the 44freeze, so the freeze should probably be extended to compensate. 45 46# The different roles in a Xen release 47 48## Release Manager 49 50A trusted developer in the community that owns the release process. The major 51goal of the Release Manager is to make sure a Xen release has high quality 52and doesn't slip too much. 53 54The Release Manager will not see much workload during development period, but 55expects to see increasing workload during the freeze period until the final 56release. He or she is expected to keep track of issues, arrange RCs, 57negotiate with relevant stakeholders, balance the need from various parties 58and make difficult decisions when necessary. 59 60The Release Manager essentially owns xen-unstable branch during the freeze 61period. The Committers will act on the wishes of the Release Manager during 62that time. 63 64## Maintainers 65 66A group of trusted developers who are responsible for certain components in 67xen.git. They are expected to respond to patches / questions with regard to 68their components in a timely manner, especially during the freeze period. 69 70## Committers 71 72A group of trusted maintainers who can commit to xen.git. During the 73development window they normally push things as they see fit. During the 74freeze period they transfer xen-unstable branch ownership and act on the 75wishes of the Release Manager. That normally means they need to have an 76Release Ack in order to push a patch. 77 78## Contributors 79 80Contributors are also expected to respond quickly to any issues regarding the 81code they submitted during development period. Failing that, the Release 82Manager might decide to revert the changes, declare feature unsupported or 83take any action he / she deems appropriate. 84 85## The Security Team 86 87The Security Team operates independently. The visibility might be rather 88limited due to the sensitive nature of security work. The best action the 89Release Manager can take is to set aside some time for potential security 90issues to be fixed. 91 92## The Release Technician 93 94The Release Technician is the person who tags various trees, prepares tarball 95etc. He or she acts on the wishes of the Release Manager. Please make sure 96the communication is as clear as it can be. 97 98## The Community Manager 99 100The Community Manager owns xenproject.org infrastructure. He or she is 101responsible for updating various web archives, updating wiki pages and 102coordinating with the PR Personnel. 103 104## The PR Personnel 105 106They are responsible for coordinating with external reporters to publish Xen 107release announcement. The Release Manager should be absolutely sure the 108release is going out on a particular date before giving them the signal to 109proceed, because there is a point of no return once they schedule a date with 110external reporters. 111 112# What happens during a release 113 114## Development period 115 116Send out monthly update email. The email contains the timeline of the 117release, the major work items and any other information the Release Manager 118sees fit. Reminders should also be sent one week before important dates (see 119above, "last posting date" and "cut-off date"). Please consider adding 120relevant events to your calendar. 121 122Occasionally check the status of the xen-unstable branch, make sure it gets 123timely pushes to master. 124 125## Freeze period 126 127Before or at very early stage of the freeze period, agree with the Community 128Manager a schedule for RC test days. 129 130Once the freeze starts, the ownership of xen-unstable branch automatically 131transfers to the Release Manager. The Release Manager can say "not releasing 132now" because of too many bugs, "until someone fixes these", or "no more 133patches until X, Y, and Z happen". 134 135Here is a list of things to do for making RCs: 136 1371. Check the status of the tree. Ask the Release Technician to make an RC if 138the tree is good. 139 1402. Send an email to xen-devel, xen-users and xen-announce to announce the RC. 141 1423. Branch and / or reopen the tree for further feature submission if 143appropriate. 144 1454. Collect and track any issues reported, determine their severity, prod 146relevant developers and maintainers to fix the issues. 147 1485. When patches to fix issues are posted, determine if the patches are good to 149be included. 150 1516. Go back to 1. 152 153It is normally OK in the early RCs that you hand back xen-unstable branch to 154committers so that they can commit bug fixes at will. As we approach late 155RCs, the standard for accepting a patch will get higher and higher. Please 156communicate clearly when committers can commit at will and when formal 157Release Ack is needed. 158 159At the same time, work with the Community Manager, PR Personnel and 160Contributors to gather a list of features for the release. Discuss the 161support status of new features with stakeholders. Help prepare the press 162release, write a blog post for the release. 163 164Make sure the key people for doing the release (especially Community Manager, 165Release Manager and Release Technician) will be either available around the 166planned release date or have named a substitute being capable to perform the 167required tasks. 168 1691. Collate a list of major changes: this should be done in collaboration 170between Release Manager, PR Personnel and key contributors. This should *not* 171be done on a public mailing list, to minimize the risk of release related 172media stories being published before the release date. 173 1742. PR Personnel will identify feature highlights, a theme for the press 175release, companies providing supporting quotes for the press release and 176media outlets we would want to reach out to and will manage the creation of 177the press release in private. 178 1793. The Community Manager will also draft blog post with the help of PR 180Personnel and Release Manager, which will be published under the name of the 181Release Manager. 182 1834. The Community Manager will create release related documentation such as 184Acknowledgements, Feature List, Man Pages and Release Notes on the wiki 185accessible via a release category. This can be done in public. 186 1875. PR Personnel will get stake-holder and Advisory Board approval for the 188press release (1-2 weeks before the release). 189 190 191When you think all pending issues are fixed and Xen is ready to be released 192from the last RC: 193 1941. Send out commit moratorium emails to committers@. 195 1962. Check all the trees (mini-os, qemu-trad, qemu-xen, seabios, ovmf etc). 197They have the correct commits and all security patches applied. There will be 198tools provided. 199 2003. Negotiate release date options with PR personnel. Typically we need 3-4 201days to line up press briefings with reporters under embargo. PR personnel 202will also need to consider industry events to ensure that PR is effective. PR 203releases typically done mid-week (Tuesday - Thursday). 204 2054. Select the release date. 206 2075. Specify the dates regarding support and security support in SUPPORT.md. 208 2096. Check with relevant stake-holders (typically community manager) whether 210wiki documentation and PR is in good shape (for an example see 211https://wiki.xenproject.org/wiki/Category:Xen_4.9 212<https://wiki.xenproject.org/wiki/Category:Xen_4.9>) 213 2147. Obtain a formal go-ahead from 215 216 * the Community Manager 217 * the Release Technician 218 219 Ask them to dry-run their checklist and confirm everything is OK. If not, 220 arrange another RC and restart this checklist. 221 2228. Do not commit to a release date until 223 224 * The exact xen.git commit id to be released is known. 225 * That commit id has been satisfactorily tested. 226 2279. Give PR Personnel final go-ahead, and instruct Release Technician to make 228release deliverables (tags and tarballs - will usually be in place the day 229before the release). At this point, PR collateral will be sent to reporters 230(typically 2-3 working days before the release date) and we cannot undo 231publications without questions being asked and risk of negative PR. It is 232acceptable to make a xen-devel@ announcement *before* the PR release date 233(blog, xen-announce@, press release). 234 23510. Make the announcement on various mailing list, publish the blog post. 236 237Allow for contingencies. It is not uncommon that some last minute (security or 238not) bugs are discovered. To provide a fix takes time, the test of the fix 239will also take time. Allow for at least 1 week from getting a fix to getting 240a push. For security bugs, coordinate with the Security Team to adjust the 241dates according to our security policy. 242 243## Hand over of Release Manager responsibility 244 245If there is a new Release Manager for the next release, make sure the 246following things happen for the new Release Manager. 247 2481. A JIRA (xenproject.atlassian.net) is created and proper permissions granted. 2492. Access to community test infrastructure is granted. 250 In the common case the public pages at logs.test-lab.xenproject.org will 251 suffice. 2523. Access to mailing list moderation panel is granted. 2534. An account for blog.xenproject.org is created. 254 The account can be created by the new Release Manager, it might be necessary 255 to adjust the access rights. 2565. An account for wiki.xenproject.org is created. 257 The account can be created by the new Release Manager, it might be necessary 258 to adjust the access rights. 259 260# Email templates and scripts 261 262Note: if you want specific actions from committers, please make sure you CC 263committers@. 264 265## RC emails 266 267``` 268Subject: Xen X.Y rcZ 269 270Hi all, 271 272Xen X.Y rcZ is tagged. You can check that out from xen.git: 273 274https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ 275 276For your convenience there is also a tarball at: 277https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz 278 279And the signature is at: 280https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig 281 282Please send bug reports and test reports to xen-devel@lists.xenproject.org. 283When sending bug reports, please CC relevant maintainers and me 284(abc@xyz.com). 285 286As a reminder, there will be another Xen Test Day. 287 288See instructions on: URL_TO_TEST_INSTRUCTIONS 289``` 290 291## Forego control of the tree 292 293``` 294Subject: No Release Ack needed before RcX 295 296Committers, 297 298The tree is in good state. No release ack is needed before RcX. Please commit 299bug fixes at will. 300 301$RM 302``` 303 304## Commit moratorium 305 306``` 307Subject: Commit moratorium for $REASON 308 309Committers, 310 311Please don't push any new patch to staging because $REASON. 312 313Another email will be sent once the moratorium is lifted. 314 315$RM 316``` 317 318## Lift commit moratorium 319 320``` 321Subject: Commit moratorium is lifted for $REASON 322 323Committers, 324 325The commit moratorium is lifted, please commit patches that are already 326Release-acked. 327 328$RM 329``` 330 331## Reminder of last posting date 332 333``` 334Subject: Last posting date for Xen X.Y is $DATE 335 336Hi all, 337 338The last posting date for Xen X.Y is $DATE. If you want your features to be 339included for the release, please make sure they are posted for the first 340time before $DATE. 341 342$RM 343``` 344 345## Reminder of cut-off date 346 347``` 348Subject: Cut-off date for Xen X.Y is $DATE 349 350Hi all, 351 352The cut-off date for Xen X.Y is $DATE. If you want your features to be 353included for the release, please make sure they are committed by $DATE. 354 355$RM 356``` 357 358## Release announcement 359 360``` 361 Subject: [ANNOUNCEMENT] Xen X.Y is released 362 363 Dear community members, 364 365 I'm pleased to announce that Xen X.Y.0 is released. 366 367 Please find the tarball and its signature at: 368 369 https://xenproject.org/downloads/xen-archives/xen-project-xy-series/xen-project-xy0.html 370 371 You can also check out the tag in xen.git: 372 373 https://xenbits.xen.org/git-http/xen.git RELEASE-X.Y.0 374 375 Git checkout and build instructions can be found at: 376 377 https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes#Build_Requirements 378 379 Release notes can be found at: 380 381 https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes 382 383 A summary for X.Y release documents can be found at: 384 385 https://wiki.xenproject.org/wiki/Category:Xen_X.Y 386 387 Technical blog post for X.Y can be found at: 388 389 URL_TO_BLOG 390 391 Thanks everyone who contributed to this release. This release would 392 not have happened without all the awesome contributions from around 393 the globe. 394 395 Regards, 396 397 $RM (on behalf of the Xen Project Hypervisor team) 398``` 399 400 401## Script to generate months update emails 402 403``` 404#!/bin/bash 405# Use ssmtp for simplicity 406# ./status-release.sh | formail -f -s /usr/sbin/ssmtp -bm -t 407 408FILE=`mktemp` 409cat << EOF > $FILE 410 411== Hypervisor == 412 413S: Per-cpu tasklet 414O: Konrad Rzeszutek Wilk 415E: konrad.wilk@oracle.com 416J: XEN-28 417 418=== x86 === 419 420=== ARM === 421 422== Completed == 423 424S: 425EOF 426 427 428AWK_FILE=`mktemp` 429cat << EOF > $AWK_FILE 430BEGIN { s2_count = 1;score = ""; emails=1; first_time = 1; subject=""} 431/== / { 432 if ( subject != "" ) { 433 if (score != "") 434 print "* ", subject, "("score")" 435 else if (version != "") 436 print "* ", subject, "("version")"; 437 else 438 print "* ", subject; 439 for (i = 1; i <= s2_count; i++) { 440 if (i in s2) 441 print " ",s2[i]; 442 } 443 if (bug != "") 444 print " Link: https://bugs.xenproject.org/xen/bug/"bug 445 if (jira != "") 446 print " - "jira 447 for (i = 1; i <= count; i++) { 448 if (i in o) 449 print " -", o[i] 450 } 451 if (emails) 452 print "" 453 first_time = 1; 454 subject="" 455 email="" 456 score="" 457 bug="" 458 jira="" 459 version="" 460 count = 1; 461 s2_count = 1; 462 delete s; 463 delete s2; 464 delete o; 465 delete e; 466 } 467 print \$0,"\n" 468 } 469/;/ { }; 470/S:/ { 471 if ( !first_time ) { 472 if (score != "") 473 print "* ", subject, "("score")" 474 else if (version != "") 475 print "* ", subject, "("version")"; 476 else 477 print "* ", subject 478 for (i = 1; i <= s2_count; i++) { 479 if (i in s2) 480 print " ",s2[i]; 481 } 482 if (bug != "") 483 print " Link: https://bug.xenproject.org/xen/bug/"bug 484 if (jira != "") 485 print " - "jira 486 for (i = 1; i <= count; i++) { 487 if (i in o) 488 print " -", o[i] 489 } 490 if (emails) 491 print "" 492 } 493 first_time = 0; 494 sub(\$1, ""); 495 sub(/^[ \t]+/, ""); 496 subject=\$0; 497 email="" 498 bug="" 499 jira="" 500 count = 1; 501 s2_count = 1; 502 delete s; 503 delete s2; 504 delete o; 505 delete e; 506 score=""; 507 version=""; 508 } 509/O:/ { sub(\$1, ""); o[count++]=\$0; }; 510/S2:/ { sub(\$1, ""); s2[s2_count++]=\$0;}; 511/E:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); email=\$0; e[emails++]=\$0;}; 512/P:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); score=\$0; }; 513/B:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); bug=\$0; }; 514/J:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); jira=\$0; }; 515/V:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); version=\$0; }; 516END { 517 } 518// { } 519EOF 520AWK_FILE_EMAIL=`mktemp` 521cat << EOF > $AWK_FILE_EMAIL 522BEGIN { emails=1;} 523/E:/ { 524 sub(\$1, ""); sub(/^[ \t]+/, ""); 525 email=\$0; 526 for ( i = 1; i <= emails; i++ ) { 527 if (i in e) { 528 if (e[i] == email) { 529 email=""; 530 break; 531 } 532 } 533 } 534 if (email != "") 535 e[emails++]=email; 536} 537END { 538 printf "Bcc: " 539 for ( i = 1; i <= emails; i++ ) 540 if (i in e) { 541 if (i == emails - 1) 542 printf "<%s>", e[i]; 543 else 544 printf "<%s>,", e[i]; 545 } 546 print "" 547 } 548// { } 549EOF 550 551echo "From: $RELEASE_MANAGER_NAME <$RELEASE_MANAGER_MAIL>" 552echo "To: xen-devel@lists.xenproject.org" 553echo "Cc: $RELEASE_MANAGER_MAIL" 554cat $FILE | awk -f $AWK_FILE_EMAIL 555rm $AWK_FILE_EMAIL 556 557echo "Subject: Xen $RELEASE_VERSION Development Update" 558PRE=`mktemp` 559cat << EOF > $PRE 560 561This email only tracks big items for xen.git tree. Please reply for items you 562would like to see in $RELEASE_VERSION so that people have an idea what is going on and 563prioritise accordingly. 564 565You're welcome to provide description and use cases of the feature you're 566working on. 567 568= Timeline = 569 570We now adopt a fixed cut-off date scheme. We will release twice a 571year. The upcoming $RELEASE_VERSION timeline are as followed: 572 573* Last posting date: $RELEASE_CUTOFF 574* Hard code freeze: $RELEASE_FREEZE 575* RC1: TBD 576* Release: $RELEASE_DATE 577 578Note that we don't have freeze exception scheme anymore. All patches 579that wish to go into $RELEASE_VERSION must be posted no later than the last posting 580date. All patches posted after that date will be automatically queued 581into next release. 582 583RCs will be arranged immediately after freeze. 584 585We recently introduced a jira instance to track all the tasks (not only big) 586for the project. See: https://xenproject.atlassian.net/projects/XEN/issues. 587 588Most of the tasks tracked by this e-mail also have a corresponding jira task 589referred by XEN-N. 590 591I have started to include the version number of series associated to each 592feature. Can each owner send an update on the version number if the series 593was posted upstream? 594 595= Projects = 596 597EOF 598 599POST=`mktemp` 600cat <<EOF > $POST 601 602EOF 603 604# Preamble 605cat $PRE 606rm $PRE 607# Body 608cat $FILE | awk -f $AWK_FILE 609rm $AWK_FILE 610rm $FILE 611cat $POST 612rm $POST 613``` 614