= phenomenon = https://fedoraproject.org/wiki/Changes/GCC6 asks for a mass rebuild for Fedora 24. We have not allowed for one in the schedule
I expect a to see a few builds fail due to the changes https://lists.fedoraproject.org/pipermail/devel/2015-November/216384.html I have had reports that many fedora 23 packages are not rebuildable due to trigger issues
= background analysis =
While we can likely get by without a mass rebuild I think we are better off having one, gcc6 is expected to land in rawhide around the end of January based on discusssions I had with Jakub. with the schedule we have we would have to do a mass rebuild the first week in January to not slip any.
= implementation recommendation =
Push Fedora 24 out 3-4 weeks and allow for mass rebuild first week of February
The addition of strlcpy and strlcat to glibc (which should happen before its 2.23 release, scheduled to be included in Fedora 24) may also cause build failures. This might be another reason to consider a mass rebuild.
strlcpy
strlcat
I prepared two options accommodating the mass rebuild at the first week of February, which seems to be feasible for me.
My personal preference is the schedule with 2 weeks slip. There will be "Change submission deadline" on 2016-01-26. Week after the deadline we can start the mass rebuild. Then we have two weeks for the mass rebuild it self before we branch stable from rawhide. F24 Final release (GA) in this option is then 2016-05-31.
The second option is the schedule with 3 weeks slip. This has the "Change submission deadline" on 2016-02-02. Mass rebuild can start one day after this deadline and we have three weeks then till branching stable from rawhide. F24 Final release (GA) in this option is then 2016-06-07.
Here are the proposed schedules:
||||== The current schedule ==|| || 2015-07-14 ||Branch Fedora 23 from Rawhide (Rawhide becomes future F24)|| || 2015-10-27 ||Fedora 23 Release|| || 2016-01-12 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || not scheduled yet ||Side Tag Builds Deadline|| || not required ||Mass Rebuild|| || 2016-02-02 ||Branch Fedora 24 from Rawhide (Rawhide becomes future F25)|| || 2016-02-16 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-03-01 ||Alpha Release|| || 2016-03-15 ||Software Translation Deadline|| || 2016-03-29 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-04-12 ||Beta Release|| || 2016-05-03 ||Final Freeze|| || 2016-05-17 ||Fedora 24 Final Release (GA)|| [[BR]] ||||== Slip of GA for 2 weeks ==|| || 2015-07-14 ||Branch Fedora 23 from Rawhide (Rawhide becomes future F24)|| || 2015-10-27 ||Fedora 23 Release|| || 2016-01-26 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || not scheduled yet ||Side Tag Builds Deadline|| || 2016-02-02 ||Mass Rebuild|| || 2016-02-16 ||Branch Fedora 24 from Rawhide (Rawhide becomes future F25)|| || 2016-03-01 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-03-15 ||Alpha Release|| || 2016-03-29 ||Software Translation Deadline|| || 2016-04-12 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-04-26 ||Beta Release|| || 2016-05-17 ||Final Freeze|| || 2016-05-31 ||Fedora 24 Final Release (GA)|| [[BR]] ||||== Slip of GA for 3 weeks ==|| || 2015-07-14 ||Branch Fedora 23 from Rawhide (Rawhide becomes future F24)|| || 2015-10-27 ||Fedora 23 Release|| || 2016-02-02 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || not scheduled yet ||Side Tag Builds Deadline|| || 2016-02-03 ||Mass Rebuild|| || 2016-02-23 ||Branch Fedora 24 from Rawhide (Rawhide becomes future F25)|| || 2016-03-08 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-03-22 ||Alpha Release|| || 2016-04-05 ||Software Translation Deadline|| || 2016-04-19 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-05-03 ||Beta Release|| || 2016-05-24 ||Final Freeze|| || 2016-06-07 ||Fedora 24 Final Release (GA)||
[[BR]]
I will not be able to attend the FESCo meeting. Please let me know in advance if there is a need for another schedule draft, if the two proposed options do not fit for any reason.
If we do this, and still aim for mid-October for Fedora 25, what does the F25 schedule look like?
Just shift the F25 schedule as a whole? From a QA perspective, shortening the release schedule would likely lead to more late night scramble testing.
If we do not allow 3 weeks for the mass rebuild we will end up with a week for people to do their build fixes for FTBFS before they have to do builds twice, once for rawhide and once for f24. it will cause extra load.
Replying to [comment:4 jdulaney]:
On the other hand, if target November, we end up in Thanksgiving, and then end-of-year vacations and Christmas, and before you know it we have the Fedora 18 schedule all over again (Hello January!), and that makes the next release either have the same 4 month short cycle to get back on track.
The alternative is to have the release date wander around the calendar, and that makes planning hard for so many things.
The current proposed schedule is for an 8-month cycle, and from last year as well, it looks like there are a lot of things which naturally push towards that and a June release. Maybe we should just plan for a regular 8 month / 4 month cycle (with the second having smaller changes and no mass rebuilds).
Replying to [comment:5 ausil]:
I think this is fine; I mean, if we skip the mass rebuild just for that reason, then people are still going to have FTBFS packages, it's just that they don't know that they are failing.
The mass rebuild doesn't somehow magically make more packages fail. It just points out the failures.
While I'm not a big fan of last-minute changes to the schedule, I'll vote +1 to the schedule change based on the recommendation of the Release Engineering team.
Also... while we'll have to work out the precise schedule, I think we'd shorten the initial development time and keep the alpha/beta/final QA periods the same or close to it. That means big features which can't fit the development target will be deferred until F26 (May 2017).
I had a discussion with the GCC folks today.
tl;dr: They cannot be ready for a mass-rebuild before February 2nd. They have high (but not perfect) confidence in being ready on February 2nd.
It sounds like there is a higher-than-usual number of late bugs that they are still working on, however they have agreed to shift priority towards addressing exclusively those that might impact a mass-rebuild (such as code-generation) for the remaining time until February.
I will be meeting with the GCC team regularly (not less frequently than once per week) from now until the mass-rebuild to keep up to date on their status. It is important to note that they cannot fully guarantee February 2nd; if a serious issue comes up, we may have to address that as well.
They have informed me that there are no major ABI changes in this release, unlike the one last year that changed the C++ ABI. So if we elect not to do a mass-rebuild and instead build only thgccose packages that naturally build from dist-git, they do not expect any particular surprises from the GCC side. The glibc side on the other hand may have some fallout from the {{{strlcat()}}}/{{{strlcpy()}}} patches, so we need to be aware of that.
The GCC team is aware that the schedule slip affects not only Fedora 24 but also Fedora 25 and has acknowledged that if they cannot hit the February 2nd date, it may be necessary to skip the mass-rebuild and instead perform it in Rawhide/F25 as soon after the branch as humanly possible (so that if the rebuild discovers issues, they can be fixed and backported to F24 before its release).
We already agreed at last week's meeting that we would slip the mass-rebuild to February 2nd if GCC required it, but I'm reopening the ticket so we can discuss our contingency plans if needed.
The schedule has been pushed out for 1 more week: [https://fedoraproject.org/wiki/Releases/24/Schedule#Key_Milestones The F24 schedule on wiki] [https://fedorapeople.org/groups/schedule/f-24/f-24-all-tasks.html Full schedule] * [https://lists.fedoraproject.org/archives/list/devel-announce%40lists.fedoraproject.org/thread/TC3NDJ2UVQU6ABAAN2HH2FG7Y6H2NZS5/ Announcement of the 1 more week slip]
To answer the question about F25 planning Matthew has opened, I made the following three schedules, each having different set of pros and cons. [[br]][[br]]
Option 1: Pros: enough time for development Cons: * GA in the middle of January 2017 * not enough engineering power between Beta and Final due to Christmas * Note: Thanksgiving Day 2016: November 24th
||||= Option 1 - long development (96d) =|| || 2016-06-07 ||Fedora 24 Release|| || 2016-09-13 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || 2016-09-13 ||Mass Rebuild|| || 2016-10-04 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)|| || 2016-10-18 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-11-01 ||Alpha Release|| || 2016-11-15 ||Software Translation Deadline|| || 2016-11-29 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-12-13 ||Beta Release|| || 2017-01-03 ||Final Freeze|| || 2017-01-17 ||Fedora 25 Final Release (GA)||
[[br]][[br]]
Option 2: Pros: GA at the beginning of November 2016 Cons: short development
||||= Option 2 - short development (46d) =|| || 2016-06-07 ||Fedora 24 Release|| || 2016-07-05 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || 2016-07-05 ||Mass Rebuild|| || 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)|| || 2016-08-09 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-08-23 ||Alpha Release|| || 2016-09-06 ||Software Translation Deadline|| || 2016-09-20 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-10-04 ||Beta Release|| || 2016-10-25 ||Final Freeze|| || 2016-11-08 ||Fedora 25 Final Release (GA)||
Option 3: Pros / Cons: GA at the beginning of December 2016 Pros / Cons: reasonable long development * Cons: Thanksgiving Day on November 24th collides with Final Freeze
||||= Option 3 - trade-off (61d) =|| || 2016-06-07 ||Fedora 24 Release|| || 2016-08-02 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) || || 2016-08-02 ||Mass Rebuild|| || 2016-08-23 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)|| || 2016-09-06 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point|| || 2016-09-20 ||Alpha Release|| || 2016-10-04 ||Software Translation Deadline|| || 2016-10-18 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline|| || 2016-11-01 ||Beta Release|| || 2016-11-22 ||Final Freeze|| || 2016-12-06 ||Fedora 25 Final Release (GA)||
[[br]]
My personal preference is the "Option 3" as we have a room to slip for one week and still have the release done prior Christmas 2016. We can also aim to a schedule having GA between Option 2 and Option 3.
I really don't want to slip into December. Even the "short" option 2 aims for beyond the October nominal target even without further adjustment.
F24 development can really start with the branch in February. What if we announce now that the change submission deadline will be the day of F23 release due to the short cycle?
I guess failing that, I think a variant of "long" Option 1 makes the most sense, but probably with even more time to allow for QA and Rel Eng without work over Christmas. But then, we're back to the same puzzle of a short cycle for F25, leading me back around to the idea of making F23, F24, and F25 all long cycles, with F24 in February and F25 in October — and then hopefully F26 back to May and happily ever after with six month cycles from then on.
I think that option 2 is a reasonable choice. F25 features can land as soon as we branch f24. do not forget that we(FESCo) said there would be no mass rebuild in f25 due to the tightness of schedule. f26 on needs to have a mass rebuild built in.
I think most Workstation folks (certainly myself) would strongly prefer option 2. We want to release about a month behind GNOME (late April or early May, late October or early November) and that's pretty much the only relevant consideration for us besides blocker bugs. The Workstation development schedule bears no relation to the Fedora release schedule as all work is done upstream, so "more time for development" is not a concern; we are done working on F25 and get started on F26 in September regardless of whether you plan F25 for October or January, because the end of September is when GNOME releases. If option 1 is seriously considered, it might be time to start talking more seriously about product-specific release schedules.
I wish the mass rebuild had been delayed until F25... oh well.
My preference goes to option 2 (consider this as +1)
Replying to [comment:15 mattdm]:
I really don't want to slip into December. Even the "short" option 2 aims for beyond the October nominal target even without further adjustment. F24 development can really start with the branch in February. What if we announce now that the change submission deadline will be the day of F23 release due to the short cycle?
These version numbers don't make sense. I'm going to assume you meant F25 and F24, respectively.
I prefer option 2.
Option 2 is better
Replying to [comment:19 jwboyer]:
Replying to [comment:15 mattdm]: I really don't want to slip into December. Even the "short" option 2 aims for beyond the October nominal target even without further adjustment. F24 development can really start with the branch in February. What if we announce now that the change submission deadline will be the day of F23 release due to the short cycle? These version numbers don't make sense. I'm going to assume you meant F25 and F24, respectively.
Err, yes. Too many numbers. Need to go back to distinctive release names. :)
Replying to [comment:22 mattdm]:
Replying to [comment:19 jwboyer]: Replying to [comment:15 mattdm]: I really don't want to slip into December. Even the "short" option 2 aims for beyond the October nominal target even without further adjustment. F24 development can really start with the branch in February. What if we announce now that the change submission deadline will be the day of F23 release due to the short cycle? These version numbers don't make sense. I'm going to assume you meant F25 and F24, respectively. Err, yes. Too many numbers. Need to go back to distinctive release names. :)
Those were terrible too. Let's just stop doing releases entirely. ROLL ON RAWHIDE.
AGREED: We're going for option 2 as stated in https://fedorahosted.org/fesco/ticket/1519 , with final F25 release on 2016-11-08 (+7,0,0) (kalev, 17:13:38)
https://meetbot.fedoraproject.org/fedora-meeting/2016-01-22/fesco.2016-01-22-17.02.html
Replying to [comment:11 sgallagh]:
I had a discussion with the GCC folks today. tl;dr: They cannot be ready for a mass-rebuild before February 2nd. They have high (but not perfect) confidence in being ready on February 2nd.
In the toolchain team meeting yesterday, we had a Go/No-Go about the February 2nd mass-rebuild. The GCC and glibc teams are confident at this time that they will be ready for a mass-rebuild to kick off in Fedora on February 2nd. There's a slight open question as to when exactly on the 2nd the latest gcc fixes will be in the buildroot, since ARM builds can be unpredictably long, but I have asked that they do their best to have it ready as close to 00:01 EST on Feb 2nd as possible.
The glibc team has opted to defer the {{{strlcat()}}}/{{{strlcpy()}}} patches due to upstream concerns with them, so this will not be adding a layer of uncertainty to the rebuild after all.
In short: we're looking good to go on Groundhog's Day.
Login to comment on this ticket.