The upstream glibc 2.25 release is planned for February 1st 2017. Therefore a 2017-02-14 change checkpoint would allow Fedora to adopt any new ABIs provided by glibc 2.25. That works perfectly for the glibc team. Thank you for reaching out to us.

Replying to [comment:1 codonell]:

The upstream glibc 2.25 release is planned for February 1st 2017. Therefore a 2017-02-14 change checkpoint would allow Fedora to adopt any new ABIs provided by glibc 2.25. That works perfectly for the glibc team. Thank you for reaching out to us.

The scheduled mass rebuild is 2 weeks before your planned release. Do you know if anything in the 2.25 release would necessitate/benefit a mass rebuild and whether it will be in the code base by the scheduled date?

I'm curious if the rebuild needs to be pushed out for either glibc or gcc.

Replying to [comment:2 jwboyer]:

Replying to [comment:1 codonell]:

The upstream glibc 2.25 release is planned for February 1st 2017. Therefore a 2017-02-14 change checkpoint would allow Fedora to adopt any new ABIs provided by glibc 2.25. That works perfectly for the glibc team. Thank you for reaching out to us.

The scheduled mass rebuild is 2 weeks before your planned release. Do you know if anything in the 2.25 release would necessitate/benefit a mass rebuild and whether it will be in the code base by the scheduled date?

I'm curious if the rebuild needs to be pushed out for either glibc or gcc.

Jakub Jelinek or Jeff Law will have to comment on gcc.

On January 1st 2017 glibc enters a freeze period where the ABI should not be changed because of the upcoming Feburary 1st 2017 release. The 30-day frozen ABI and API makes testing easier and prepares for the release. Therefore what eventually ends up being the 2.25 release will have the same ABI as what we have on January 13th when the mass rebuild is scheduled (if needed).

That's the normal operating procedures. It could be the case that a late breaking issue before the release causes the ABI to be changed. This would immediately cause the Fedora glibc team to reach out to the Fedora community to discuss a late mass rebuild to avoid ABI issue with the final release.

Lastly, the only guarantee we have from upstream is that on Feburary 1st 2017 when the release comes out, that ABI is now set in stone and will be supported e.g. new syscalls, new headers, new ABIs etc.

In summary:
- A mass rebuild 2 weeks before the glibc release is relatively low-risk and should match the final release ABI/API. Moving the mass rebuild any earlier increases the risk of a second late mass rebuild being required.
- The Fedora glibc team will notify the Fedora community if a mass rebuild is needed for glibc as we approach the mass rebuild deadline. At present we can't comment on the need for an F26 mass rebuild.
- The Fedora glibc team will notify the Fedora community if an ABI/API breakage occurs in the two week window between mass rebuild and glibc upstream release that requires a late mass rebuild. - At any point in time choosing to go with an older glibc may still require a mass rebuild to purge new ABI/API that appeared in Rawhide. Thus there is almost no gain in going 'backwards' to an older glibc.

Does that answer your questions?

Replying to [comment:3 codonell]:

Does that answer your questions?

Yes, in fantastic and complete detail. I was mostly asking to see if we needed to push the rebuild date out, but it doesn't seem to be the case.

I think the mass rebuild on Jan 13 is way too early from the GCC side. At that point we will likely to have just transitioned to regression bugfixing only for GCC 7-- but it's probably too early to have a high degree of confidence in the code generator. The risk here is that if we have a serious codegen or ABI bug and you've done the mass rebuild, then you'd end up having to do another mass rebuild to address those issues. Clearly that's something we'd want to avoid.

For f24 we couldn't reasonably green-light the mass rebuild until Jan 25, and that was after a bit of of a firedrill on the GCC side.

For f22 we were doing internal mass rebuilds Feb.

What makes a lot more sense is to have a go/no go by say Jan 30.

Mass rebuilds need 3 weeks at least in the schedule, I would be more comfortable with branching on Feb 7th. We (releng) want to drop Alpha, but that is dependent on QA having sufficient automated testing and having controls in place to ensure that Rawhide is always up to a Alpha standard, then we can do only Beta and Final. It is not at all dependent on pungi or any changes there. I have talked to QA about the idea and they do not object.

Change submission deadline should probably be Jan 7th or perhaps befor christmas. Fesco needs to review and approve them all with the deadline on Jan 10th we could not do them all until the week after for anything coming in last minute. Fesco meeting time is up for rewiev again as we have a new FESCo.

Replying to [comment:6 ausil]:

Change submission deadline should probably be Jan 7th or perhaps befor christmas. Fesco needs to review and approve them all with the deadline on Jan 10th we could not do them all until the week after for anything coming in last minute. Fesco meeting time is up for rewiev again as we have a new FESCo.

At the last meeting, we discussed it with rathann and he was fine with the existing time, so unless the time has become problematic for the pre-existing membership, it will remain the same as the last six months.

The proposed schedule works well for Workstation, thanks. If it doesn't leave enough time for GCC, then I would greatly prefer to delay the GCC upgrade until F27. (We were disappointed with the significant delay in the F25 schedule.)

In the FESCo meeting, I asked a question that didn't have a ready answer from QA/Rel-Eng (neither of whom was represented there). Right now, there is a two-week gap that occurs between Branch and Alpha Freeze which is kind of an oddball state (no Bodhi activation, so it's really just Rawhide except that you have to build in twice as many places).

My main question here is whether we could keep the rest of the schedule the same except for two key changes:
* Move the mass-rebuild date to January 25th so that we are building with a stable GCC
* Eliminate the two-week gap between Branch and Alpha Freeze by combining those two events.

This would of course need to be highly publicized, since it would be a change from how we have done things historically - Kevin mentioned at the meeting that "there's always some churn after branching... people wake up and realize they forgot to push something, etc." but that's always been true of Alpha Freeze as well. Personally, I think it would actually be beneficial that after Branching it would be necessary to get a Freeze Exception in order to make late changes.

Such a change might have QA and Releng impact that I'm not immediately seeing, so I've CCed some key players from those groups to get their opinion on my proposed schedule (reproduced here for ease of discussion):

||||== sgallagh revised schedule ==||
|| 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)||
|| 2016-11-08 ||Fedora 25 Release||
|| 2017-01-10 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) ||
|| 2017-01-25 ||Mass Rebuild||
|| 2017-02-14 ||- Branch Fedora 26 from Rawhide (Rawhide becomes future F27)[[BR]]- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point||
|| 2017-02-28 ||Alpha Release||
|| 2017-03-14 ||Software Translation Deadline||
|| 2017-03-28 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline||
|| 2017-04-11 ||Beta Release||
|| 2017-05-02 ||Final Freeze||
|| 2017-05-16 ||Fedora 26 Final Release (GA)||
[[BR]]

For those ~30% of Fedora packagers who are working at Red Hat, having deadlines around beginning of February is not helpful due to both FOSDEM and devconf.cz conferences being traditionally around the same dates. devconf.cz is January 27-29th 2017, FOSDEM is February 4-5th 2017 next year. It means very little packaging work could be done if somebody attends both events and also has traditional team meetings in Brno in between the events.

We used to have branched and freeze/bodhi activation atthe same point in time. it caused a lot of issues. However we have since started making rawhide composes all the time. It came about that when we brnached things would be broken and it gave us time to get things working before freeze, rather than dealingw ith fixing issues while frozen. however there is really too much work to be done on one day at branching and freezing and activating bodhi. I would be okay with reducing it to a week gap. but not having things all fall on the one day.

Could we reduce the mass-rebuild time from three weeks to two? Then the schedule would look like:

||||== sgallagh revised schedule 2 ==||
|| 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)||
|| 2016-11-08 ||Fedora 25 Release||
|| 2017-01-10 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) ||
|| 2017-01-25 ||Mass Rebuild||
|| 2017-02-07 || Branch Fedora 26 from Rawhide (Rawhide becomes future F27)
|| 2017-02-14 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point||
|| 2017-02-28 ||Alpha Release||
|| 2017-03-14 ||Software Translation Deadline||
|| 2017-03-28 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline||
|| 2017-04-11 ||Beta Release||
|| 2017-05-02 ||Final Freeze||
|| 2017-05-16 ||Fedora 26 Final Release (GA)||
[[BR]]

My primary feedback here is that a Memorial-Day-week release is a death sentence to all PR. That means that this schedule can slip one week and still be in May; a two-week slip needs to be a three-week slip until June 6.

I'd like to see dates in the schedule for the aarch64 merge. We cannot do a mass rebuild before that is merged with the main koji. I also think that we shouldn't reduce the mass rebuild time to 2 weeks because of this. Having the 3 planned weeks seems like a more realistic plan.

Based on discussions in today's FESCo meeting, the new proposal is:

||||== FESCo revised schedule 3 ==||
|| 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)||
|| 2016-11-08 ||Fedora 25 Release||
|| 2017-01-10 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) ||
|| 2017-02-01 ||Mass Rebuild||
|| 2017-02-21 || Branch Fedora 26 from Rawhide (Rawhide becomes future F27)
|| 2017-02-28 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point||
|| 2017-03-14 ||Alpha Release||
|| 2017-04-11 ||Software Translation Deadline||
|| 2017-04-25 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline||
|| 2017-05-09 ||Beta Release||
|| 2017-05-23 ||Final Freeze||
|| 2017-06-06 ||Fedora 26 Final Release (GA)||
[[BR]]

If we move up bodhi enablement to just a few days after branching it might look like:

||||== FESCo revised schedule 4 ==||
|| 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)||
|| 2016-11-08 ||Fedora 25 Release||
|| 2017-01-10 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) ||
|| 2017-02-01 ||Mass Rebuild||
|| 2017-02-21 || Branch Fedora 26 from Rawhide (Rawhide becomes future F27)
|| 2017-02-23 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point||
|| 2017-03-07 ||Alpha Release||
|| 2017-04-04 ||Software Translation Deadline||
|| 2017-04-18 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline||
|| 2017-05-02 ||Beta Release||
|| 2017-05-16 ||Final Freeze||
|| 2017-05-30 ||Fedora 26 Final Release (GA)||
[[BR]]

Both of those are effectively the same in the end, if we slip to avoid conflicting with Memorial Day and losing all press coverage.

I'd like us to get to a schedule where we're repeatedly targeting actual-October for the (northern hemisphere) fall release, so that we can get the spring release change submissions all in before Christmas (aim to have them due early/mid December), so the spring release can be more consistently April/May than May/June.

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

Why a different release schedule overall if we can't hit those months? Because we seem to end up entangled in holidays every time otherwise.

Replying to [comment:18 mattdm]:

Both of those are effectively the same in the end, if we slip to avoid conflicting with Memorial Day and losing all press coverage.

I'd like us to get to a schedule where we're repeatedly targeting actual-October for the (northern hemisphere) fall release, so that we can get the spring release change submissions all in before Christmas (aim to have them due early/mid December), so the spring release can be more consistently April/May than May/June.

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

That seems like a great discussion to have in a broader ticket, not specific to f26.

To add to that, if Modularity really becomes a thing in Fedora, then we need to really think about a release from that perspective anyway. I'm not convinced a 6mo release cadence for ALL THE THINGS means as much if modules allow some of the things to ship on top of a base runtime that releases e.g. yearly.

So yes, let's have that conversation. Just... not in this ticket?

Replying to [comment:18 mattdm]:

Both of those are effectively the same in the end, if we slip to avoid conflicting with Memorial Day and losing all press coverage.

I'd like us to get to a schedule where we're repeatedly targeting actual-October for the (northern hemisphere) fall release, so that we can get the spring release change submissions all in before Christmas (aim to have them due early/mid December), so the spring release can be more consistently April/May than May/June.

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

I really don't think that's possible with GCC not hitting a functionally-stable state until late January every year. That's the only time we can do a mass-rebuild, unless we want to break our long-standing relationship with that project and cease to be their upstream delivery vehicle (in which case we would do the mass-rebuild on the last stable release rather than the development release).

The problem this introduces is that GCC upstream kind of relies on Fedora using the development releases as a way to find issues before it goes stable; it might just end up moving the goalposts.

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

I really don't think that's possible with GCC not hitting a functionally-stable state until late January every year. That's the only time we can do a mass-rebuild, unless we want to break our long-standing relationship with that project and cease to be their upstream delivery vehicle (in which case we would do the mass-rebuild on the last stable release rather than the development release).

I don't think we should be deciding how we do releases based on upstream schedules. What about GNOME schedule for workstation? glibc/gcc/systemd/docker/golang blah blah blah.... I think we need to pick a schedule that works for Fedora. It's really only moving it by a few weeks but it we try and make every upstream happy we'll never deliver anything when it suits us.

Replying to [comment:22 pbrobinson]:

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

I really don't think that's possible with GCC not hitting a functionally-stable state until late January every year. That's the only time we can do a mass-rebuild, unless we want to break our long-standing relationship with that project and cease to be their upstream delivery vehicle (in which case we would do the mass-rebuild on the last stable release rather than the development release).

I don't think we should be deciding how we do releases based on upstream schedules. What about GNOME schedule for workstation? glibc/gcc/systemd/docker/golang blah blah blah.... I think we need to pick a schedule that works for Fedora. It's really only moving it by a few weeks but it we try and make every upstream happy we'll never deliver anything when it suits us.

It's not about "every upstream", it's specifically about GCC. Fedora and GCC have long had a symbiotic relationship: Fedora is the distribution that GCC uses in order to get the release from "feature-complete" to "stable". What happens is that GCC gets it to a point where it is believed to be free of code-generation errors, then Fedora does its mass-rebuild using this late prerelease. Through the Alpha and Beta process, GCC fixes any remaining bugs revealed by Fedora (which is a huge testbed of real-world uses of GCC) and then releases a fully-stable GCC in time for Fedora Final Freeze.

If we opted to move Fedora schedules earlier, then we would by necessity have to switch to doing mass-rebuilds in the autumn with older versions of GCC. This also means that GCC would have to switch to another distribution for its comprehensive real-world testing (or else ship potentially risky compilers), which is an outcome I wouldn't like to see.

Would GCC be able to adjust their release schedule a bit so that they release, let's say two weeks earlier?

Replying to [comment:23 sgallagh]:

Replying to [comment:22 pbrobinson]:

If that's not possible, I think we should seriously reconsider a different release schedule plan overall.

I really don't think that's possible with GCC not hitting a functionally-stable state until late January every year. That's the only time we can do a mass-rebuild, unless we want to break our long-standing relationship with that project and cease to be their upstream delivery vehicle (in which case we would do the mass-rebuild on the last stable release rather than the development release).

I don't think we should be deciding how we do releases based on upstream schedules. What about GNOME schedule for workstation? glibc/gcc/systemd/docker/golang blah blah blah.... I think we need to pick a schedule that works for Fedora. It's really only moving it by a few weeks but it we try and make every upstream happy we'll never deliver anything when it suits us.

It's not about "every upstream", it's specifically about GCC. Fedora and GCC have long had a symbiotic relationship: Fedora is the distribution that GCC uses in order to get the release from "feature-complete" to "stable". What happens is that GCC gets it to a point where it is believed to be free of code-generation errors, then Fedora does its mass-rebuild using this late prerelease. Through the Alpha and Beta process, GCC fixes any remaining bugs revealed by Fedora (which is a huge testbed of real-world uses of GCC) and then releases a fully-stable GCC in time for Fedora Final Freeze.

If we opted to move Fedora schedules earlier, then we would by necessity have to switch to doing mass-rebuilds in the autumn with older versions of GCC. This also means that GCC would have to switch to another distribution for its comprehensive real-world testing (or else ship potentially risky compilers), which is an outcome I wouldn't like to see.

For the record this is exactly the same situation for glibc, which has been historically tied tightly to the Fedora release schedule to provide new features for Fedora and continuous integration into Rawhide. However, for glibc we have a stabilizing time-boxed release which is more stable and amenable to adjusting to match various distribution requirements.

Replying to [comment:16 sgallagh]:

Based on discussions in today's FESCo meeting, the new proposal is:

||||== FESCo revised schedule 3 ==||
|| 2016-07-26 ||Branch Fedora 25 from Rawhide (Rawhide becomes future F26)||
|| 2016-11-08 ||Fedora 25 Release||
|| 2017-01-10 ||Change Checkpoint: Proposal submission deadline (System Wide Changes) ||
|| 2017-02-01 ||Mass Rebuild||
|| 2017-02-21 || Branch Fedora 26 from Rawhide (Rawhide becomes future F27)
|| 2017-02-28 ||- Alpha Freeze[[BR]]- Change Checkpoint: Completion deadline (testable)[[BR]]- Bodhi activation point||
|| 2017-03-14 ||Alpha Release||
|| 2017-04-11 ||Software Translation Deadline||
|| 2017-04-25 ||- Beta Freeze[[BR]]- Change Checkpoint: 100% Code Complete Deadline||
|| 2017-05-09 ||Beta Release||
|| 2017-05-23 ||Final Freeze||
|| 2017-06-06 ||Fedora 26 Final Release (GA)||

The above schedule was accepted during last meeting:

• ACCEPTED: Fedora 26 schedule proposal
  https://fedorahosted.org/fesco/ticket/1609#comment:16 (+:9 -:0 0:0)

The schedule has been published here: https://fedoraproject.org/wiki/Releases/26/Schedule

I made some minor Changes into the [https://fedoraproject.org/wiki/Releases/26/Schedule#Key_Milestones published schedule] and I would like to ask FESCo to approve these, just to make sure there is no misunderstanding.

The Changes are:
* Task ''"Change Checkpoint: Proposal submission deadline (System Wide Changes)"'' has split into two deadlines. One deadline is for Changes requiring mass rebuild and the second one is for all System Wide Changes. This is based on [https://fedorahosted.org/fesco/ticket/1567 ticket 1567].

• Task ''"Mass rebuild"'' has moved from 2017-02-01 to 2017-02-03 as mass rebuilds are always starting on Friday.

Thanks, Jan

I don't mind the change deadline change, but is there a good reason to shorten the mass rebuild by two days, just because we've started on Fridays in the past?

Replying to [comment:29 sgallagh]:

I don't mind the change deadline change, but is there a good reason to shorten the mass rebuild by two days, just because we've started on Fridays in the past?

It is a requirement coming from Release Engineering (to start on Friday). I hope ausil might comment on this better than me.

We discussed this in today's meeting and the consensus was to start the mass rebuild on Wednesday to leave two more days for the cleanup.

AGREED: Keep the originally-approved Wednesday mass-rebuild (+1:7, 0:0, -1:0) (kalev, 16:10:22)

The decision is now reflected on the [https://fedoraproject.org/wiki/Releases/26/Schedule#Key_Milestones F26 Schedule page].

Detailed schedules on [https://fedorapeople.org/groups/schedule/f-26/f-26-key-tasks.html FedoraPeople.org] will follow on Monday, once I have full access to the schedule build environment.