#1349 Fedora 22 scheduling strategy (and beyond)
Closed None Opened 9 years ago by mattdm.

At Flock, we talked about the idea of moving to a calendar-based scheduling strategy — not making hard calendar deadlines, but simply setting the release date target based on the same two dates every year rather than working from the previous release.

This would make it easier to plan around conferences and holidays in advance, and make marketing and PR planning easier, and make it easier to work on features targeted at the ''next'' release. Plus, it will make end of life and upgrade scheduling easier for end-users to plan for.

We'd still stick to our general approach of being release-criteria based; the traditional one-week slips would still move everything back. In the event that one release has a lot of slip, that means the next release would be a shorter cycle — and we'd know that. This could be a good thing in itself, as a long, painful release can probably benefit from the next release intentionally having fewer changes. (In fact, it might even naturally lead to a cadence of alternating feature releases and stabilization releases, although I don't think we should ''force'' that.)

At Flock, it seemed like the traditional Mother's Day / Halloween targets are generally good. That's 5 months and 7 months instead of 6/6; the alternative I guess would be October/April. I think 5/7 is okay, though, and therefore propose '''the second Tuesdays in May and October''' as standing targets. (I'm not obstinately attached to that, though.) Whatever the date, everything else would then fall out of the [http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle existing release scheduling process], except that we could have tentative dates pre-attached to everything.

I know we're still deep in F21 and so this is a little ways off, but since rawhide ''is'' already F22 and we have a few F22 features in progress, so I think it'd be nice to start getting that end-date in mind, especially if it's going to be extra short or long. (And, I would like to establish whether if a catastrophe happens and F21 is pushed back to January 2015, we want to have a short cycle with a May (or even April) release, or not.)


Huh. This is apparently ''already'' codified in http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle. Who knew? :) The proposal is, really, to commit to sticking to this, with the possible-short-cycle implication.

I fully support this plan. I think it makes a lot of sense to have fixed release dates; it's really hard to plan feature development without knowing what dates to roughly target.

The codified dates from http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle (the Tuesday before May 1st or October 31st) sound good to me too.

therefore propose the second Tuesdays in May and October as standing targets

I would be a little bit hesitant to go with the 5/7 month split, but 5.5/6.5 months that the Mother's day (2nd Tuesday of May) and Halloween (31st of October) releases entail would be fine too.

Replying to [comment:3 kalev]:

I would be a little bit hesitant to go with the 5/7 month split, but 5.5/6.5 months that the Mother's day (2nd Tuesday of May) and Halloween (31st of October) releases entail would be fine too.

It's gonna be a Tuesday, so either "Last Tuesday in October" or "4th Tuesday in October".

My main concern is that (as we see this time around), if we slip out of October, we end up running into Thanksgiving, which puts us in December, and because of the holidays then at high risk of going all the way into January. So if we want to keep it more 6/6, I think October / April is better. (Or even September / March.)

So yeah, doing the dates and what we said we were supposed to do for as schedule since FC1 all sounds great. But now you're already worrying about slips and holidays, which is what we already do today.

What is new here? What are we actually trying to accomplish with this ticket? What do we need to change in order to stick with the dates and avoid slips?

Concretely, I'm proposing that slips in one release should not affect the scheduling of the next release, and that accepted features should be based on the regular target dates, rather than setting the schedule based on features.

Replying to [comment:6 mattdm]:

Concretely, I'm proposing that slips in one release should not affect the scheduling of the next release, and that accepted features should be based on the regular target dates, rather than setting the schedule based on features.

That works with the following assumptions:

  • Changes for the next release are decided well before development of that release starts (not exactly true)
  • We have enough of a contributor base to drive development of two releases simultaneously (skeptical)
  • Continued slips of one release don't impact features already accepted for the next release (not exactly true)

So I like the concrete proposal, but it's not like it's a new proposal. We've said similar things many times and have not be able to accomplish this. Maybe we should back up and figure out why.

Replying to [comment:2 mattdm]:

Huh. This is apparently ''already'' codified in http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle. Who knew? :) The proposal is, really, to commit to sticking to this, with the possible-short-cycle implication.

As I knew it this proposal was already the policy. We got off track because F18 slipped so far people didnt want a really really shortened f19 so it was bumped out.

Replying to [comment:7 jwboyer]:

Replying to [comment:6 mattdm]:

Concretely, I'm proposing that slips in one release should not affect the scheduling of the next release, and that accepted features should be based on the regular target dates, rather than setting the schedule based on features.

We have had a time-based rather than a feature-based schedule for some time now. Until F21, basically.

In fact, what you proposed has been the goal (at least through John Poelstra and myself as the schedule wranglers) since ... well, a really long time now. :)

That works with the following assumptions:

  • Changes for the next release are decided well before development of that release starts (not exactly true)
  • We have enough of a contributor base to drive development of two releases simultaneously (skeptical)

Do you not consider the release currently in process + whatever is going on in rawhide as basically 2 releases simultaneously?

  • Continued slips of one release don't impact features already accepted for the next release (not exactly true)

So I like the concrete proposal, but it's not like it's a new proposal. We've said similar things many times and have not be able to accomplish this. Maybe we should back up and figure out why.

F17 was a month late (those reasons I don't recollect); F18 was the intro of re-worked Anaconda. Normally we would have gone with the strategy of "sorry, feature isn't done, but no big deal, because the next release is only 6 months away" -- but (and anyone else is free to correct me here) trying to revert to the previous version was pretty much going to take as much time, if not more, than trying to work out the bugs. And then we ran into the holiday season, which pretty much stalled things (which we normally had avoided because the devel phase ran over the holidays).

F19, F20 both ran approximately 6 months-ish; just not according to the regular shipping schedule plan.

F21.... well, this was the big reboot. And we expected it to take longer - and I think some thought as well that it would give us a chance to get back to the May Day / Halloween schedule of harmony. I don't think I would have originally described the extra time as time to develop features -- probably more around developing the underlying infrastructure to develop/test/build/ship Fedora in a new way (3 of them, in fact). Not sure it's worked out that way though. :)

The key problem here may be that my time on FESCo has apparently mostly coincided with an out-of-the-ordinary process. I'm certainly not trying to make a big fuss where one isn't necessary. F19 and F20's "not according to the regular shipping schedule plan" is primarily what I'm thinking about, I guess.

Replying to [comment:10 mattdm]:

The key problem here may be that my time on FESCo has apparently mostly coincided with an out-of-the-ordinary process. I'm certainly not trying to make a big fuss where one isn't necessary. F19 and F20's "not according to the regular shipping schedule plan" is primarily what I'm thinking about, I guess.

I think that may indeed be skewing things. Fedora has always slipped. Always. I don't think there's a release at least since the core+extra merge where we haven't slipped. However, as Robyn pointed out, we normally aimed for the usual dates until about F18.

So fast-forward. F21 is looking December-ish. Possibly post New Year. Assuming we can reign things in for F22, Mothers day would be an OK date but it would be a brief development window.

Replying to [comment:9 rbergero]:

Replying to [comment:7 jwboyer]:

That works with the following assumptions:

  • Changes for the next release are decided well before development of that release starts (not exactly true)
  • We have enough of a contributor base to drive development of two releases simultaneously (skeptical)

Do you not consider the release currently in process + whatever is going on in rawhide as basically 2 releases simultaneously?

Frankly, no. Rawhide as it exists today is not a "release". It's a dumping ground. My desire for it not to be might be tainting my point of view here a bit and adding a criteria I would like to see and not one that is really necessary.

Replying to [comment:12 jwboyer]:

Frankly, no. Rawhide as it exists today is not a "release". It's a dumping ground. My desire for it not to be might be tainting my point of view here a bit and adding a criteria I would like to see and not one that is really necessary.

Other than having resources for development of two simultaneous releases, there are two basic alternatives:

  1. develop on the branch and dump stuff kind of arbitrarily into rawhide at the same time, and also when there isn't a branch.
  2. develop in rawhide and make the branch for stabilization and release only — effectively, put it in freeze all the time it exists.

Obviously 1 is what we do, and also pretty obviously getting to 2 would be a big shift if we wanted to do it. (Maybe with a lot more automated integration testing and with the ability to have lightweight feature branches.)

Another specific proposal would be to simply add the reoccurring nominal-target spring and fall dates to a new column of the chart at:

http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Schedule_Methodology

For example, instead of:

||= '''Task / Milestone''' =||= '''Start Day (Tuesdays or Thursdays)'''=||= '''Length''' =||
|| Branch point || Tue: Alpha Freeze minus 2 weeks || n/a ||

we'd have something like:

||= '''Task / Milestone''' =||= '''Start Day''' =||= '''Length''' =||
|| Branch point || First Tuesday in February / July || 2 weeks ||

I think this might actually help, because currently when thinking about planning for the next release, even if the basic target is known, the actual dates for various milestones feel uncertain. Actually writing at least a month and week for each of them makes it more concrete, for me at least.

One thing that I just recalled: the F18 anaconda rewrite really highlighted how there are cases where setting date targets before you know what the content is makes for a terrible way to keep the schedule. As a result of that, pjones and several others (I think myself included) were pushing for dates set on content (Changes).

We can go back to setting dates first, and then being very strict about what seems doable in terms of Changes in that timeframe. An anaconda rewrite is possibly the worst-case example of something that is very hard to land in rawhide and actually get testing done properly, which is why the F18 branch took longer. However, being strict on the content within a release is going to require FESCo to be diligent and work with the maintainers to understand how long their Change is going to take, etc. It is also going to require that the maintainers buy into the target dates and process.

It's my theory, at least, that having dates for the full schedule early will make it easier for maintainers to buy in and make more realistic assessments.

I don't want to plan too much around the anaconda rewrite example, since it does seem to be a worst case. If a future feature that seems likely to be as disruptive comes up, we can explicitly plan for a special cycle.

The current scheduling methodology is bit different to what is written on the wiki page (sorry, I wanted to update it and then forgot - I'll do it after this ticket ;-) and it's actually compromise between time driven and feature driven releases. We just try to be clever, not to shoot ourselves to the knee.

For F19, we tried to be, as FESCo decided, more feature driven (as response to Anaconda issues). It did not work very well, because of no clear schedule given in advance. It caused quite a lot of troubles especially for QA and other teams to sync with schedule. It was just too hard to plan any longer activity. Final schedule was set pretty late but on the other hand, we had only one week slip.

For F20, we came with so called "No earlier than" scheduling. Initial draft still aims to around May/June, October/November. But at this time, we already talk about what's doable, what's not. There's note, that release is no early than specific date with disclaimer it may/will change. This gives all teams rough estimate how much time do they have. Then C(c)hanges steps into process. Based on what was proposed, release is delayed. We used it for F21 as many changes were "F21 would not be Fedora.Next" without xyz.

Also many times, we receive different requests from different groups. For F20, releng asked for two more weeks for example. There's general agreement within people doing releases, it's better to change schedule radically than slipping week by week and put more pressure on everyone. Sometimes it's even impossible to revert some changes and release on time - Robyn pointed out F18 and Anaconda situation.

Other thing is how we do QA, it currently conflicts with fully time driven release. And you know, some bugs are unpredictable and can cause major slips. But it was mentioned already and I prefer not to do compromises in terms of quality.

So yes, we can be almost strictly time driven again but it means, we have to be able to say "no". Really fulfill contingency deadlines for system wide changes etc. Be willing to have smaller releases if one release slips, even it could lead to very small release... It can slow down major changes (and you know how some folks are eager for changes) but gives more predictability.

Btw. we already do partially overlapping releases, next release schedule never fully follows GA of previous release (I think this ticket is about it).

Personally, I like current way of compromise between time and feature drive development. Smaller releases are more time driven, bigger releases as F21 are than more feature driven. It gives us quite a lot of flexibility and it's always after extensive discussion. It's good to talk about release! It's living organism :).

Fedora 21 Schedule ticket https://fedorahosted.org/fesco/ticket/1178
Fedora 20 Schedule ticket https://fedorahosted.org/fesco/ticket/1095
Fedora 19 Schedule ticket https://fedorahosted.org/fesco/ticket/966

edit: whoops, wrong ticket

Dropping meetings keyword until we actually discuss things more.

Apparently not much discussion happened after removing the meeting keyword. Putting it back as we should now seriously think about F22 scheduling.

I'll prepare draft for next week's FESCo meeting. As I don't see it on today's agenda, so we have real date to be discussed. Not sure I'll make it today, as I'm working on another schedule right now.

I mistakenly forgot to add this topic to the agenda and it seems the mistake was actually a good thing! :) So yes, please, prepare the draft for the next week meeting.

Looking on our possibilities, especially if we want to aim mid May, we don't have many options and it will be smaller release. Not actually that bad thing after huge F21... Change submission deadline will be soon but there are already a few changes in the queue I have to process, it's deadline for system wide changes mostly and even with Christmas in between, it's still almost two months. And development is already back to Rawhide.

  • Tue 2015-01-20 Change Proposals Submission Deadline
  • Tue 2015-02-10 Branch Fedora 22 from Rawhide/Change Proposals Freeze (Testable|Complete)
  • Tue 2015-02-24 Alpha Change Deadline
  • Tue 2015-03-10 Alpha Release Public Availability
  • Tue 2015-03-31 Beta Freeze
  • Tue 2015-04-14 Beta Release Public Availability
  • Tue 2015-05-05 Final Freeze
  • Tue 2015-05-19 Final Release Public Availability (GA)

We should reserve the right to change this draft schedule in case F21 slips into January. Schedules as no earlier than for most of milestones above.

As Josh pointed out, we don't have many options. We can be really strict before we accept changes/be strict at checkpoints to execute contingency plans.
Personally, I still prefer the current way. Initially aim for May/October but reschedule based on the scope of approved changes/other scheduling requests. It's a good compromise - developers has some initial expectation but we also have flexibility. For holidays conflicts - it's life. We're big international project, there always will be possibility to hit some of big holidays. Christmas is really show stopper for releases but it already happened ones and it wasn't that big disaster neither (just date was scary ;-).

Targeting mid-May and mid-October is a 5/7 month cycle, as noted above; are we sure we want to switch to that over the traditional start of May and end of October targets? In particular, a mid-October target will make it virtually impossible to release with GNOME 3.x.1; we'd wind up with 3.x.0, which historically have not been particularly reliable releases. The extra two weeks in October would result in a higher-quality Workstation release.

(I'm not arguing against mid-May for F22 -- that seems great, especially since the schedule is tight -- just that in general, start of May and end of October is optimal for Workstation.)

jreznik_2nd will send to devel-announce: retrospective wiki page, changes must be testable by early feb 2015, and we are targeting a may release. Specific dates to come. (+6,0,0)

Leaving ticket open for more discussion.

AGREED: Postpone the ticket to next week's meeting since mattdm can't attend today's meeting (+6, 0, -0)

Leaving the ticket open.

  • LINK: http://fedoraproject.org/wiki/Releases/22/Schedule (mattdm, 18:43:55)
  • change submission deadline in less than two weeks (mattdm, 18:44:51)
  • AGREED: FESCo would like for F22 to strictly adhere to a schedule, rather than adjusting the schedule based on submitted features. We intend to enforce the contingency plan very strictly this cycle (+8, 0, -0) (sgallagh, 19:04:19)
  • Tentative date for side-tag merge is 2015-01-28 (sgallagh, 19:09:55)
  • Tentative date for mass rebuild (if needed) is 2015-01-30 (sgallagh, 19:10:05)
  • ACTION: jreznik to send schedule reminder announcement (sgallagh, 19:12:05)
  • AGREED: FESCo approves the current proposed schedule with a planned final delivery on 2015-05-19 (+8, 0, -0) (sgallagh, 19:15:08)

Okay, so, stepping back to the bigger calendar picture:

If we change our position here and decide to allow a few more weeks in the schedule, that puts us into a nominal June release and a probably July one for F22. That in turn means that a six-month schedule for F23 would put us in January 2016, but assuming the same basic thing repeats, February or March. Then, that puts F24 in October 2016 (and, again, if we're extending beyond 6 months for features, creeping back to the holidays and risking slipping until 2017).

If, on the other hand, we stick to a strict release cycle for F22 and F23, we have F22 at the end of May, and F23 in October, and then a slightly more relaxed F24 cycle for F24 with a May 2016 target.

Basically, while it seems like putting more up-to-the-minute changes into F22 is in line with the "First" mandate, it actually means that we're slowed down overall.

With the shorter schedule, "work in Rawhide, and it'll be in a release this year" is more realistic. As we get to longer and longer schedules, work in Rawhide becomes less and less relevant to a ship-to-users timeframe, and therefore less and less interesting to work on. And more work in rawhide rather than work on the branch is a better way to split resources -- given that we ''don't'' have enough people to actually develop in two places at once, it's both more efficient and less risky to focus on the devel head.

So, I'd really like to stick to our plan here. If that means forgoing a mass rebuild, what are the repercussions?

As I won't be able to attend the FESCo meeting today I am +1 to keeping the more strict time based schedule and cut the changes that are unrealistic for this shorter schedule.

http://meetbot.fedoraproject.org/fedora-meeting/2015-01-14/fesco.2015-01-14-18.01.html

AGREED: While Fedora prefers to always carry the latest features, FESCo has determined that the Fedora 22 schedule is not compatible with including a mass-rebuild. FESCo would prefer to hold to a time-based schedule. (+6, 1, -1)

Login to comment on this ticket.

Metadata