= Proposal topic =
There is a perceived problem with regressions in updates.
Discuss policies to help prevent regressions from entering the updates repository
= Problem space =
Recently, there have been a few high profile updates that caused regressions. Further updates did not fix the regressions. This has prompted people to ask how to fix the issue.
One thought is that more testing is the key to preventing regressions. This ticket is to discuss policies that encourage more testing of packages before they get to the updates repository.
= Solution Overview =
One proposal is here: http://lists.fedoraproject.org/pipermail/devel/2010-March/132730.html
There are also counter proposals in the thread. It would be nice to have a wiki page to distill these out.
= Active Ingredients =
Making a change would take coding work from bodhi, autotest, and possibly releng or koji developers.
The policy would affect the workflow of all packagers, qa personnel, releng, and end users.
= Owners = mjg owns the proposal linked to from this ticket.
I have just posted another proposal to devel-list:
http://lists.fedoraproject.org/pipermail/devel/2010-March/132886.html
It is unfinished, but the key parts may be considered when evaluating this topic.
I propose we defer this, in order to: * give QA some time to finalize their proposal, * allow for more mailing list feedback.
All this is coming on a very short notice (e.g. QA's draft proposal was posted less than 2.5 hours before the meeting) and at least 1 FESCo member (me) didn't have time to read all of the replies yet (and I suspect I'm not the only one, plus in my case technical issues due to a hardware problem at Gmane didn't make it easier).
I did read a bunch of replies to Matthew's original proposal last night and posted some replies myself (and they made it through to the list a couple hours ago, it seems, they were stuck in some queue at Gmane). But I can't spend 24/7 on FESCo proposals, I have sleep, paid work, Fedora packaging, KDE SIG meetings (today was our meeting day) etc. to do as well. So more time between posting the proposal on one hand and discussing and voting it on the other would be very much appreciated!
Wiki page with all proposals thus far created. Needs analysis of commonalities and unique features if fesco doesn't get to doing their own analysis first:
https://fedoraproject.org/wiki/UpdatePolicy%28draft%29
From the 2010-03-09 FESCo meeeting:
{{{ * LINK: https://fedoraproject.org/wiki/UpdatePolicy(draft) (nirik, 20:57:23) * will work on nottings proposal over the next week and revisit next week. (nirik, 21:31:19) }}}
The Fedora Board has published the following updates vision statement:
https://fedoraproject.org/wiki/Stable_release_updates_vision
Please take this into consideration in any updates policies you work on.
So we're now getting a diktat from the half-unelected Board which appears to completely ignore the desires of the majority of our users[1], instead repeating already disproven arguments such as the following?
[1] http://forums.fedoraforum.org/showthread.php?p=1337744
"So we're now getting a diktat from the half-unelected Board which appears to completely ignore the desires of the majority of our users[1]"
I created the poll Kevin references. I'd just like to put on record that I don't believe it necessarily reliably represents 'the desires of the majority of our users'. It's not a big enough sample size, and the fact that it's a forum poll may mean that the people taking it are not truly representative of the overall user base.
If we did a better poll it may have the same result; it may not. I don't think there's sufficient data to know either way.
Mainly I think the poll just raises whether new-release 'updates' have become a recognized and desired 'feature' of Fedora for our users as a legitimate question, despite the fairly strong 'common wisdom' among developers that it's been more or less an accident and not something anyone particularly wants. But it doesn't conclusively answer that question.
This is just a factual note as to what I think the poll means, not a declaration of support for any particular position on the 'what kind of updates should we do' question.
Hello FESCo folks,
The Board and I spent a significant amount of time discussing and putting together the vision statement we posted yesterday, which was unanimously supported by the Board. FESCo is entrusted with specific implementation and policy that marries up with that vision, and we've tried to make your job easier. If any part of what we posted is unclear, technically infeasible, or otherwise objectionable to FESCo as a group, let us know the specific problematic piece, and what would need to change to remove the blockage. Then the Board will work with you to help resolve it.
The process of identifying specific items blocking consensus, and then deciding how to change them, worked extremely well for us in creating the vision statement, and ended up maximizing the support of everyone involved. We've been using that approach on the Board for more of our drafts and meetings lately, and my sense is everyone has found the work much more productive and effective as a result. I'd highly recommend that FESCo try the same process to come to a consensus as a group about concerns with the statement we published.
My complaint was not about the decision-making method, but about the resulting output, which: * appears to radically change Fedora's approach to updates in a way which goes against the wishes of a huge portion of our users (estimated to over two thirds by the only numeric data we have, as imperfect as it may be), * is justified by arguments already disproven during previous discussions (see my reply for a summary of what's wrong with those arguments).
Re https://fedoraproject.org/wiki/UpdatePolicy(draft)#notting.27s_Proposal I have 3 proposals for amendments:
Amendment proposal A: Strike section 3.
Rationale: For non-critical packages, the maintainer knows best what's best for his/her package. We should trust the maintainer to exert the proper judgement. If the maintainer decides to push the package directly to stable, he/she probably has a good reason to decide that, and positive karma is extremely hard to get for niche packages.
If (and only if) amendment proposal A cannot be agreed on, then there is:
Amendment proposal B: In section 3, add that non-critical packages can, in addition to the 2 proposed ways they can go stable, also go stable if they fulfill the criteria we require for critical packages (i.e. the requirements stated in section 2).
Rationale: It doesn't make sense to apply stricter criteria to non-critical packages than to critical ones, and the criteria for critical packages do not logically imply the ones for non-critical packages. Adding them as a third option rectifies this inconsistency.
Completely independently of amendment proposals A and B, I have:
Amendment proposal C: In section 2, codify that the group of testers empowered to give the magic karma needs to include at least 1 member (or in the event the criteria get changed to require some karma n rather than 1, n members) of each SIG behind one of the desktops considered "critical" by section 2.
Rationale: This ensures the SIGs controlling the desktops can make autonomous decisions about when to push updates to their desktops without being reliant on external groups such as QA.
This was approved with amendment B from comment 12.
Need to work on a implementation plan and gather what needs to change. Will leave this open for tracking that.
I just wonder, how is the upgrade path defined? Is it from current updates to next updates-testing? Or from current updates to next updates? Or from current updates to next updates and updates requested for stable?
And who specifies the "positive bodhi karma threshold" of a package? Is this left to the submitter of the update?
Replying to [comment:14 till]:
That probably could stand some clarification, yes. We'll discuss how this interacts with pending packages.
Yes.
The accepted policy has been posted to the wiki at https://fedoraproject.org/wiki/Package_update_acceptance_criteria.
Replying to [comment:16 notting]:
Replying to [comment:14 till]: I just wonder, how is the upgrade path defined? Is it from current updates to next updates-testing? Or from current updates to next updates? Or from current updates to next updates and updates requested for stable? That probably could stand some clarification, yes. We'll discuss how this interacts with pending packages.
This seems to have been forgotten, because it was not mentioned in the meeting minutes and from a sweep over the log I could not find anything related.
And who specifies the "positive bodhi karma threshold" of a package? Is this left to the submitter of the update? Yes.
So can the submitter just set the threshold to zero (or one and add his own +1 karma)?
This seems to have been forgotten, because it was not mentioned in the meeting minutes and >from a sweep over the log I could not find anything related.
It wasn't forgotten, these questions are waiting pending checking with bodhi/pkgdb admins and see what makes sense from an implementation standpoint. Please stand by. ;)
Zero isn't positive, so no on that front. An updater setting it to one and providing their own karma would certainly be frowned upon (and can be enforced in the tool.)
Replying to [comment:20 notting]:
So how about to change {{{reach the positive bodhi karma threshold specified by the updates submitter OR}}} to {{{reach the positive bodhi karma threshold specified by the updates submitter with at least one positive karma point from someone else than the submitter OR}}} to make this clear.
ok. outstanding questions for implementing:
https://fedoraproject.org/wiki/Package_update_acceptance_criteria
If it's just added the critical path, the implementation is already in place. if it's a new group, more coding will be required in bodhi/pkgdb to implement.
Currently it has the same critera as the critical path, but may require different/more involved testing as the desktops and apps are complex gui's.
2, How do we want to populate the "important package" set. Currently critical path is a list, do we wish to generate a list for this? Will it change per release?
For "spend some minimum amount of time in updates-testing, currently one week" This would not autopush after a week, correct? Only allow the submitter to request stable after 1 week?
For the "reach the positive bodhi karma threshold specified by the updates submitter" Should submitter be able to add +1?
Should submitter be able to set threshold at +1?
Things we need in place:
AutoQA
proventesters group in place and populated with enough folks.
a defined "important" package list/set and changes to bodhi/pkgdb if this is different from critical path.
@kde-desktop contains A LOT more stuff (even if you only consider "default" entries) than what notting's proposal considers "important".
Replying to [comment:23 kkofler]:
Yeah, do you have a subset of that we should use? Or how should we handle it?
in reply to comment #24 at todays meeting:
1/2: We will have sig's and rel-eng populate @critical-path-desktopname groups in comps. All those will be added to the critical path.
This is not an autopush, only ability to push after a week.
submitters karma does not count.
Submitters can use +1 as their karma threshold if they choose.
This ticket is very informative, and thanks to the FESCo members for this information. It might be a little hard to digest for package maintainers who haven't followed the associated threads closely. Can I ask FESCo to summarize the implementation plans and schedule on https://fedoraproject.org/wiki/Package_update_acceptance_criteria ? That would help contributors understand where we are in the process, what we're waiting for, and what they should expect to happen in the future.
The comps changes have been done. Generation of these into critical path entries for particular releases has not been finished yet; it's waiting on some discussion as to the best way to do this.
We are also waiting for:
AutoQA to land.
A few minor bodhi changes. I will see about talking with lmacken/filing tickets on these.
Announcements to maintainers.
I'm not sure we can get a accurate schedule for when this might be able to go live yet, but should as it gets closer to done.
Critical path updates are tracked at https://fedorahosted.org/rel-eng/ticket/3740
I filed the following bodhi tickets:
https://fedorahosted.org/bodhi/ticket/433 https://fedorahosted.org/bodhi/ticket/434
If the implementation of the new updates policy is waiting on AutoQA has anyone made a determination as to when it will be ready? Does this mean AutoQA must have passed the package review process?
It would be good to state a rough time line even if it is not 100% accurate.
Will the implementation address the board's vision statement?
Replying to [comment:31 poelstra]:
No, the autoqa package has not yet been submitted for review. There remains significant work to identify and resolve unpackaged dependencies. Any volunteers interested in helping that move forward, please contact the QA team.
Our intent is to automate the package update acceptance criteria as documented at https://fedoraproject.org/wiki/Package_update_acceptance_criteria.
We are tracking several efforts to document and automate the tests. We have already received feedback from the test plan, and are working the AutoQA milestones now. * Test plan intended to exercise the criteria is available at [https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan Package_Update_Acceptance_Test_Plan] * Specific test automation tracking ... * [https://fedorahosted.org/autoqa/milestone/Package%20Sanity%20Tests Automate package sanity] * [https://fedorahosted.org/autoqa/milestone/Package%20update%20tests Automating upgrade path, rpmlint and rpmguard] * [https://fedorahosted.org/autoqa/milestone/autoqa%20depcheck Automating dependency and package+file conflict detection]
The Board's vision statement is orthogonal to this ticket. The Board's vision statement is about what kind of updates should be submitted, this ticket is about what criteria a submitted update needs to fulfill before getting pushed out as a stable update.
Replying to [comment:10 pfrields]:
Hello FESCo folks, The Board and I spent a significant amount of time discussing and putting together the vision statement we posted yesterday, which was unanimously supported by the Board. FESCo is entrusted with specific implementation and policy that marries up with that vision, and we've tried to make your job easier. If any part of what we posted is unclear, technically infeasible, or otherwise objectionable to FESCo as a group, let us know the specific problematic piece, and what would need to change to remove the blockage. Then the Board will work with you to help resolve it.
Did FESCo officially accepted https://fedoraproject.org/wiki/Stable_release_updates_vision ? Is there a separate ticket tracking it's implementation (as suggested in another comment)?
To the best of my knowledge, no specific proposal related to that vision has come before FESCo yet.
Replying to [comment:34 poelstra]:
My understanding was that this vision was passed on from the Board. I don't know that we would need to approve anything here, as they are giving us direction here. I suppose if there are things that fesco does not agree with in it, they/we should approach the Board and ask for reconsideration.
We need to implement policies and update docs to match the vision, IMHO.
IMHO, that "vision" is insane, destructive (we'll lose the vast majority of our users), unimplementable and unenforcible and we should tell the Board as much and refuse to implement any of it.
Replying to [comment:37 kkofler]:
Feel free to bring your concerns to the Board. They have a list, an irc channel now, and do open irc meetings. I'm sure they would welcome your constructive ideas.
I think I have muddied the waters of this ticket with my questions. I will open a second ticket specifically for https://fedoraproject.org/wiki/Stable_release_updates_vision and this ticket can continue to track the process of releasing update.
New ticket # for implementation of Board's vision: https://fedorahosted.org/fesco/ticket/382
Luke is looking at getting a new bodhi release ready for later in the week that should have the critical path/important packages and hopefully the 'all other packages' section in it. Will update as we know more.
I'd like to ask FESCo at the next meeting to consider what impact negative karma should have on candidate updates. Particularly critpath updates, and particularly negative karma from proventesters.
I'd argue that critpath updates should require exclusively positive karma from proventesters, not just a total summed +1 or +2. So a critpath update should be held if it has any -1 from proventesters, even if two other proventesters gave it +1.
(There's one obvious practicality issue here - whether the calculation can take account of the -1 feedback being revoked, so the update can actually be accepted once the problem is resolved and the proventester who filed the -1 wants to turn it into a +1. Not sure, internally, how Bodhi handles revoking feedback and if it could cope with this, would need Luke to chime in there.)
I don't think you can revoke karma to 0; you can only replace your vote from +1 to -1 (or vice versa.)
Replying to [comment:42 adamwill]:
Imho proventesters should be able to override other proventesters -1 karma, e.g. if they verified that the problem is solved by an update edit or some other package pushed to testing or because the bug was already there in some previous stable update. Relying on a certain person to revoke their -1 karma is imho just asking for trouble. Not because of malicious behaviour, but because of possible time constraints. But the method to override a -1 karma should be different from just giving +1 karma to make sure that this override happens intentionally.
Replying to [comment:44 till]:
[snipped]
+1 to Till, sounds reasonable.
(I won't be able to make it for the meeting tonight, that's why I'm voting here)
2010-07-06 meeting: lmacken is going to write up exactly what the current behavior is and we will see about how we want to adjust it. Will revisit next week.
AutoQA is still being worked on, will revisit next week for a possible timeline for implementing.
Replying to [comment:46 kevin]:
I made an initial pass at documenting the karma system on the wiki:
https://fedoraproject.org/wiki/Bodhi
We decided not to make any changes currently. AutoQA should hopefully go live in the f14 cycle.
We still need to get the "at least one week in testing" requirement for other packages in place.
I have three requests:
Replying to [comment:49 till]: {{{ Adjust the policy for other packages to require only a net karma sum of 1 instead of the threshold defined by the maintainer. From the policy POV this is what is minimally required, but implementing it using the autokarma value makes it only harder for maintainers. If they want to decide after e.g. two +1 comments to push the package to stable, they have either to enable the autokarma setting or adjust the value from the default of 3. This leads only to more work, without any additional gain. }}}
Can you clarify here? We're not entirely clear what situation you're referring to.
Replying to [comment:50 notting]:
Replying to [comment:49 till]: {{{ Adjust the policy for other packages to require only a net karma sum of 1 instead of the threshold defined by the maintainer. From the policy POV this is what is minimally required, but implementing it using the autokarma value makes it only harder for maintainers. If they want to decide after e.g. two +1 comments to push the package to stable, they have either to enable the autokarma setting or adjust the value from the default of 3. This leads only to more work, without any additional gain. }}} Can you clarify here? We're not entirely clear what situation you're referring to.
The current policy says "reach the positive Bodhi karma threshold specified by the updates submitter" and it is valid to use a value of one for this. But this also means that the package is automatically pushed to stable once the package got this one +1 comment. If a maintainer does not want the update to be pushed to stable automatically and uses the default value of 3 and then has the need to push the update to stable ASAP (and the update already got one +1 comment), he first needs to lower the threshold and is then able to push it. This makes it just a more complicated procedure with no additional gain.
Example timeline:
But a better workflow would be for the last step to just push the update to stable.
I just read the meeting log so I now assume to know what the misunderstanding was. {{{ 19:41:49 <notting> - submitter set autokarma at +3 19:41:57 <notting> - package has the policy-required +1 karma 19:42:10 <notting> - bodhi does not allow a push to stable until the autokarma threshold is reached 19:42:15 <notting> if that's the case, we should fix that }}}
This is excactly the case. There is only a autokarma setting the maintainer can set and not a "ready for stable but do not push it automatically" setting. But this setting is also not really helpful, because only the maintainer or co-maintainers can even push the update to stable manually and therefore the system does not need to check this somehow.
But +1 is not enough to push it to stable - you need either +2 (with +1 from proventester) or 'whatever the maintainer specified' (per the policy). So I'm still a little confused. (And yes, I misspoke in the meeting.)
Replying to [comment:53 notting]:
But how does the maintainer specify this threshold? And why should he even need to do this? Can you please explain the problem that exists if the update has just +1? Who is going to push the update to stable if not the maintainer? I honestly do not understand you.
If the maintainer has specified autokarma at +3, then if the update only has +1, the update has not reached the acceptance criteria in the policy. The acceptance criteria is one of:
Karma of +1 (in the case where the updater specified +3) doesn't fit these criteria.
So, I'm still confused by what you're asking. Are you looking for the ability of the maintainer to set a karma threshold that won't allow it to be pushed until it reaches that karma, but won't automatically push it?
Replying to [comment:55 notting]:
Yes. Or to make it simplier and have the same effect: just change the policy to require +1 for non critpath updates.
The current policy already allows do to this, but only with a PITA workflow, because one has to use the autokarma setting for this and edit it and the editing is also somehow broken.
ok. We have a policy in place now.
https://fedoraproject.org/wiki/Updates_Policy
I'm going to close this ticket and open a new one for adjustments to the current policy.
Login to comment on this ticket.