#403 Abuse of Privileges / Procedure Overruled
Closed None Opened 13 years ago by kanarip.

Recently, rvokal@redhat.com sponsored two new Red Hat employees -that I know of- into the Fedora CVS Packager Commit group. These two users that I know of (psabata and ppisar) have since been forcibly permitted co-maintenance on over 1200 packages.

I suspect none of the original package owners or co-maintainers has been asked permission -since I wasn't either- and the procedures in place for requesting co-maintainership have been skipped entirely.

I would thus like FESCo to consider;

1) To renounce these practices as apropriate and acceptable and thus enforce the proper use of procedures put in place by our appropriate Fedora Project steering committees and as ratified by the Fedora Project Board, or

2) To mark these practices as acceptable and document the procedural exception that inherently applies to Red Hat employees.


I would like to add that the users in question seem to have not yet reviewed a single package, have not yet submitted a package for review, and appear to not own themselves a single package in Fedora, yet have been sponsored in the CVS packager group.

https://fedorahosted.org/rel-eng/ticket/3780

is the ticket and discussion for this. The perl development list was CC'd on all of it.

There's several issues involved here which all independently are problems and all together are bigger then the parts. I'll start with history, move on to the issues, and then get to potential remedies.

== History ==
Mass changes in package ownership are hard on several accounts.

  • There is no easy to use UI for packagers to make changes to all of their packages -- the cvsadmins do have a command line tool to make changes but not all of the commands would work for normal users.
  • Making a large number of changes sends out a large amount of email. (people in these groups can use the command line tool with some special workarounds to not send mail: cvsadmin + sysadmin-web + sysadmin-cvs). (people in sysadmin-db can make changes directly in the database but they have to spend time crafting the custom sql query to perform the operation).

Because of those issues, I (as noted above, other cvsadmins can conceivably do this but it's almost always me) let package maintainers (owners and approveacls holders) request that I make a mass change to packages such as adding new comaintainers, removing inactive accounts,

This particular request was a little out of the ordinary. You can see it here: https://fedorahosted.org/rel-eng/ticket/3780

This request was to add comaintainers to every package that the perl-sig has responsibility for. The perl-sig is a pseudo user meaning that it is largely used to get commit and bug mail to go to the perl-sig's mailing list rather than an actual owner of packages (other SIGs/teams do this pseudo ownership slightly differently -- for instance, the kernel team places the kernel-maint pseudo-user as the package owner of the kernel package so that the bugzilla bugs are first assigned to the pseudo user and only assigned to a real person when that maintainer takes the bug. No one can log in as the pseudo user so even here, the primary purpose is triage, not commit privileges. Not having participated in the perl-sig, it seemed like the perl-sig would need to be informed of this request and given the opportunity to reject it. The requestor (the perl package maintainer) CC'd the perl-sig on the ticket which posted the ticket to the perl-sig mailing list. One positive response and no negative responses were gathered in a week so I processed the request.

Separate from the history of doing mass acl updates, packagers who are Red Hat employees have enjoyed a fast track to sponsorship in the past. Because their job requires packaging, their managers have been willing to sponsor them into the packager group and claim responsibility for fixing any problems if the packager messes up. This was contrary to the recommendations when Fedora started that packagers should do reviews in order to show their abilities before being sponsored (although those were just recommendations). With the recent changes to the sponsor and maintainer responsibilities, it seems like the direction is to bring this sort of sponsorship-before-proof to everyone rather than push everyone to a proof-before-sponsorship model.

== Issues ==

The following issues come up in relation to this:

  • Should Red Hat managers still be sponsoring people in because they need to maintain packages as part of their job rather than having them earn trust by performing reviews? As stated above, it seems like Fedora is moving towards this model everywhere so this doesn't seem like a big issue.
  • The perception that Red Hat employees are treated differently. In this case, the requestor was a Red Hat employee, the two new comaintainers are Red Hat employees, the one respondent does not appear to be, and I am a Red Hat employee. I claim that I do the same for any requests but there haven't been others that involve adding a comaint to packages that are all part of a sig.
  • Even though the perl-sig was added to the ticket, it's now come to my attention that the perl-sig isn't a cohesive sig:
  • adding the perl-sig as a CC is recommended in the packaging guidelines so there may not be the expectation by packagers that they and their packages are actually "owned" in common with the perl-sig: https://fedoraproject.org/wiki/Packaging:Perl#Set_inital-cc_to_.27perl-sig.27
  • The perl-sig's mailing list seems to be used more for automated messages (commit and bugzilla messages) rather than communication between members.
  • The perl-sig may never have had a meeting and the body of membership seems to be fluid and ad hoc. This is not uncommon among SIGs; it's just that the impression I had was of something more formal since the perl packages all seem to get the perl-sig pseudo-user added.
  • Who should have authority to request changes?
  • And should there be an override procedure for that?
  • Should there be any requirements for the packagers that are added via a mass change?

== Options ==

  1. Requestor has approveacls on the package(s) they wish to change, make the change
    * If they don't, fesco approval needed
  2. Update perl guidelines to make clear that adding perl-sig carries the idea that perl-sig is a comaintainer and might allow others to be added.
  3. Update perl guidelines to have separate comaint and cclist with different expectations.
  4. Don't do mass updates

I like the first option the best

Thank you for the explanation Toshio, it clarifies how this came to happen.

I would appreciate if FESCo would take into account the sponsoring of these users into the CVS packager group, which happened prior to adjusting the !PackageDB ACLs.

Another one to put down under options:

  1. Define further criteria before one is allowed to be added as a comaintainer on packages via a mass update.

I am not for this one as it doesn't hinge on fairness, a packager can approve the comaintainers themselves, it's just more work via the webui, and I feel that specifying a policy of someone who is already a comaintainer (ownership/approveacls) or fesco dispensation to perform a mass update on a package covers the goals better.

(Note that I'm considering kanarip's comment to be Number five on the list of options:

  1. Work out a new policy for sponsoring into packager that raises the bar for people to get into packager so that the packager must prove themselves capable of packaging before gaining approval. The guidelines should match for Red Hat and non-Red Hat people as much as possible.

I don't like the "raise the bar" portion of this but I do like the fairness of not having two sets of (informal as there's no you must do this to be sponsored into packager rules); one for Red Hat and one for the community. I feel that we've been aiming to lower the bar for entry into packager for a long time so we don't want to go back and rewrite our goals in the opposite direction. As for fairness, I think that comes about by clarifying that you can sponsor people who are still learning packaging as long as you're willing to mentor them.

Toshio, thanks for summing it up. I'd just add one more side note here. As a manager of these two people in question by approving them to a packager group I take a full responsibility for their actions. We do have internal Red Hat processes to educate these people not only about Red Hat internal things but also about Fedora. Honestly, Red Hat providing its resources for work on Fedora is still great advantage of the distro and the bar for hiring people in various teams is still high - that said, both Petrs are skilled programmers with packaging experience from other distros or companies packaging their how grown software. My sum up here, new Red Hat employees should always have a shortcut here, they go thru hiring process which proves their skills as well as education process which gets them well aware of Fedora processes.

Replying to [comment:8 rvokal]:

I'd just add one more side note here. As a manager of these two people in question by approving them to a packager group I take a full responsibility for their actions.

Every sponsor is ultimately responsible for his sponsorees, so why do you think this is worth mentioning here?

We do have internal Red Hat processes to educate these people not only about Red Hat internal things but also about Fedora.

Honestly I think persons undermining the processes and guidelines in Fedora are not the right ones to educate people about Fedora.

My sum up here, new Red Hat employees should always have a shortcut here, they go thru hiring process which proves their skills as well as education process which gets them well aware of Fedora processes.

The hiring process may prove their skills to Red Hat Human Ressource dept. but not the Fedora community or the maintainers of the packages in question. How do you want to gain trust from them without any prove of skills, cooperation or community engagement?

Just because someone is hired by RH does not mean they get a free pass into Fedora. This is a community driven distribution and everyone needs to be prove themselves to the community before being given this access.

IMO every package owner should have been contacted directly and allowed to either grant or deny access to their packages. Yes it may be onerous and time consuming but it is the right way to do it.

Okay, that sounds like another option which is a variant of option 1:

  1. If the requestor has approveacls on the package(s) they wish to change, make the change
    • If they don't, package owners need to be contacted directly to either grant or deny the request to their packages.

Adding meeting keyword here for next meeting.

Replying to [comment:11 toshio]:

Okay, that sounds like another option which is a variant of option 1:

  1. If the requestor has approveacls on the package(s) they wish to change, make the change
    • If they don't, package owners need to be contacted directly to either grant or deny the request to their packages.

I think the latter establishes the correct way; It can be done automatically and there's no reason to have to collect results before executing anything; Just set the default ACL request to "Pending Review" rather then "Approved", and you enforce a feedback dialog with proper results. In cases where a maintainer remains unresponsive, the correct procedures for "unresponsive maintainer" can be started.

It's completely understandable new-hires at Red Hat with the specific assignment of giving foo some love (in this case, perl), are not required to manually go through the PackageDB web interface to request access to all perl packages.

The former notwithstanding, such mass-requesting (through a script), even if just for "Pending Review" ACLs, should be approved by the appropriate governing body (FESCo), not Red Hat Human Resources nor Management of some Red Hat Engineering department.

Replying to [comment:13 kanarip]:

Replying to [comment:11 toshio]:

Okay, that sounds like another option which is a variant of option 1:

  1. If the requestor has approveacls on the package(s) they wish to change, make the change
    • If they don't, package owners need to be contacted directly to either grant or deny the request to their packages.

I think the latter establishes the correct way; It can be done automatically and there's no reason to have to collect results before executing anything; Just set the default ACL request to "Pending Review" rather then "Approved", and you enforce a feedback dialog with proper results. In cases where a maintainer remains unresponsive, the correct procedures for "unresponsive maintainer" can be started.

This would not solve either of the two basic problems I outlined in my summary at the beginning. "There is no easy to use UI for packagers to make changes to all of their packages" and "Making a large number of changes sends out a large amount of email." The purpose of mass updating is not just to help the requestor but also the maintainers. Asking for commit rights to 1000 packages from 40 maintainers as an automated step is only the tip of the iceberg; it still sends email for each request and then has the maintainer go through the webui to approve them.

It's completely understandable new-hires at Red Hat with the specific assignment of giving foo some love (in this case, perl), are not required to manually go through the PackageDB web interface to request access to all perl packages.

The former notwithstanding, such mass-requesting (through a script), even if just for "Pending Review" ACLs, should be approved by the appropriate governing body (FESCo), not Red Hat Human Resources nor Management of some Red Hat Engineering department.

Once again, this portion has nothing to do with Red Hat. Mass acl updates are processed for anyone requesting them. Up until now, I am the de facto "approver" in as much as I check that there is a relation between the requestor of the updates and the packages in question. These things aren't limited to me as a person but to the roles I hold within Fedora. Any cvsadmin can perform these requests. Any cvsadmin with sysadmin-web help can do it without sending email.

I think you're creating a new option:

  1. FESCo approval of all mass acl updates.

I'd much rather see delegation according to a stated criteria of common cases than adding an approval step to every request. Option 1 for instance means: where a person who has approveacls on a set of packages states they want another packager added as a comaintainer we're not doing anything that they can't do themselves. We're just doing it faster and more efficiently. Where the requestor (and others giving explicit approval -- which could be arranged via out-of-band email, for instance) are not comaintainers of the packages, it goes to fesco for approval.

I'd like to add another option based on option 1:

9.
- Requestor has approveacls on the package(s) they wish to change
- If they don't, it needs fesco approval.
- email is sent to $package-owners explaining who requested the change and what actions are going to happen.
- Change is made.

(ie, add a step to mail maintainers that fesco approved the mass change or the maintainer has acls on all the affected packages, etc)

Replying to [comment:14 toshio]:

Replying to [comment:13 kanarip]:

Replying to [comment:11 toshio]:

Okay, that sounds like another option which is a variant of option 1:

  1. If the requestor has approveacls on the package(s) they wish to change, make the change
    • If they don't, package owners need to be contacted directly to either grant or deny the request to their packages.

I think the latter establishes the correct way; It can be done automatically and there's no reason to have to collect results before executing anything; Just set the default ACL request to "Pending Review" rather then "Approved", and you enforce a feedback dialog with proper results. In cases where a maintainer remains unresponsive, the correct procedures for "unresponsive maintainer" can be started.

This would not solve either of the two basic problems I outlined in my summary at the beginning. "There is no easy to use UI for packagers to make changes to all of their packages" and "Making a large number of changes sends out a large amount of email." The purpose of mass updating is not just to help the requestor but also the maintainers. Asking for commit rights to 1000 packages from 40 maintainers as an automated step is only the tip of the iceberg; it still sends email for each request and then has the maintainer go through the webui to approve them.

I would argue that whether the change automatically applied is "Approved" or "Pending Review" does not change anything wrt. the load of email being sent out.

Wrt. problem #1 you state, very little maintainers maintain such a volume of packages, that going through the motions in the WUI lays down an unacceptable load on the maintainer. I would also argue that this is exactly the feedback cycle from maintainers that would be appropriate; whether someone goes out of their way and contacts the maintainers individually, manually, collects results and composes lists, or maintainers get to give their response directly through the WUI doesn't make all that much of a difference.

That being said, suppose there is the one or two maintainers who's been requested to approve/deny 140 package ACLs over 6 branches, through the WUI, then this user could get back in touch and have the work be done scripted/automatically anyway (like has now been done with the original request).

It's completely understandable new-hires at Red Hat with the specific assignment of giving foo some love (in this case, perl), are not required to manually go through the PackageDB web interface to request access to all perl packages.

The former notwithstanding, such mass-requesting (through a script), even if just for "Pending Review" ACLs, should be approved by the appropriate governing body (FESCo), not Red Hat Human Resources nor Management of some Red Hat Engineering department.

Once again, this portion has nothing to do with Red Hat.

Alright then, "nor any other body not the appropriate body".

Replying to [comment:16 kanarip]:

I would argue that whether the change automatically applied is "Approved" or "Pending Review" does not change anything wrt. the load of email being sent out.

Sending the email saying that things were Approved was a mistake. One that I'm glad was made in this case since it's brought up the problem stated here that doing something on behalf of "the perl-sig" is not a good idea and that we need to codify who needs to make a request but still a mistake. As I outlined in the summary, the ideal is for the mass update to be done with email sending disabled.

Wrt. problem #1 you state, very little maintainers maintain such a volume of packages, that going through the motions in the WUI lays down an unacceptable load on the maintainer. I would also argue that this is exactly the feedback cycle from maintainers that would be appropriate; whether someone goes out of their way and contacts the maintainers individually, manually, collects results and composes lists, or maintainers get to give their response directly through the WUI doesn't make all that much of a difference.

I'm not sure that I said that anywhere (I have done "mass update" requests for as little as two packages in the past and maintainers have the option of using a bugzilla-based cvsadmin request if they don't want to add someone to every single branch of their package.)

I do agree that when a request is not coming from a maintainer, that cycle is needed. I disagree on two points:

  1. When the request comes from the maintainer, the cycle of request/approve becomes extraneous so no need to do that.
  2. The webui is decidedly the wrong tool for the job when there's a large number of packages -- it would be better to open a ticket and ping the relevant package owners via private email.

The former notwithstanding, such mass-requesting (through a script), even if just for "Pending Review" ACLs, should be approved by the appropriate governing body (FESCo), not Red Hat Human Resources nor Management of some Red Hat Engineering department.

Once again, this portion has nothing to do with Red Hat.

Alright then, "nor any other body not the appropriate body".

I dispute that FESCo is the appropriate body.

  1. In most cases, the package maintainers are the appropriate body. Therefore, when a request is made by them, asking permission from FESCo would be asking an inappropriate body.
  2. FESCo delegates decision making powers to other groups that are actually doing work. In this case, I think they should be delegating decision making power to the cvsadmins (as currently). However, FESCo can and delimit what decisions the delegate can make or how they should make them. I think option 1 or option 9 do this well.

Passing everything through FESCo seems to add an unnecessary step to the process. Only sending cases where it's not obvious that the maintainers or comaintainers have approved it through FESCo seems like a better policy.

In addition to the broad policy question, we should also get a decision on what to do about the specific change that lead to this. Spot has a proposal here:

https://fedorahosted.org/fedora-infrastructure/ticket/2248

I can remove all the people's acls in the db and then a cvsadmin and sysadmin-web member can add them back to to the packages that mmaslano specifically is the owner/comaintainer of.

Just leaving some opinions here, since I am likely to miss the fesco meeting today:

  1. I don't think fesco needs to be involved in this

  2. Accepting comaintainers is the responsibility of the individual package maintainers

  3. If this is for gaining access to all packages of a sig, the proper procedure would seem to be to discuss this in a sig meeting / on the sig list, and have all the involved package maintainers sign off on the added comaintainers before going ahead with cvsadmin mass updates

  4. Setting acls for thoughsands of packages seems to run into serious scalability issues. Would it be an option to instead grant access to a more limited set of, say 40, packages for the new people, and let them request provenpackager after they've proven themselves on this limited set ?

At today's meeting we decided moving forward for any mass acl change:

  1. The requestor should have approveacl's on all packages to be operated on.
    If they do not, the request can be sent to fesco, but should not be acted on until approved.

  2. The change committer should mail affected package co-maintainers and inform them of who made the request and exactly what packages of theirs are affected.

Additionally, in reguards to this mass acl change: revert permissions on packages where approving maintainers didn't have approveacls.

I will leave this open for documentation/implementation.

I've added documentation on how to do a mass acl change here: https://fedoraproject.org/wiki/CVS_Admin_SOP#Performing_mass_comaint_requests

Documentation of what the criteria is should be there or linked from there.

Permissions of perl packages have been reverted. A cvsadmin can readd them to mmaslano's packages now. Details of how are here: https://fedorahosted.org/fedora-infrastructure/ticket/2248#comment:3

I added policy info to the sop. ;)

I am going to close this now and we can track implementation of the revert in the infrastructure ticket above.

Login to comment on this ticket.

Metadata