#5931 [Proposal] Move new branch and unretire requests to pkgdb2
Closed: Fixed None Opened 9 years ago by pingou.

Till suggested me recently that maybe we could port the new-branch requests to pkgdb2, together with the requests that for a package to be unretired.

I have been working with these ideas, they are simply a way for users to ask an admin to perform an action (atm: request.branch and request.unretire).

The output can be seen for admins via the UI at:
http://209.132.184.188/admin/actions/

Or for everyone via the API at:
http://209.132.184.188/api/admin/actions/

Note: the logic to change the status of a request isn't there yet but I thought I would first run the idea by you guys to get some early feedback, especially since if you like the idea, some tools will need adjustments.


Dennis mentioned having problem to log in, so here is a screenshot of the first link above:
http://ambre.pingoured.fr/public/Screenshot_pkgdb2_admin_actions.png

To summerize the process as it is with the code in gits now:

{{{
* new package request
- User enters the information after the review for his/her package
- Admin runs pkgdb-admin
- pkgdb-admin checks
- if the summary of the review attached fits the expectations (w/ the information entered in pkgdb)
- if all the people that commented on the ticket are packagers
- if the person that set the fedora-review flag to '+' is a packager
- pkgdb-admin presents the results of the checks to the admin
- Admin approves/denies the creation of the package

  • new branch request
  • User requests a new branch for a package in pkgdb
  • If the user requested a Fedora branch and already has commit or approveacls on the package
    -> request granted automatically
  • else
    -> request stored
  • Admin runs pkgdb-admin
  • pkgdb-admin runs some checks
    -> checks to be implemented
  • Admin approves/denies the creation of the new branch
    }}}

So basically, the piece missing atm is the list of checks we want to perform automatically before creating a new package/branch and the piece updating the git repos based on the fedmsg message received (although this is already partly there).

A small preview of what pkgdb-admin looks like currently:

{{{
$ python pkgdb2client/admin.py --test list
Testing environment
#2 (Awaiting Review) - pingou requested the unretirement of "8Kingdoms" on "master"
#9 (Awaiting Review) - pingou requested a new branch "f20" for "python26-argparse"
#10 (Awaiting Review) - pingou requested the new package "guake" on "master"
#13 (Awaiting Review) - pingou requested a new branch "epel7" for "R-ALL"
#20 (Awaiting Review) - pingou requested the new package "python-watchdog" on "master"
#21 (Awaiting Review) - pingou requested the new package "python-watchdog" on "f20"
#22 (Awaiting Review) - pingou requested the new package "python-watchdog" on "f21"
Total: 7 actions

$ python pkgdb2client/admin.py --test process 20
Testing environment
#20 (Awaiting Review) - pingou requested the new package "python-watchdog" on "master"
+ All checks cleared for review 1089392: python-watchdog
What should we do about this requests?
approve, deny, pass: approve
Package created

$ python pkgdb2client/admin.py --test process 21
Testing environment
#21 (Awaiting Review) - pingou requested the new package "python-watchdog" on "f20"
Package python-watchdog found, requalifying request.package in request.branch
FAS password for user pingou:
+ All checks cleared for package python-watchdog
What should we do about this requests?
approve, deny, pass:
No valid action specified, just ignoring for now
Action 21 un-touched

$ python pkgdb2client/admin.py --test process 21
Testing environment
#21 (Awaiting Review) - pingou requested the new package "python-watchdog" on "f20"
Package python-watchdog found, requalifying request.package in request.branch
FAS password for user pingou:
+ All checks cleared for package python-watchdog
What should we do about this requests?
approve, deny, pass: deny
user: pingou updated action: 21 from Awaiting Review to Denied

$ python pkgdb2client/admin.py --test process 22
Testing environment
#22 (Awaiting Review) - pingou requested the new package "python-watchdog" on "f21"
Package python-watchdog found, requalifying request.package in request.branch
FAS password for user pingou:
+ All checks cleared for package python-watchdog
What should we do about this requests?
approve, deny, pass: approve
user: pingou set for pingou acl: commit of package: python-watchdog from: to: Approved on branch: f21
user: pingou set for pingou acl: watchbugzilla of package: python-watchdog from: to: Approved on branch: f21
user: pingou set for pingou acl: watchcommits of package: python-watchdog from: to: Approved on branch: f21
user: pingou set for pingou acl: approveacls of package: python-watchdog from: to: Approved on branch: f21

}}}

TOFIX: find out a way to store the FAS and bugzilla cookies somewhere so that we don't have to re-enter the passwords every-time...

Today at the meeting we decided:

  • for new epel branch we will try Till's suggestion:
    2/ We manage to create the branch in such a way that they point to the first
    commit in the repo (suggestion from Till), from there the maintainers should
    be able to fast-forward to the branch they think should apply

  • I'll work with Dennis to see how we can get RHEL's information to check if a package is in RHEL before creating an EPEL branch for it

  • If you can think of other checks to perform (cf https://fedorahosted.org/rel-eng/ticket/5931#comment:5 ) feel free :)

Just keeping a note: to find the first commit in a repo {{{ git rev-list --max-parents=0 HEAD }}} will return its hash. So we could jsut use this when creating the new branch.

FYI: python-bugzilla already caches its cookie

Ok I fixed the problems which I had with keeping the auth for FAS, pkgdb and bugzilla accross multiple runs of pkgdb-admin.

So as far as I can see, all what's left is the part for EPEL branches that relies on us publishing the JSON output of the RHEL repos. Once we have the cron in place and the JSON published it will be easy to add the check :)

  • Added the cron job creating the JSON structure from the yum metadata
  • Adjusted pkgdb-admin to check if the package is in RHEL upon EPEL branch requests:
    {{{
    $ python pkgdb2client/admin.py --test process 16
    Testing environment

16 (Awaiting Review) - pingou requested a new branch "epel7" for "kernel"

! kernel is present in RHEL 7 with version: 3.10.0 on arch: ppc64, noarch, x86_64
What should we do about this requests?
approve, deny, pass:
}}}

So code wise we are almost there, the only thing which I just discovered to be missing is: there is no feedback mechanism.

There is no way for rel-engs to explain why they denied the package/branch creation (user not packager, package in RHEL...).
One option would be to use the review in bugzilla for this.

Thoughts?

Ok progress:

  • When an admin Denies a requests by the API or the UI, they must provide an explanatory message
  • Users can review their own requests and Obsolete it if they want
  • EPEL requests are in a pending state for a week during which people with approceacls on the package (and pkgdb admin) can set to 'Awaiting Review' or 'Blocked'
  • pkgdb-admin show requests 'Awaiting Review' (and requests in pending state automatically switch to 'Awaiting Review' after 7 days)
  • pkgdb2 blocks EPEL branch requests or new package requests for package already present in RHEL with a message inviting to request the branch/package on the rel-eng trac

From the discussion on the meeting, all the notifications (new branch request, request status change...) happens via fedmsg.

I've drawn an overview of the workflow for new branch on my blog: http://blog.pingoured.fr/index.php?post/2015/01/22/New-branch-request-process

Reviewing the changes the question popped-up: do we want the requests for Fedora branch to be automatically approved?

If yes that means that you requested a new package in rawhide and Fn, someone comes along and requests Fn-1 builds it, ships it to realize later that it just does not work on Fn-1. As bugzilla only supports 1 assignee per product, you would get the bugs of Fn-1 assigned to you although you specifically did not request it.

If not, we could use the same 7 days time period during which we use for EPEL branches, but that means more work for the rel-eng folks as they will have to process the request manually.

Thoughts on this?

Ok latest version (w/o taking into account the question in the previous comment) is up for testing in stg:

https://admin.stg.fedoraproject.org/pkgdb/

What happens if one does not select the master branch in the user interface to request a new package? Or if only the f20 branch is selected but not the f21 one. Afaik the package needs to be retired/blocked in koji if it is not meant for newer Fedora branches. But then there should be also a dead.package file in GIT for each branch, maybe with some automated text like "This is a X only package" and then substitute X with EPEL or Fedora N.

See also:
https://lists.fedoraproject.org/pipermail/devel/2015-January/207267.html

There is also a restriction missing: Currently no new branches for Fedora N-2 can be requested when Fedora N was released.

Currently no new branches for Fedora N-2 can be requested when Fedora N was released.

I can try to come up with something for this.

Fixed as part of: https://github.com/fedora-infra/pkgdb2/pull/134

The latest packagedb-cli release 2.8.1 includes the pkgdb-admin utility.

If you want to test it, you will have to point it to pkgdb in staging using the --pkgdburl and the --insecure arguments.

It's on bodhi and on pypi

to be deployed today, we will evaluate in two weeks when to shut down the old method for requesting branches.

The change is mostly made, afaics the following is still missing:

  • add a SOP about the current process for releng (on my TODO list)
  • maybe add more detailed information to the Fedora wiki about the meaning of the different states or link to the to-be-written SOP

DONE: ~~~- get the fedora-cvs flag removed from bugzilla, request opened: https://bugzilla.redhat.com/show_bug.cgi?id=1282930~~~

This is now all done I think.

Login to comment on this ticket.

Metadata