#4564 Dist-git for Copr
Closed: Fixed None Opened 9 years ago by msuchy.

We would like to have dist-git for Copr (to have ability to upload src.rpm to Copr). I got recommendation from various stake-holders to not share it with current Fedora instance. So we need another instance.

Where I can set it up? Do we some bare metal? Or should I create new VM in persistent tenant of Fedora Cloud?


Well, lets try and gather/be clear on requirements here first. :)

Can anyone make new git repos?

Do fedora/epel releases have branches like we do for pkgs git?

Who can commit to the repos?

I assume source would be in some lookaside? Or would copr fetch it from upstream? or ?

Would you need bug tracking, etc?

How similar will you want this to what what have already for pkgs.fedoraproject.org setup?

Would something like progit ( https://github.com/pypingou/ProGit has a link to dev instance) be a good fit for this?

Perhaps we should describe use cases and figure out what we best need to meet them?

Can anyone make new git repos?

Tricky question :)
Thinking about it, I would say that likely no. I would propose following scenario:
When package in this project does not exists yet, Copr will put this information in its DB.
Dist-git machine will pool Copr via API for those request (similar to process-git-requests) and create them. This can be run e.g. ever 30 - 60 seconds.
This mean Copr will be responsible for correct data. And only some process on dist-git machine will be creating new git repos.

Do fedora/epel releases have branches like we do for pkgs git?

No. Copr is all in one. If you have separate src.rpm you should use different project.
Branches should be named <FAS_login>/<project_name>

Who can commit to the repos?

Owner. (let start with this). And then other FAS accounts which have in Copr "builder" permission for this project.

I assume source would be in some lookaside? Or would copr fetch it from upstream? or ?

Lookaside.

Would you need bug tracking, etc?

No. Not primary goal. But if we deploy ProGit then we can have issue tracker for free.

How similar will you want this to what what have already for pkgs.fedoraproject.org setup?
Would something like progit ( ​https://github.com/pypingou/ProGit has a link to dev instance) be a good fit for this?

Both are nice-to-have. But not really important. We can live without it. But if I would ever bored, I try to deploy ProGit.

Perhaps we should describe use cases and figure out what we best need to meet them?

Required:

  1. User execute

foo copr-build msuchy/bar some.src.rpm

where foo is either copr-cli or fedpkg. This will import some.src.rpm into dist-git in branch msuchy/bar (only if user is owner of this project or have "builders" permission).

  1. Copr builder is able to clone dist-git and build src.rpm from this git.

Optionally:

  1. User is able to clone/commit/push git of that package.
    (I can imagine this can be done using fedpkg --config /etc/rpkg/copr.conf, where some commands will not have sense e.g "scratch-build")

Have you spoken to the fedpkg maintainer(s) about if they are willing to add copr git handling?

So, the copr user will only still send copr src.rpms? (unless fedpkg is expanded).

say I sent 'foo-1.0-1.src.rpm' and have copr build it. It makes a kevin/foo/ git repo and unpacks the rpm there and uploads the source to lookaside? How does it know what is 'source' ?

Then I send 'foo-2.0-1.src.rpm' and have copr build it. Does it unpack this over the existing foo? How does it know to delete or remove files? Only things present in the most current src.rpm?

For use cases I was looking for higher level. How does having git help them out if they are only ever sending src.rpms? Is the idea to improve collaboration? or to replace people using github? or ?

It seems like 'dist-git' and 'fedpkg' are maybe the wrong places to integrate copr. It seems to me like a much-improved fedorahosted setup (something more like github, but FOSS and hosted in infra) would be a better fit.

Then, we could teach the copr client tools to know how to use those accounts/repos as an input for new copr builds.

For automation, we could teach copr to listen for fedmsg messages from this future-improved-fedorahosted to kick off new rebuilds.

Have you spoken to the fedpkg maintainer(s) about if they are willing to add copr git handling?

Not yet. But if they will hesitate, we can leave it in copr-cli.

Then I send 'foo-2.0-1.src.rpm' and have copr build it. Does it unpack this over the existing foo? How does it know to delete or remove files? Only things present in the most current src.rpm? How does it know what is 'source'?

Yes, the same as 'fedpkg import' is working now.

How does having git help them out if they are only ever sending src.rpms?

They can clone that git, see revision of specs. Collaborate, contribute. Right now, you submit just url of src.rpm and usually you will delete it then. So if you repeat that build, it will fail, because that url is unavailable.

One of the issues we are trying to address, is that Centos guys hesitate to rebuild SCL for Centos, because there is no sources, no upstream. And the sources are not those tar.gz (as they are same as for normal packages), but rather those spec files, which evolve continuously.

And yes, we can put those src.rpm simply in some directory on fedorahosted. But that would be waste of space, as each new release (with same version) would store the same tar.gz inside of new src.rpm file. Or we would have to delete old ones, which would not preserve history.
Or we can invent something completly new, but I would like to avoid that. As I would like to converge to either Koji or OBS.

Have you spoken to the fedpkg maintainer(s) about if they are willing to add copr git handling?
Not yet. But if they will hesitate, we can leave it in copr-cli.

Looking at the code. It seems that fedpkg is not best target anyway. Most of the code is in pyrpkg and fedpkg is just small layer above this. Imho it would have more sense to create coprpkg as small layer above pyrpkg.

So, we never really finished deciding anything here. ;)

Options I guess:

dist-git for copr as it's own thing in the cloud.

  • It's going to take a lot of space. Every built rpm unpacked into git/lookaside.
  • How do people push to it? with copr token?
  • There would be no bug tracker/pull requests/etc, it would all be email or whatever to the owner of the copr.
  • If we tell people they can depend on this as an upstream source it makes it in my mind something we need to make a good deal more available. Mirroring. Monitoring. etc.

fedora hosted.

  • We could still use this, with git repos and the like, but would have to come up with a sync method. It could give projects trac space, etc.
  • old and kinda clunky.

pagure (used to be progit)

  • IMHO this is the best bet, but it's not quite ready for this.
  • We could even do a seperate instance for copr thats different from the main pagure one.
  • This would give people git repos, PR's, etc.

Perhaps we should discuss more on list or in irc?

Replying to [comment:6 msuchy]:

Have you spoken to the fedpkg maintainer(s) about if they are willing to add copr git handling?
Not yet. But if they will hesitate, we can leave it in copr-cli.

Looking at the code. It seems that fedpkg is not best target anyway. Most of the code is in pyrpkg and fedpkg is just small layer above this. Imho it would have more sense to create coprpkg as small layer above pyrpkg.

I've created https://github.com/release-engineering/dist-git/issues/2 - proposal to handle this without adding more rpkg wrappers specific for distro or buildsys.

This has been already deployed to production. Therefore I think we can close it now.

Login to comment on this ticket.

Metadata