Ticket #5214 (closed task: fixed)

Opened 23 months ago

Last modified 3 months ago

Setup koji tags and repo for Docs

Reported by: crantila Owned by: rel-eng@…
Milestone: Fedora 20 Beta Component: koji
Keywords: docs Cc: docs@…
Blocked By: Blocking:

Description

Posted here as per https://fedorahosted.org/fedora-infrastructure/ticket/3320

phenomenon

To support simpler and more robust updates to docs.fp.o, need to setup koji to handle a new tag and repo for Docs.

implementation recommendation

<Sparks> What would be required to stand up a separate instance of koji for Docs? <dgilmore> Sparks: why would you want a seperate instance? <dgilmore> Sparks: i dont see any valid reason to do so <Sparks> dgilmore: For the Docs website. Publican has the ability publish documentation from packages (separate repo from the Fedora repo) for the website. We want to replace the git repo that operates it now. <dgilmore> Sparks: and why does that need a seperate koji? <Sparks> The git repo has gotten HUGE and is becoming an issue. <dgilmore> Sparks: so, why does that mean a seperate koji instance <dgilmore> Sparks: we could use a seperate tag and targets in koji <dgilmore> i dont see why it would need its own koji <Sparks> dgilmore: It's either that or try to get everyone setup as packagers. <dgilmore> Sparks: get everyone setup as packagers <Sparks> dgilmore: Except that they really won't be packagers. <dgilmore> Sparks: though you dont need to be a a packager to get a koji cert <dgilmore> you just need fas <dgilmore> Sparks: what would be the workflow? <Sparks> dgilmore: Okay, and with that we can send packages through koji and tag them separately? <Sparks> dgilmore: Basically we just tell Publican to build the package and submit it to koji. The Publican software does all the work. <dgilmore> Sparks: I still really dont know what your trying to do. pretend im an idiot(not really that hard) and explain what it is and how it should work <Sparks> dgilmore: I'm not far off... <Sparks> dgilmore: So Publican will make an SRPM package, submit it to koji destined for a repo. Our Publican backend will install those packages and publish the data to the website. <Sparks> ...as I understand it <dgilmore> Sparks: so we would need to set up seperate tags and tagets for it, defining a koji policy allowing srpms to be built. likely we would add a new group to koji and add people allowed to build docs to it and limit access to the tags/targets to people in that group <dgilmore> Sparks: so its all doable <Sparks> dgilmore: Well that's a lot easier. :) <Sparks> dgilmore: We already have a group in FAS for people that should have access (docs-publishers). Should I open a ticket? <dgilmore> Sparks: koji doesnt know anything about fas <dgilmore> Sparks: please file a ticket. We wont be able to make changes until after f17 is done <Sparks> dgilmore: That's fine. We're not completely ready for the transition so after F17 would be good. <Sparks> dgilmore: Thanks!

Change History

comment:1 Changed 20 months ago by rlandmann

Noting that step 1 here is to have a dist-6E-epel-infra root in which we can build a more up-to-date Publican and the various deps that are too old in EL6 or EPEL.

We can't begin the packaging effort without this first step.

comment:2 Changed 15 months ago by sparks

Dennis: Could we discuss this at FUDCon?

comment:3 Changed 14 months ago by sparks

Dennis: We talked about this at FUDCon and you said that you had already completed part of this.

Has this been completed now?

Can you send details of the implementation, please?

Is there anything else needed from us?

comment:4 Changed 14 months ago by ausil

I also said i needed you to ping me on irc so we can sort out the details and make sure it is done right which has not happened

comment:5 Changed 13 months ago by rlandmann

Hi Dennis; to get started here, we will need at least these packages added to the new tag:

  • publican
  • wkhtmltopdf
  • wkhtmltopdf-qt
  • perl-DateTime?-Format-DateParse?
  • perl-HTML-Tree
  • perl-XML-TreeBuilder?
  • perl-Locale-Maketext-Gettext
  • publican-fedora
  • Fedora_Documentation-web-home
  • Fedora-Burning_ISO_images_to_disc-18-web-en-US
  • Fedora-Fedora_Live_Images-18-web-en-US
  • Fedora-Installation_Guide-18-web-en-US
  • Fedora-Installation_Quick_Start_Guide-18-web-en-US
  • Fedora-Musicians_Guide-18-web-en-US
  • Fedora-Power_Management_Guide-18-web-en-US
  • Fedora-Release_Notes-18-web-en-US
  • Fedora-Security_Guide-18-web-en-US
  • Fedora-Technical_Notes-18-web-en-US
  • Fedora-UEFI_Secure_Boot_Guide-18-web-en-US
  • Fedora-Virtualization_Getting_Started_Guide-18-web-en-US
  • Fedora-Virtualization_Security_Guide-18-web-en-US

comment:6 follow-up: ↓ 7 Changed 13 months ago by immanetize

Hey Rudi,

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Thanks, --Pete

comment:7 in reply to: ↑ 6 ; follow-up: ↓ 9 Changed 13 months ago by rlandmann

Replying to immanetize:

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

Yes; that's precisely the mechanism. The only other considerations here are that the docs packages can be made available for folk to install on their own intranets if they want, or even on their own desktops (we build subpackages of the docs for local installations)

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Long answer, because I acknlowledge that at first it looks like a lot of overkill :)

We're deliberately setting this up to be as flexible and extensible as possible; which means we need to consider potential namespace collisions for packagers, and for anyone who wants to install the documentation packages locally themselves. This is a very real risk when many of our %{name}s are as generic as "Installation Guide".

%{product} and %{version} pretty much ensure that a particular book title remains unambiguous.

We package versions and languages separately to ease maintenance. For example, a build problem in one language (because of a flaw in somebody's PO file) should never block somebody else releasing the book in their language. Also, a half-finished and unproofread translation should never get published and put on public display just because another translator finished their work long ago and is itching to release. And of course, we don't want to block the English release because not all translations are ready yet. Similar considerations apply to docs for specific Fedora versions. If a translation team only completed the translation of a particular book for F17, and has half-finished, broken translations for F18, F16, and F15, we need to be able to publish the F17 version of the book without also publishing broken translations for three other Fedora releases, plus empty translations (ie, English text with Translated headings provided by our various tools) for everything else all the way back to FC1.

%{format} doesn't require any extra repos -- we only build two formats, web and "desktop", and the "desktop" is a subpackage of -web-

Like I said, I know it looks like a lot of overkill at first, and I won't pretend that there isn't work for rel-eng to creating a bunch of new repos every Fedora release. Against that, the names of these repos are highly predictable and change only slowly from release to release; this task should be highly automatable. Also, the package naming structure implemented by Publican was designed around Red Hat's own practical experience in maintaining its own docs suite over the last ten years or so; the Publican team believes that it is "as simple as it can be, and no simpler" ;)

comment:8 Changed 13 months ago by rlandmann

And a few more packages I've just realised we'll need in that buildroot and tag:

I'm very confident that's it to get set up and running.

comment:9 in reply to: ↑ 7 Changed 13 months ago by immanetize

Hey Rudi, a few more questions, inline below.

Replying to rlandmann:

Replying to immanetize:

Can you give us a high level overview of how this will work? As I understand it, docs publishers would take content from the docs git repos and herd it into koji to be built; a server on the other side of the process will have a script that updates the docs it publishes when the repo filled by our tag gets updated packages.

Yes; that's precisely the mechanism. The only other considerations here are that the docs packages can be made available for folk to install on their own intranets if they want, or even on their own desktops (we build subpackages of the docs for local installations)

We've been discussing shipping docs in the branch repos as well, and doing so allows local utilities to parse them as well as the users. A -docs subpackage that must be downloaded from docs.fedoraproject.org *and* a document package from the f19 main repo seem redundant. RPMs only at docs.fp.o is more work for the users, who would have to get updates manually instead of inline with their other package updates.

On the packages, is there a reason to follow the form of %{product}-%{name}-%{version}-%{format}-%{lang} for package names? To me, it would seem more logical to let the package name just be %{name}. It will get tedious and burdensome to task rel-eng with churning through package names for every language, document, and release we want to publish.

Long answer, because I acknlowledge that at first it looks like a lot of overkill :)

We're deliberately setting this up to be as flexible and extensible as possible; which means we need to consider potential namespace collisions for packagers, and for anyone who wants to install the documentation packages locally themselves. This is a very real risk when many of our %{name}s are as generic as "Installation Guide".

%{product} and %{version} pretty much ensure that a particular book title remains unambiguous.

I facepalmed about half an hour after posting; of course, we have to have %{version} in the name, or we can't host multiple versions at once within the same tag.

We package versions and languages separately to ease maintenance. For example, a build problem in one language (because of a flaw in somebody's PO file) should never block somebody else releasing the book in their language. Also, a half-finished and unproofread translation should never get published and put on public display just because another translator finished their work long ago and is itching to release. And of course, we don't want to block the English release because not all translations are ready yet. Similar considerations apply to docs for specific Fedora versions. If a translation team only completed the translation of a particular book for F17, and has half-finished, broken translations for F18, F16, and F15, we need to be able to publish the F17 version of the book without also publishing broken translations for three other Fedora releases, plus empty translations (ie, English text with Translated headings provided by our various tools) for everything else all the way back to FC1.

I'm still missing something here. Versioned packages are logical, but language-specific ones not as much. If a translation has broken POs, we should be able to simply not include them. Given that we keep the release version branch as 'release ready,' for this release, we would simply limit translations committed in the origin/f19 branch to the ones that are sufficiently complete and that build. Broken languages only hold up the process for as long as it takes to identify them - which we'll have to do regardless. So, why not ship multi-lang RPMs?

%{format} doesn't require any extra repos -- we only build two formats, web and "desktop", and the "desktop" is a subpackage of -web-

Like I said, I know it looks like a lot of overkill at first, and I won't pretend that there isn't work for rel-eng to creating a bunch of new repos every Fedora release. Against that, the names of these repos are highly predictable and change only slowly from release to release; this task should be highly automatable. Also, the package naming structure implemented by Publican was designed around Red Hat's own practical experience in maintaining its own docs suite over the last ten years or so; the Publican team believes that it is "as simple as it can be, and no simpler" ;)

comment:10 follow-up: ↓ 12 Changed 13 months ago by ausil

all the packages have been added. i need to still setup a policy allowing build from srpm

comment:11 Changed 11 months ago by sparks

  • Milestone changed from Fedora 18 Alpha to Fedora 19 Final

Where are we on this?

comment:12 in reply to: ↑ 10 ; follow-up: ↓ 13 Changed 11 months ago by rlandmann

Replying to ausil:

all the packages have been added. i need to still setup a policy allowing build from srpm

Thanks ausil! I see the el6-docs tag now :)

Should I separately request branches for those packages (and dist-git repos for those that don't have them) or will you set that up as part of this ticket?

Cheers and thanks again!

Rudi

comment:13 in reply to: ↑ 12 Changed 10 months ago by rlandmann

Replying to rlandmann:

Should I separately request branches for those packages (and dist-git repos for those that don't have them) or will you set that up as part of this ticket?

Actually, scratch that; after talking to AJ last week, I think that SRPM builds will be fine for everything that we need to do here.

Could you please go ahead and allow build_from_srpm and I think we're done here! :)

Cheers Rudi

comment:14 follow-up: ↓ 15 Changed 8 months ago by immanetize

  • Milestone changed from Fedora 19 Final to Fedora 20 Beta

Is this ready? We still need to work on serving the content after Koji builds it...

comment:15 in reply to: ↑ 14 Changed 8 months ago by rlandmann

Replying to immanetize:

Is this ready? We still need to work on serving the content after Koji builds it...

True. This ticket only covers the Koji part of the work, which is independent of the rest to some degree.

comment:16 Changed 3 months ago by rlandmann

  • Status changed from new to closed
  • Resolution set to fixed

All done! Many thanks ausil -- working perfectly now :)

Note: See TracTickets for help on using tickets.