#529 Bootstrap recipes for stages 1-3 in fedora git
Closed: Fixed None Opened 8 years ago by jcapik.

During stages 1-3 of the Fedora bootstrap process three sets of files called recipes are required. The recipes contain build instructions and keywords for generating the right build order (the content is similar to the %prep, %build and %install section of the rpm spec, just stripped to the supported and required minimum) and should be a part of the fedora git so that the component owners/mainainers can fix bootstrap issues caused by the component upgrades by themselves.
Some components can have multiple recipes per stage and that's why I suggest the following naming convention:

STAGE%{stage_number}-%{recipe_filename}

Examples: \
STAGE1-glibc-headers \
STAGE1-glibc \
STAGE2-rpm \
STAGE3-redhat-rpm-config

The above would allow to automate the whole bootstrap process and to delegate the responsibility for the component bootstrap to the component owners/maintaners without interfering with other files in the git repository.

The purpose of this request is to get blessing for storing the recipes in the Fedora git.


I am not sure this is a packaging committee issue. Does this require changes to the existing packaging guidelines? Is this something with which packagers of "regular" packages are ever going to have to interact?

Your statement about "storing the recipes in Fedora git" is somewhat confusing to me. If you're going to create additional git modules or such, FPC isn't the committee that decides that.

If you're saying that existing package specs will need modification in some precise way, then we would need to look at the specification for doing that in the form of a guideline draft. We can't just tell you to go ahead without actually seeing a draft.

I believe it will require at least minor changes in the packaging guidelines in the future. With making the owners/maintainers responsible for the component bootstrap, they should modify the recipes so that the recipes follow changes made in the rpm spec. Once the packager makes changes in the package, he/she should also check whether the changes didn't affect the bootstrap process and if so, they should fix the recipes or patch the sources to strip unnecessary dependencies and dependency loops caused by the update. The packagers are often unaware of bootstrap problems they introduce with changes in the package. Moreover, I need approval/blessing for pushing the recipes to the fedora git as some owners/maintainers could take it as touching their sources without permission. If such decision is not your responsibility, then I'd like to ask you who can decide that.

One more note .... previously I wanted the maintainers to add the bootstrap recipes in the list of sources in the spec file and that would definitely lead to changes in the packaging guidelines. But later I changed my opinion as that would make the solution less flexible and packagers would have to update source rpms with each fix in the recipes even when they do not lead to changes in the content of binary packages.

+1

As a core runtime developer and glibc maintainer I think this is great idea. We need to be able to bootstrap Fedora to ensure we can build smaller more integrated versions of our products. Not to mention the continuous integration testing benefits that allow core OS changes to be tested across a complete product rebuild.

What do you mean with recipes? Multiple bash scripts that are named like that in the git repositories of the packages?

The [https://fedoraproject.org/wiki/Packaging:Guidelines#Bootstrapping current guidelines] try to solve it with the appropriate provides/requires, which should be incorporated into a possible draft for the different stages.

The problem is that it's just an idea without a proposal. I can't even figure out what jcapik actually wants to have happen here. Since we talked about this at the FPC meeting, I'll include some stuff from that discussion here.

Basically:
* If you propose modifications to existing spec files, please describe those modifications in detail.
* If you wish to upload additional files to git which are basically spec files under different names, this would be against the guidelines but we would almost certainly grant an exception given some description of just what's to be uploaded.
* If you want to upload other things to git, just go ahead. You don't need our approval for that. Honestly you don't really need anyone's approval for that if you have provenpackager privileges, but it would be good to document what you're doing somewhere. If you want that documentation to be part of the packaging guidelines (which might not be a bad idea, but isn't absolutely essential if you're only talking about doing this for a very few packages) then please write it up and let FPC know.

{{{
16:43:58 <zodbot> tibbs|w: #529 (Bootstrap recipes for stages 1-3 in fedora git) – fpc - https://fedorahosted.org/fpc/ticket/529
16:44:05 <tibbs|w> Does anyone understand this at all?
16:44:17 <tibbs|w> I mean, I understand what they're trying to do, but not at all how they plan to do it.
16:44:42 <mbooth> Is this even in our remit?
16:44:50 <mbooth> They just want a git repo?
16:44:55 <tibbs|w> It depends on how they plan to do it, I think.
16:45:37 <geppetto> I don't think they want a git repo. … I think they want to do something to a bunch of git repos. for packages
16:45:44 <tibbs|w> If they're talking about basically importing some new packages, not our business unless they want to skip the review process and FESCo decided they want us involved in that. (WHich is a discussion for a bit later.)
16:46:11 <tibbs|w> If they're indeed going to go in and make changes to existing packages to add some macros or ifdefs or something then, yeah, that's our business.
16:46:28 <tibbs|w> But so far we just have a "you have to pass it to see it" kind of thing.
16:46:53 <geppetto> I think they want to store a few extra files in Fedora git for some core packages, which will do something during RCM (or mass rebuilds? or somewhere else?)
16:47:11 <tibbs|w> It could be that they're asking if we would be amenable to accepting such a thing before they go working on it, but I doubt it.
16:47:21 <geppetto> ¯_(ツ)_/¯
16:47:38 <tibbs|w> If they just want to check other files into current packages.... they should feel free.
16:47:46 <tibbs|w> If they want to modify the spec files, please tell us how.
16:48:15 <mbooth> So we should ask all this in the ticket then
16:48:28 <tibbs|w> If they want to import specs under different names, that does currently violate the guidelines but I would be happy to give them an exception.
16:48:36 <tibbs|w> I've kind of been asking but haven't gotten many answers.
16:48:53 <tibbs|w> I will convert the questions we've come up with here into more direct questions for them.
16:49:06 <tibbs|w> There may be a bit of a language barrier here as well; I'm not sured.
16:49:12 <geppetto> I guess also point them to this discussion when asking the next question … so it's obvious we don't know what they want or why
16:49:12 <mbooth> "Moreover, I need approval/blessing for pushing the recipes to the fedora git as some owners/maintainers could take it as touching their sources without permission." -- just ask the maintainers?
16:49:53 <tibbs|w> Yeah, or just use provenpackager. I mean, if this is just some random extra files, jeez.
16:50:11 <geppetto> If they want to push some files into some core packages … I guess I'd +1 that … but they'd be much better off asking people so those maintainers know wtf those files are there for and don't just delete them
16:50:52 <tibbs|w> I don't even think we need to +1 that; it doesn't violate any existing guidelines to do that.
16:51:00 * geppetto nods
16:52:01 <tibbs|w> So, basically, if they're modifying existing specs, they should tell us how. If they're going to be uploading other things which are essentially specs, we should discuss it but I doubt we'd have any issue approving it. And if they're not doing either, then they should just go ahead.
}}}

Ok guys, I'm sorry for not being descriptive enough.

When building Fedora from scratch, we start with NOTHING. That means we don't have things like rpmbuild, etc. and therefore need the simplest possible replacement for the spec files, called recipes. They're used for cross-compiling a toolchain and building core tools till we get to the phase when we have a full featured rpmbuild. Recipes are text files containing instructions for building the basic rootfs environment without rpmbuild. Their content is similar to the %build and %install content, but it can differ from the spec content a lot (depending on the package). We take the sources and patches from .src.rpm packages, but cannot build it the regular way till we get to the stage3.

Example of recipe:
https://jcapik.fedorapeople.org/fedora-bootstrap/misc/glibc-headers

We'd like the recipes to be stored in the fedora git along the spec files and package sources and you're right, as a proven packager I don't need to ask anyone and commit it directly, but ... 1.) I believe it's better to ask, even when the number of recipes is quite high (~300) and asking the maintainers for permission is always easier with blessing from some authority 2.) Some of the maintainers might be unresponsive 3.) Some of the maintainers could disagree with storing the extra files in the git AND THIS IS WHERE WE NEED YOUR HELP. We need to make it official.

Storing the recipes in the git is just the first step and we're currently asking only for this. In the future and depending on the project results we'd like to make additional proposals to extend the responsibility of the package maintainers to cover also the recipes modification.

Few more notes about the project motivation ... the complete bootstrap is required each time a new architecture appears. We "recently" had to do that in case of armv7hl, aarch64, ppc64le and soon we'll have to do that for mips. As the distribution grows and evolves, the bootstrap process becomes extremely painful and it's harder and harder to keep it sane and working due to a missing test system and automated reports. It took many months to introduce the above architectures due to the fact, that the process is not fully automated. The whole procedure needs to evolve with the package sources. Without keeping the bootstrap process sane, we could soon get to a point where we'll be unable to bootstrap without patching the sources heavily due to hard-coded dependency loops and huge trees of unnecessary deps. Such issues appear randomly in various components because the maintainers often do not care about things like bootstrap (and someone even doesn't know that things like bootstrap exist). By making the recipes a part of the git repos we'll make the maintainers aware and it should remind them that they need to keep the component flexible and modular. Moreover, a partial bootstrap is needed at least once a year with interface changes caused for example by perl updates, etc.

We do understand what bootstrapping is and the challenges you face in doing it. Our questions were purely about the details of what these things look like.

To me, this isn't any different than, say, a shell script I might check in to generate a tarball from a git checkout. While it might be nice to standardize those, it's certainly not required, and I don't see how the stuff that you want to do differs materially from that.

So, please, just move forward. If you would like to document these things (at any level of detail, including just saying what filenames to use) please feel free to whip up a draft. If this is going to get into a whole bunch of packages and these things will need to be written by regular packagers then I would suggest that at least some basic docs be put together relatively soon but I don't think it's essential to have that before you proceed.

So please, just go ahead and come back in a bit when you have all of the details worked out.

Ok, thanks.
I'm going to push the recipes to the Fedora git and once the solution gets stable, I'll write a draft and get back to you.

Metadata Update from @tibbs:
- Issue assigned to tibbs

7 years ago

Login to comment on this ticket.

Metadata