Today, gcc, gcc-c++ and make are always available in minimal build root. However, I believe that the time come and we should not install them by default anymore. There is definitely myriad of packages written in python, perl, java, ruby, go and other languages, which don't use and of the tools mentioned above. Removal of unneeded dependencies from minimal build root may also help with bootstrapping of OS.
Removal of the above mentioned packages from minimal build root could be achieved in several steps:
This was originally triggered in this ML discussion [2].
[2] https://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions_2 [1] https://lists.fedoraproject.org/pipermail/packaging/2014-December/010398.html
We talked about this a little bit during the open floor portion of the meeting today. (It came in too late to get on the schedule.)
I think there was agreement that actually changing the buildroots is way out of FPC's domain, and that anything that would require such extensive changes to the package set should go through FESCo and the feature process.
However, there's also a proposal to do what is within FPC's mandate, which is to remove the portion of the guidelines which discusses what's in the minimal buildroot and the exception related to that, and simply require that build dependencies be properly specified.
Proposal:
Remove from the guidelines at https://fedoraproject.org/wiki/Packaging:Guidelines#BuildRequires_2 all text beginning with and including the string "There are two suggested ways of detecting missing BuildRequires" and ending with (but not including) the section " BuildRequires and %{_isa}".
If we want to keep some mention of how to test in a minimal build environment, we can mention mock and probably koji (which didn't even exist when this guideline was written) or link to an external document which describes how to do test builds. (I would personally prefer the latter.)
Replying to [comment:2 tibbs]:
I was not expecting to see this discussed already, thanks :)
I am happy to open FESCo ticket it you think, but I am not sure about the change. I believe that your following proposal would be FESCo's suggestion anyway, so that is good start from my POV.
However, there's also a proposal to do what is within FPC's mandate, which is to remove the portion of the guidelines which discusses what's in the minimal buildroot and the exception related to that, and simply require that build dependencies be properly specified. Proposal: Remove from the guidelines at https://fedoraproject.org/wiki/Packaging:Guidelines#BuildRequires_2 all text beginning with and including the string "There are two suggested ways of detecting missing BuildRequires" and ending with (but not including) the section " BuildRequires and %{_isa}". If we want to keep some mention of how to test in a minimal build environment, we can mention mock and probably koji (which didn't even exist when this guideline was written) or link to an external document which describes how to do test builds. (I would personally prefer the latter.)
If you are going to drop the exceptions, please note that they are referenced also in review guidelines:
MUST: All build dependencies must be listed in BuildRequires, except for any that are listed in the exceptions section of the Packaging Guidelines ; inclusion of those as BuildRequires is optional. Apply common sense.
What is the expected benefit of this? The only one really listed is bootstrapping, but all of these tools have to already be ported and running in order to have any build of rpm running (either when building the very first RPMs, or to run any build in mock), so I can’t see how this would help.
OTOH, if I am not mistaken, dropping packages from the default buildroot would also drop them from the buildroot cache image, slowing down any build that does need any of these packages.
And philosophically, !BuildRequires was never used to specify every single dependency necessary, there was always an assumption of a “basic Linux environment for building software”. We don’t have
!BuildRequires: running-kernel, at-least-10MB-of-RAM, rpm-library-reading-this-spec-file, please-strip-ELF-modules-for-me.
(Personally I would rather, in same vague future, go in the opposite direction, defining a “The Fedora Application Development Environment” shipping everything we recommend to use as suitable and stable tools and libraries to be used by both first-party and third-party applications against, and use ''that'' as the default buildroot / implicit !BuildRequires list.
Currently we kind of have this only for shell and very minimal C and C++, but this could go in the direction of including ''much more'' rather than less (at least assuming there are “many more” suitable and stable tools and libraries…)
And, further, we could even consider an extreme deviation from status quo and freeze/stabilize the development environment separately from the next release being built (i.e. build F22 using F21 tools, or freeze languages before libraries/applications).
This is clearly not a FESCo consensus ☺ and I don’t really want to start this discussion now; but that would at least have some larger benefits for the ecosystem, unlike trimming things down which implicitly trims down what can be thought of as “the Linux programming environment”.)
Finally,
However, there's also a proposal to do what is within FPC's mandate, which is to remove the portion of the guidelines which discusses what's in the minimal buildroot and the exception related to that, and simply require that build dependencies be properly specified. Proposal: Remove from the guidelines at https://fedoraproject.org/wiki/Packaging:Guidelines#BuildRequires_2 all text beginning with and including the string "There are two suggested ways of detecting missing BuildRequires" and ending with (but not including) the section " BuildRequires and %{_isa}".
If you do something like this, please make extremely sure that this guidelines change and announcement preserves the interpretation that contents of the default buildroot don’t need to be explicitly listed as !BuildRequires. In the extreme, requiring packages to !BuildRequires: bash in the spec file, when bash is implicitly necessary to have the spec file built at all, would be just absurd.
Replying to [comment:5 mitr]:
The buildsys does not use that cache, so it's not impacted in any case.
It would result in a shorter time to init the minimal buildroot.
No such thing exists
And philosophically, !BuildRequires was never used to specify every single dependency necessary, there was always an assumption of a “basic Linux environment for building software”. We don’t have !BuildRequires: running-kernel, at-least-10MB-of-RAM, rpm-library-reading-this-spec-file, please-strip-ELF-modules-for-me.
There is an assumption and always will be that some environment already exists. adding a BuildRequires on rpm-build for instance would be silly
(Personally I would rather, in same vague future, go in the opposite direction, defining a “The Fedora Application Development Environment” shipping everything we recommend to use as suitable and stable tools and libraries to be used by both first-party and third-party applications against, and use ''that'' as the default buildroot / implicit !BuildRequires list. Currently we kind of have this only for shell and very minimal C and C++, but this could go in the direction of including ''much more'' rather than less (at least assuming there are “many more” suitable and stable tools and libraries…) And, further, we could even consider an extreme deviation from status quo and freeze/stabilize the development environment separately from the next release being built (i.e. build F22 using F21 tools, or freeze languages before libraries/applications). This is clearly not a FESCo consensus ☺ and I don’t really want to start this discussion now; but that would at least have some larger benefits for the ecosystem, unlike trimming things down which implicitly trims down what can be thought of as “the Linux programming environment”.)
We used to have this many many moons ago, It was intentionally moved away from for very good reasons. I would be strongly against going back to it. it would for one things strongly impact buildroot creation time making builds take longer.
Finally, Replying to [comment:2 tibbs]: However, there's also a proposal to do what is within FPC's mandate, which is to remove the portion of the guidelines which discusses what's in the minimal buildroot and the exception related to that, and simply require that build dependencies be properly specified. Proposal: Remove from the guidelines at https://fedoraproject.org/wiki/Packaging:Guidelines#BuildRequires_2 all text beginning with and including the string "There are two suggested ways of detecting missing BuildRequires" and ending with (but not including) the section " BuildRequires and %{_isa}". If you do something like this, please make extremely sure that this guidelines change and announcement preserves the interpretation that contents of the default buildroot don’t need to be explicitly listed as !BuildRequires. In the extreme, requiring packages to !BuildRequires: bash in the spec file, when bash is implicitly necessary to have the spec file built at all, would be just absurd.
we do have to have a minimal buoildroot. its is specified in the build group in koji and the buildsys-build group in comps. AFAIK the contents of that is controlled by the FPC though I have failed to find anything that says who is supposed to ack changes, the last of which was explicitly adding xz.
What I think is lacking is the knowledge of how many packages are effected both ways. I do not think we should remove make from the minmal buildroot, many packages use make even ones not using gcc and gcc-c++ but knowing what packages would need updating to add BuildRequires on gcc and/or gcc-c++ and what packages do not and would have a slightly shorter build time as a result.
removing gcc and gcc-c++ from the minal buildroot will not help bootstraping at all, the initial thing that prompted the email referenced was perl effecting bootstrapping of a new arch.
I am purely speculating here but given the reporters area of work, this request is purely about trying to speed up his own builds. which is understandable. but there is work underway that will give better results.
A koji task starts with creating a srpm the task installs fedpkg which has a massive amount of deps and takes quite some time to complete. releng is planning to switch fedpkg for fedpkg-minimal which is a shell script to fetch the sources and does not have a massive dependency chain.
I think to make an informed decision on removing gcc and gcc-c++ we need to have a lot more data on the impact. Additionally the call to phase things in I do not feel will work. we will get to the cutover point and a large number of packages will need to be fixed as they will not have been done so beforehand. At least based on experience. I am not opposed at making the change.
What is the expected benefit of this?
Well, I mentioned bootstrapping, others with more experience in this field disagree and the truth is that gcc is one of the first packages you need to have anyway.
But yes, as Dennis mentioned, I expect that the buildroot will be prepared faster.
I expect that some bandwith will be saved.
But mainly I expect that less packages in buildroot = less possibilities for breakage.
Currently (tried rawhide today), there is installed 164 packages, 119 M to download, 445 M to install. If single of them is broken, you might be doomed. If you need to find what is broken, you will need to consider 164 packages.
Funny enough, I tried to replace:
{{{ config_opts['chroot_setup_cmd'] = 'install @buildsys-build' }}}
in by mock config by:
{{{ config_opts['chroot_setup_cmd'] = 'install rpm-build shadow-utils' }}}
rpm-build is there obviously to be able to build package, shadow-utils to have useradd command available. And surprisingly, it still required 140 packages. That is quite unexpected. I would expect it to drop to half at minimum.
So I did my own build of RPM with disabled dependency on perl-generators (rhbz#1158860) and it still installs 111 packages, downloads 60 M and installs 286 M. One of the installed packages is bash for example, so it is still implicit available in build root. Actually, the current buildsys-build group could be reduced to rpm-build, shadow-utils, util-linux, which, gcc, gcc-c++ and make to provide the same minimal build root.
(This is OT, but since I am already playing with this, the minimal root mock can use, i.e. "install bash shadow-utils" consist of 48 packages, downloads 31 M and installs 176 M and plain bash installs 19 packages, downloads 20 M and installs 143 M)
Replying to [comment:8 ausil]:
And philosophically, !BuildRequires was never used to specify every single dependency necessary, there was always an assumption of a “basic Linux environment for building software”. We don’t have !BuildRequires: running-kernel, at-least-10MB-of-RAM, rpm-library-reading-this-spec-file, please-strip-ELF-modules-for-me. There is an assumption and always will be that some environment already exists. adding a BuildRequires on rpm-build for instance would be silly
Based on the investigation above, this is essentially the only implicit requires you need (and yeah, shadow-utils).
How to list packages which has no BR at all? This could be the amount of packages which needs review:
{{{ $ repoquery --repoid rawhide-source --arch src * | wc -l 16228 $ repoquery --repoid rawhide-source --arch src --whatrequires * | wc -l 14977 }}}
Which gives 16228 - 14977 = 1311. For every other package, I assume that the build dependencies would be pulled in transiently. Although thare are rather naive numbers/assumptions, I thought that some number might be better than no number at all :)
Additionally the call to phase things in I do not feel will work. we will get to the cutover point and a large number of packages will need to be fixed as they will not have been done so beforehand. At least based on experience. I am not opposed at making the change.
Yes, you are right, "phase things in" and "cutover" never work and despite that, we are able to achieve something and move forward. If you asked me if Fedora can have 3 flavors, I would say no and surprisingly it happened, although there were definitely issues after cutover. UsrMove and systemd were another impossible things.
We discussed this at today's meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-01-15/fpc.2015-01-15-17.00.txt):
Just for the record, there was concerns about fedora-release, redhat-rpm-config, rpm-build, etc, but when you have rpm-build available in build root, all these are already pulled in as rpm-build dependencies. And rpm-build itself can be hardly considered as something which should be specified as BR, since you need rpm-build already prior you are concerned about some BR. So dropping the exceptions section without any further notice should be just fine IMO.
I've submitted a proposal for the changes that FPC can make as https://fedorahosted.org/fpc/ticket/497. The remaining issues in this ticket (actually changing what gets into the buildroot) is out of our scope, and should probably be taken up with releng or FESCo.
Metadata Update from @tibbs: - Issue assigned to tibbs
Login to comment on this ticket.