#620 Perl Build-Requires Packaging guidelines update
Closed: Fixed None Opened 7 years ago by ppisar.

In order to remove perl from minimal build root, it's necessary to remove perl-generators (RPM dependency generators written in Perl) from minimal build root. Therefore any package that delivers a Perl code and that wants to benefit from automatically generated Perl dependencies must build-require perl-generators explicitly. I believe updating Perl Packaging Guidelines is the best way how to to deliver this knowledge to Fedora packagers.

I propose these new [https://fedoraproject.org/wiki/PackagingDrafts/Packaging:Perl/BuildRequires] Perl Packaging Guidelines.

The key change is the ''Build-Requires'' section. The draft page is based on the current guidelines. You can use wiki history difference tool to see the changes from "10:58, 7 April 2016‎ (Copied from Packaging:Perl on 2016-04-07)" commit.

You can see [http://lists.fedoraproject.org/archives/list/perl-devel@lists.fedoraproject.org/message/QDPPEZ32JWR4WSSJ6XR2XHGCAN3WAGUT/ discussion on perl-devel mailing list].

The alternative solution how to avoid from build-requiring perl-generators mentioned on the mailing list was to use rich dependencies (currently forbidden) or force every package to build-require perl even though the perl is not needed at build-time (that goes against the idea of smaller build root and more modular distribution).

The proposed guidelines also documents role of some other Perl packages related to packaging.


Why not just have perl require perl-generators?

Because perl is /usr/bin/perl. Nothint more. In the ideal future. perl-generators is for package developers while perl is for normal users. The same reason why RPM macros are not delivered with perl.

Why not add the dep on the generators to perl-devel to mirror pretty much every other package? The package description for perl-devel even says "Most perl packages will need to install perl-devel to build."

+1, except for the fact that most of my perl packages do not require perl-devel.

It's true that noarch perl packages can get away without a build dep on perl-devel, but if we're going to add a new dep to every perl package, it might as well be the obvious one. The only reason I can think of not to do it that way is if perl-devel has additional dependencies which would be unwanted for the majority of perl packages.

I recommend to read the linked mailing list thread to not repeat the same questions and answers here.

perl-devel sounds reasonable until one figures out that most of the Perl packages are noarch packages that do not need any perl-devel files (header files, etc.) for building.

Now, it does not make much difference, because perl(ExtUtils::!MakeMaker) build-required by majority of Perl packages pulls in perl-devel transitively (to support building architecture-specific XS modules). But [https://bugzilla.redhat.com/show_bug.cgi?id=1129443 this dependency already has been criticized] and it's likely it will be transformed into a Recommended or Suggests level. That's the reason why the draft urges all non-noarch packages to build-require perl-devel.

Last but not least, there are non-Perl packages that installs some Perl scripts and do not use the perl or any Perl module at build time. These also haver to require Perl generators somehow.

I understand the strong unspoken desire we-want-spec-files-comptiable-with-EPEL but with this attitude we will never improve anything in Fedora packaging.

I strongly believe each needed feature should be explicitly declared. Only this way we can decompose Fedora safely without fear of regressions caused by hidden interdependencies and side effects that nobody can comprehend. Therefore the draft separates perl-generators from perl-devel from perl from perl-libs. Each of these packages have different purpose.

I should have commented earlier but I'd prefer the build dependency on perl-generators to be optional. This would:

a) add more flexibility to perl packaging and wouldn't automatically label everyone who doesn't want this feature for any valid reasons as noncompliant with the guidelines, and

b) would probably better communicate that buildrequiring perl-generators actually adds value -- you can either do that or, if you choose, require all your perl dependencies manually (and, of course, maintain that list of deps).

Replying to [comment:7 psabata]:

I should have commented earlier but I'd prefer the build dependency on perl-generators to be optional.

That's possible. Something like replacing:

  • perl-generators – It automatically generates run-time Requires and Provides for installed Perl files. Whenever you install a Perl script or a Perl module, you must build-require the perl-generators. (This does not apply to EPEL ≤ 7 where the functionality is provided by rpm-build.)

with:

  • perl-generators – It automatically generates run-time Requires and Provides for installed Perl files. Whenever you install a Perl script or a Perl module, you must either build-require the perl-generators or list all the provided and required Perl dependencies manually. (This does not apply to EPEL ≤ 7 where the functionality is provided by rpm-build.)

Could we possibly encapsulate any necessary dependencies which will be needed by all Perl packages into a macro? This could conceivably provide specfile compatibility with EPEL (without having a pile of ifdefs).

We could also just have a dummy perl-generators package on EPEL.

But I still honestly don't understand why we can't do this in a way that doesn't require changes to every single Perl package. I understand the argument about requiring what you need, but we're talking about exactly two scripts that live in a directory which already exists on every Fedora system. The perl package itself could just own these with essentially no overhead and no requirement to change every package.

The bigger issue is silent breakage for packages which install perl scripts but don't "need" perl to build. If they expect dep generation then they now won't get it. I think I'd want to see someone at least try to locate these before any change is actually made.

There will be macros for easier packaging CPAN distributions into Fedora/EPEL. But they will be optional add-ons documented as a best practice. Not a mandatory guidelines. I want to standardized a minimal contract. Not a full-fledged macro monster that will be again unmaintainable and everybody will fear to touch it because it would break compatibility. The macros will provide a versioned API to overcome the backward compatibility issue. If you are interested in the macros see [http://thread.gmane.org/gmane.linux.redhat.fedora.perl/150073]. But that's completely different topic.

(If I see how difficult is for FPC to even digest this tiny draft that actually only codifies current state of affair as the main Fedora guidelines already state "list all dependencies". I do not want to go through other more complex macros now.)

But I still honestly don't understand why we can't do this in a way that doesn't require changes to every single Perl package.

If you introduce a macro wrapping the dependencies, you will still have to modify each spec file to call the new macro. I cannot see any difference.

There is no way how to hide the packing complexity into one magical macro. If we do that, than the packing anarchy that is on Fedora level would move to Perl level, but it would remain there. One macro bundling all the dependencies is as bad as one build-require bundling all the dependencies. Actually it's worse because now you have another level of abstraction that you have to maintain for eternity and that packagers must comprehend.

The draft covers all possibilities while still giving freedom to depend on the only minimal package set really needed.

Please bring it to FPC agenda instead of trying to tell me that no changes are the best changes. I wouldn't proposed any changes if they were unnecessary.

I could now remove the perl-generators dependency from rpm-build without updating guidelines. That would break a tenth of Fedora. Then I would mass-fix all packages. Then Ralf would yell to me that I fixed his packages. And finally everything would be all right. Only new packagers would find Perl guidelines unhelpful because they would not match the reality. This week I accidentally removed perl-devel and that broke hundreds of arch-specific packages. Removing perl-generators would break thousands of packages.

So please go ahead and vote about the draft on the FPC meeting. I only want to be polite and give people a clear guidance and time to absorb it.

The bigger issue is silent breakage for packages which install perl scripts but don't "need" perl to build. If they expect dep generation then they now won't get it. I think I'd want to see someone at least try to locate these before any change is actually made.

Intersection between source packages that do not build-require perl or perl() and source packages whose binary packages require perl().

Probably some are here [https://apps.fedoraproject.org/koschei/groups/perl-users]. For example globus-common or ldns. Many of them actually should build-require some perl stuff but they just forgets it and we did not fixed them yet. Partially because their maintainers claim "till it builds, don't touch it".

I do not want Fedora to be built on accidentally building packages. I want Fedora where packagers can rely on guidelines, so they can write solid packages that work.

Did you intend for your second paragraph to be incredibly insulting? FPC isn't just supposed to rubber-stamp everything that comes along, and if you weren't prepared for discussion and disagreement, then I'm not sure why you would propose any changes in the first place. Your ticket didn't contain sufficient information to answer some of the obvious questions and so you seem to be frustrated that those questions are being asked.

I brought up macros because there was a question of breaking single-spec compatibility with EPEL and I thought it might be a possible solution. If you're considering macros as part of a larger plan, then great. Also note that I try to read perl-devel but I'm sure you know how useless that list is for any type of discussion.

Really, you just need to drop the passive aggressivity and understand that:

  • reasonable people are going to have issues with requiring changes to every single Perl package
  • that there's a problem with silent breakage of packages
  • FPC folks are at least going to want to see your plans for dealing with breakage resulting from your changes.

This isn't just about changing a couple of lines in the packaging guidelines. You should have included information on potential fallout in your original message, and at least summarized the mailing list discussion. A few minutes for you, who understands the background, versus half an hour for each of us.

And yes, Ralf may object. In fact, I know he does. Getting him to articulate his objections will be difficult. (I tried on IRC yesterday.) That doesn't mean you can completely discount his input and is no reason to simply antagonize him.

So, in any case, I think the majority of FPC just wants to know that there are plans in place for:

  • Fixing up the existing Perl packages which are reasonable and follow the usual protocol of making announcements and giving packagers a heads up to do what they need to do.
  • Finding a reasonably complete set of packages where silent breakage will occur and making sure those maintainers know about the changes.

I personally don't think that a requirement to change all packages is a deal breaker but it's so trivially avoided that the arguments for forcing it become difficult to accept. The silent breakage is far more problematic and really needs to be addressed.

Replying to [comment:11 tibbs]:

Did you intend for your second paragraph to be incredibly insulting?

Frankly, yes. I will try to be more constructive.

I brought up macros because there was a question of breaking single-spec compatibility with EPEL and I thought it might be a possible solution.

Instead of macros, one can add a dummy perl-generators package into EPEL that will require rpm-build. Or even adding a Provides: perl-generators into some package that is already in the EPEL build root would be sufficient. If you already has an EPEL-specific package where to put macros, you could simply put the Provides into that package.

I consider single-spec compatibility an illusion. You have different init system, you have different macros, you have different file layout, you have different APIs. We have hardworking Paul Howarth in the Perl and he makes his packages compatible to old EPELs. It requires many macro conditions and patches. I believe you would not want to have all the Fedora packages like that.

Other options would be not to put any macros into EPEL and add them only to Fedora. How should the macro look like?

{{{
%perl_generators \
%if 0%{?fedora} \
Build-require: perl-generators \
%endif
}}}

And then used like?

{{{
%{?perl_generators}
}}}

Adding these macros makes Fedora fatter. It's not EPEL that's becoming compatible. It's Fedora that is being made compatible to EPEL. I think it's unhealthy and it prevents Fedora from bigger changes. Do you understand that this approach means the macro would be necessary in Fedora (because of EPEL, not because of Fedora) to end of life of RHEL-7? And this expand-if-exists is dangerous. One typo and you will break the package without noticing.

Also note that I try to read perl-devel but I'm sure you know how useless that list is for any type of discussion.

You don't have to follow the list with tons of notifications. The thread pointed from the first request is enough. There is no other thread related to this topic.

  • reasonable people are going to have issues with requiring changes to every single Perl package

I cannot see how to remove perl from build root without touching every single Perl package. This was discussed in the perl-devel list ad nauseam without any bullet-proof solution:

  • Require perl-generators from perl (or perl-libs) if rpm-build is installed using rich dependencies. Smart approach, but rich dependencies are forbidden, it does not cover packages that do not need perl when building. Covering them by an exception (if you don't build-require perl, build-require perl-generators explicitly) keeps a bad taste of schizophrenia when some times you have to and some times you don't have to. I don't like these automagic rules.

  • Bundling perl-generators into perl. Forcing everybody to install a developmental tool whenever they install a perl does not seem right to me. It does not cover the non-perl packages too. And what if somebody comes with an revolutionary perl-generators written in Java. (We already have smarter generators, see tangerine package, but we do not use them at build-time because of their deep dependency chain and relative slowness.)

  • Thus I think an explicit dependency is the most versatile solution. It's perfectly fine for other ecosystems to build-require their build-time developmental tools. Why the Perl should be an exception?

  • that there's a problem with silent breakage of packages

This can happen any time a package splits or we update the generators. But Perl has a monitoring [https://ppisar.fedorapeople.org/perl_rebuild/scratch/latest/index.xhtml#done]. The linked one monitors only packages the run-require perl(:MODULE_COMPAT_). We can run a scratch rebuild specifically for all packages that run-require any perl(). Actually we do that on a representative subset of packages before we push upgraded generators.

  • FPC folks are at least going to want to see your plans for dealing with breakage resulting from your changes.

The plan is to fix them. If we notice a binary package lost perl() dependencies, we add the missing !BuildRequires: perl-generators.

Actually there should be no breakage. Because the plan is to add the !BuildRequires: perl-generators into every package that run-requires perl(*) and that does not build-require the generators yet before removing the perl-generators from the rpm-build.

This preventive step has already been documented in the Fedora change page [https://fedoraproject.org/wiki/Changes/Build_Root_Without_Perl]. I thought I mentioned it here in FPC. My fault. I'm so sorry for the confusion.

In other words, we fix the spec files first, then we remove the perl, then we check nothing broke. And if it did, we fix it.

This isn't just about changing a couple of lines in the packaging guidelines. You should have included information on potential fallout in your original message, and at least summarized the mailing list discussion. A few minutes for you, who understands the background, versus half an hour for each of us.

I thought the few lines in the original request is enough and that you prefer reading the perl-devel discussion because my summary could be considered biased.

And yes, Ralf may object. In fact, I know he does. Getting him to articulate his objections will be difficult. (I tried on IRC yesterday.) That doesn't mean you can completely discount his input and is no reason to simply antagonize him.

Ralf has a lot of experience. But his obsession with evil Red Hat is funny.

So, in any case, I think the majority of FPC just wants to know that there are plans in place for:

  • Fixing up the existing Perl packages which are reasonable and follow the usual protocol of making announcements and giving packagers a heads up to do what they need to do.

Exactly, that's why I want FPC's blessing first.

  • Finding a reasonably complete set of packages where silent breakage will occur and making sure those maintainers know about the changes.

I will do some computations and paste the package list / statics here.

I personally don't think that a requirement to change all packages is a deal breaker but it's so trivially avoided that the arguments for forcing it become difficult to accept.

I have the problem how to explain people that if they want only perl, they must install perl-generators.

While I don't like the idea of a simple macro that hides difference between Fedora and EPEL, I understand that it looks appealing to other people. Could you elaborate how it should be used? See the %perl_generators I wrote above for an inspiration.

Here are the numbers:

  • Rawhide contains 3169 source packages that build-require perl or perl(*).
  • Rawhide contains 3280 source packages whose binary package run-requires or provides perl(*).

The first number represents packages whose spec files are aware of Perl inside.
The second number represents packages that deliver some Perl code that was discovered by perl-generators. (Or where a maintainer specified the dependencies manually. It's not possible to distinguish without parsing builr.log.)

The difference between second and first set are packages that will miss generated perl(*) dependencies, if they do not build-require perl-generators explicitly. The number is 384 packages. (One has to discount perl from the number because perl obviously does not build-require perl.)

I will not post the list here as it's quite long. An Example is gnulib spec file that produces gnulib-devel binary package. Another example is systemtap spec file that produces systemtap-testsuite and systemtap-client binary packages.

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2016-04-28/fpc.2016-04-28-16.00.txt):

...we discussed this for a long time, so you might want to read the whole thing from the logs. But the above is the short version.

Tangentially related, perhaps, but if Perl is removed from the buildroot, should this section be modified?

https://fedoraproject.org/wiki/Packaging:Guidelines#Scripting_inside_of_spec_files

It doesn't need to be removed, of course, given the sentence at the end. I'm just not sure how far we're trying to go with this.

Replying to [comment:15 james]:

  • ACTION: Cut down on the number of different packages needed for
    building. Simplify the BR section to cover the 3 cases: 1. perl not
    required to build, but need generators. 2. perl required for build.
    1. perl-devel required for build. (geppetto, 17:11:11)

I cannot understand what exactly you want now. Does that mean you want to squeeze all the dependencies into one dependency symbol? Like this:

  1. case: perl-generators will run-require perl-macros and perl-srpm-macros. Spec files will declare only:

{{{
BuildRequire: perl-generators
}}}

  1. case: perl will run-require perl-generators. Spec files will declare only:

{{{
BuildRequire: perl
}}}

  1. case: perl-devel will require perl. Spec files will declare only:

{{{
BuildRequire: perl-devel
}}}

Does FPC understand that it will mean that users – not only packagers – will get perl-generators into their systems whenever they install the perl?

  • ACTION: Also need to know why we have perl-generators and
    perl-rpm-build-perl (geppetto, 17:15:04)

perl-generators are dependency generators used in Fedora when building Perl packages. perl-rpm-build-perl is a random CPAN distribution that coincidentally does the same think. Not the only one.

...we discussed this for a long time, so you might want to read the whole thing from the logs. But the above is the short version.

perl-generators and perl-macros are two different projects with two different upstreams. We also have perl-!DepGen-Perl-Tests that delivers integration tests. perl-macros will grow in the future. We can put dependency between them or try to merge them in the future.

Replying to [comment:16 tibbs]:

Tangentially related, perhaps, but if Perl is removed from the buildroot, should this section be modified?

https://fedoraproject.org/wiki/Packaging:Guidelines#Scripting_inside_of_spec_files

It doesn't need to be removed, of course, given the sentence at the end. I'm just not sure how far we're trying to go with this.

I think it's completely independent. I would remove the listing and documented only the Lua interpreter bundled into the rpm-build as it is not well known.

Replying to [comment:17 ppisar]:

Replying to [comment:15 james]:

  • ACTION: Cut down on the number of different packages needed for
    building. Simplify the BR section to cover the 3 cases: 1. perl not
    required to build, but need generators. 2. perl required for build.
    1. perl-devel required for build. (geppetto, 17:11:11)

I cannot understand what exactly you want now. Does that mean you want to squeeze all the dependencies into one dependency symbol? Like this:

No, that's not what we want at all. We'd appreciate it if you could attend the next FPC meeting to avoid such misunderstandings. We're simply asking you to list all BRs required in the 3 cases enumerated above. If we missed another use-case, please also tell us.

  • ACTION: Also need to know why we have perl-generators and
    perl-rpm-build-perl (geppetto, 17:15:04)

perl-generators are dependency generators used in Fedora when building Perl packages. perl-rpm-build-perl is a random CPAN distribution that coincidentally does the same think. Not the only one.

The names are confusing. Is perl-rpm-build-perl used anywhere in Fedora? Aren't you reinventing the wheel in perl-generators? The perl-generators name doesn't reflect its purpose clearly.

...we discussed this for a long time, so you might want to read the whole thing from the logs. But the above is the short version.

perl-generators and perl-macros are two different projects with two different upstreams. We also have perl-!DepGen-Perl-Tests that delivers integration tests. perl-macros will grow in the future. We can put dependency between them or try to merge them in the future.

That's what we meant when asking to reduce the number of dependencies to be explicitely listed. It seemed to us that perl-macros and perl-generators are required in each and every case, so it might make sense to either merge them or at least introduce a dependency between them.

Replying to [comment:19 rathann]:

We'd appreciate it if you could attend the next FPC meeting to avoid such misunderstandings.

Tomorrow 17:00 UTC? I will try to come.

We're simply asking you to list all BRs required in the 3 cases enumerated above.

So you want examples how to apply the rules from the guidelines.

  1. case:

{{{
BuildRequires: perl-generators
}}}

  1. case:

{{{
BuildRequires: perl-generators
BuildRequires: perl
}}}

  1. case:

{{{
BuildRequires: perl-generators
BuildRequires: perl
BuildRequires: perl-devel
}}}

That assumes you ignore the perl-macros and perl-srpm-macros packages as the latest draft does. This can be solved run-requiring them from perl-generators.

The perl-generators exists for running Perl dependency generators, the perl exists for executing perl from a spec file ({{{perl Makefile.PL}}} line in %build section), the perl-devel exists for sources including Perl headers ({{{#include "perl.h"}}} line in *.xs source files).

perl-generators are dependency generators used in Fedora when building Perl packages. perl-rpm-build-perl is a random CPAN distribution that coincidentally does the same think. Not the only one.

The names are confusing.

The perl-rpm-build-perl name comes from perl- prefix for Perl packages and rpm-build-perl from [http://search.cpan.org/dist/rpm-build-perl/ upstream name].

The perl-generators name comes from perl- prefix and a name not to confuse. Original idea, If I remember correctly, was perl-rpm-build because the files were moved from rpm-build package, but that would be too similar to perl-rpm-build-perl. At the and perl-generators contains only the generators. No macros or other auxiliary packaging files.

Is perl-rpm-build-perl used anywhere in Fedora?

I don't know. Is that relevant?

Aren't you reinventing the wheel in perl-generators?

No. The code was already in the rpm-build. All the others including perl-rpm-build-perl reinvent the wheel. They try to become better dependency generators, but at the end none of them managed it. They also use different approach. While perl-generators is a static analyzer, perl-rpm-build-perl is a dynamic analyzer. Each of them has its pros and cons but we still believe that the perl-generators are superior when considering accuracy, speed and dependency chain.

The perl-generators name doesn't reflect its purpose clearly.

That's a matter of taste. Do you have a better name?

That's what we meant when asking to reduce the number of dependencies to be explicitely listed. It seemed to us that perl-macros and perl-generators are required in each and every case, so it might make sense to either merge them or at least introduce a dependency between them.

We could introduce perl-build (or perl-devel-noarch) package that would run-require perl-generators, perl-macros, and perl-srpm-macros. So the initial three examples would look like:

  1. case:

{{{
BuildRequires: perl-build
}}}

  1. case:

{{{
BuildRequires: perl-build
BuildRequires: perl
}}}

  1. case:

{{{
BuildRequires: perl-build
BuildRequires: perl
BuildRequires: perl-devel
}}}

But is the name really so important so that we have to create yet another package or rename the package?

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2016-05-05/fpc.2016-05-05-16.00.txt):

  • 620 Perl Build-Requires Packaging guidelines update (geppetto,


    16:05:33)
  • LINK:
    https://fedoraproject.org/wiki/Packaging:Guidelines#Devel_Packages
    doesn't specify any kind of dependency from *-devel to the base
    package. (tibbs|w, 16:13:13)
  • ACTION: Perl Build-Requires Packaging guidelines update (+1:6, 0:0,
    -1:0) (geppetto, 16:16:13)

Announcement text:

The Perl guidelines were modified to include information about necessary build dependencies now that Perl itself is being removed from the default buildroot.
https://fedoraproject.org/wiki/Packaging:Perl
https://fedoraproject.org/wiki/Packaging:Perl#Build_Dependencies
* https://fedorahosted.org/fpc/ticket/620

The new guidelines are missing all the changes from the draft. Especially they keep the warning block:

Do not explicitly buildrequire "perl-devel"

Explicitly requiring perl-devel, even when wrapped in a conditional construct, is strongly discouraged, and is generally considered a blocker at review and a packaging bug. Instead, see the next section on requiring core modules -- making sure that these core modules are BR'ed when used will pull in the correct development Perl packages.

that does contradict the new rule about a need for build-requiring perl-devel.

I recommend to remove the warning block.

You're right, I somehow missed the removal of that bit when incorporating the diff. I've fixed it up now. Please let me know if I missed anything else.

It's much better. Thank you.

Metadata Update from @james:
- Issue assigned to tibbs

7 years ago

Login to comment on this ticket.

Metadata