#76 Mention new filtering mechanisms in AutoProvidesAndRequiresFiltering
Closed: Fixed None Opened 12 years ago by tibbs.

rpm 4.9.0 has new (and I presume better) mechanisms for filtering dependencies. My understanding is that these don't suffer from the problems our existing macros have with respect to disabling the internal dependency generator and the issues surrounding that.

However, I can't find any documentation at all on this besides this message: http://lists.fedoraproject.org/pipermail/devel/2011-February/148317.html

Since F15 has a sufficiently new RPM, it may be worth mentioning these on https://fedoraproject.org/wiki/Packaging:AutoProvidesAndRequiresFiltering

It also occurs to me that at least %perl_default_filter could be updated in F15+ to use the new-style stuff.


Figured I should mention that the current draft is at https://fedoraproject.org/wiki/User:Tibbs/AutoProvidesAndRequiresFiltering. Open questions that need answers are prefixed in the draft with "XXX".

Could you please add examples of filtering into documentation and mention that the old one (sometimes) doesn't work? If filters for more requires are used, then it's needed |. Also regexp can't use perl regexps, but only extended regexps.

For example Perl:
/etc/rpm/macros.perl must be changed to something like:
{{{
%perl_default_filter %{expand: \
%global __provides_exclude_from %{perl_vendorarch}/auto/..so|%{perl_archlib}/..so|%{_docdir}
%global __requires_exclude_from %{_docdir}
%global __provides_exclude perl\\(VMS|perl\\(Win32|perl\\(BSD::|perl\\(DB\\)|perl\\(UNIVERSAL\\)
%global __requires_exclude perl\\(VMS|perl\\(BSD::|perl\\(Win32
}
}}}

If there is needed another filtering in module, it must be used in this way:
{{{
%perl_default_filter
%global __provides_exclude %{?__provides_exclude}|perl\(DBI\)
%global __requires_exclude %{?__requires_exclude}|perl\(RPC::
}}}
All requires or provides must be on one line, otherwise it won't work.

After playing around with this for a bit, I've discovered that literal parentheses need 3 backslashes, not 4 or 2 as in the above examples; e.g.:

%global __requires_exclude ocaml\(Sig\)

This is going to mean trouble for macros that expand to strings containing non-literal POSIX extended regexp characters, like +, (, ), ., etc. A "raw string" format would both solve that problem and make the example above more readable. The language does appear to be extended POSIX regexps; I see references to the symbols "regcomp" and "regexec" in librpm, FWIW.

Also, either %global or %define can be used, so %global should be preferred.

Replying to [comment:3 mmaslano]:

If there is needed another filtering in module, it must be used in this way:
{{{
%perl_default_filter
%global __provides_exclude %{?__provides_exclude}|perl\(DBI\)
%global __requires_exclude %{?__requires_exclude}|perl\(RPC::
}}}

I think this is a potentially dangerous recommendation. If %!__provides_exclude (or requires, of course) isn't already defined, this expands to
{{{
%global __provides_exclude |perl\(DBI\)
}}}
And due to the | operator, ends up matching (and excluding) everything. To avoid this happening, I would recommend being more explicit in the spec file
{{{
%global __provides_exclude %{?__provides_exclude:%__provides_exclude|}perl\(DBI\)
}}}

All requires or provides must be on one line, otherwise it won't work.

After initially defining (globalling?) !__provides_exclude, it can be extended, e.g.
{{{
%global __provides_exclude %{?__provides_exclude:%__provides_exclude|}perl\(DBI\)
%global __provides_exclude %__provides_exclude|perl\(foo\)
}}}

Current perl_default_filter ignores pre-defined {{{_exclude}}} macros, so one needs to call {{{%perl_default_filter}}} first and then append values to {{{_exclude}}} macros in package spec file. Otherwise the values defined in package spec file will get lost.

Replying to [comment:2 tibbs]:

Figured I should mention that the current draft is at https://fedoraproject.org/wiki/User:Tibbs/AutoProvidesAndRequiresFiltering. Open questions that need answers are prefixed in the draft with "XXX".

  • What regexp language is in use?

    Modern POSIX.2 regular expressions - regex(7)

  • Is there any means of or need to escape characters from macros before using them in these patterns?

    Yes, the literal characters {{{"^.[$()|*+?{"}}} should be escaped with a
    double backslash. A literal "\" is represented by four(!) backslashes.

  • Is "macro" the proper term for these things?

    As far as I'm aware, yes, but I'm no expert.

  • Include example of a compound pattern. Depends on the regexp language in use.

{{{
%global __provides_exclude perl\((UNIVERSAL|DB)\)
}}}

Will match both 'perl(UNIVERSAL)' and 'perl(DB)'. Note that the
literal parentheses are escaped: "\\(" and "\\)".
  • Can the perl_default_filter bit be converted?

    perl_default_filter has already been converted for f16, but not in an
    automatically backward compatible fashion. All affected perl-* spec
    files have been updated to work with both f14-f15 style macros and
    the f16+ macros. The recommended incantation goes something like

{{{
%{?filter_setup:
%filter_from_requires /perl(some_bad_require)/d
%filter_from_provides /perl(some_bad_provide)/d
}
%?perl_default_filter
%global __requires_exclude %{?__requires_exclude:%__requires_exclude|}perl\(some_bad_require\)
%global __provides_exclude %{?__provides_exclude:%__provides_exclude|}perl\(some_bad_provide\)
}}}

  • %define or %global to set these?

    I guess %global, but I'm not an expert.

  • Is it possible to cleanly add a pattern to one of these macros, so, for example, perl_default_filter could tack on an additional pattern and not simply override whatever the user had set.

    It's not particularly clean. If you can't be sure that the macros is
    already set to something, you need to do something like

{{{
%global __requires_exclude %{?__requires_exclude:%__requires_exclude|}some_other_pattern
}}}

If you know that the macro is already set (i.e. you've already included
the previous incantation in your spec file), the macro can be further
extended by

{{{
%global __requires_exclude %{__requires_exclude}|yet_another_pattern
}}}

And yes, perl_default_filter can and should be updated in the same
way to avoid clobbering any existing definitions.

The new macros don't work with "fedpkg srpm" when using fedora-14 as development environment:

Real world example (Package rt3 - Apologies for the size):

{{{

fedpkg srpm

error: Macro % has illegal name (%define)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Macro % has illegal name (%define)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Macro % has illegal name (%define)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Macro % has illegal name (%define)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Macro % has illegal name (%define)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
error: Recursion depth(17) greater than max(16)
15< (empty)
14< (empty)
13< (empty)
12< (empty)
11< (empty)
10< (empty)
9< (empty)
8< (empty)
7< (empty)
6< (empty)
5< (empty)
4< (empty)
3< (empty)
2< (empty)
1< (empty)
error: Macro %__requires_exclude failed to expand
0< (empty)
}}}

Replying to [comment:8 corsepiu]:

The new macros don't work with "fedpkg srpm" when using fedora-14 as development environment:

Because you need rpm 4.9.

Replying to [comment:10 ppisar]:

Replying to [comment:8 corsepiu]:

The new macros don't work with "fedpkg srpm" when using fedora-14 as development environment:

Because you need rpm 4.9.
Well, though this is technically true, my view is a bit different: You cannot presume fedora packages to use rpm-4.9 based hostsystems (I currently use f14, others probably are using CentOS5). Therefore these macros can not go anywhere before they don't break building srpms on with rpm < 4.9.

I agree that incompatible change of rpm filters was unfortunate. It's a problem maintain various macros in older and new releases, but the current state of Fedora documentation isn't good. There should be at least mentioned different possibilities with different releases. The old filtering macros are not working in some cases. At least some Perl packages have section starting with comment "old filters" and another section started with comment "RPM4.9 filters".

mmaslano, I agree that this is needed, but no one on the FPC understands (or can find documentation for) the new "RPM 4.9 filters". We'd need someone who does understand them to write up a set of guidelines on how to use them properly.

Just a note that I did find http://rpm.org/wiki/PackagerDocs/DependencyGenerator which has actually existed for a while now.

Basic use of the macros is quite simple, and if the default perl filtering stuff uses the new macros then some of my concerns go away. The semester's ramping up for me so free time is going to be tight, but I'll try to revise the draft and see what questions remain.

I started working on this in reply to a different ticket: https://fedorahosted.org/fpc/ticket/189 before seeing that ticket is a duplicate of this one. I'm working on merging the things that seemed to still be pertinent: https://fedorahosted.org/fpc/ticket/189#comment:10

ticket #189 has been approved. Marking this as fixed

Login to comment on this ticket.

Metadata