Related: https://fedorahosted.org/fpc/ticket/492. Related: https://bugzilla.redhat.com/show_bug.cgi?id=1221363
Would like FPC to establish guidelines for use of weak dependencies in F23. The motivation is that it seems like they would be quite handy for some ongoing development, as described in bz above. Currently, use of weak dependencies appears to be forbidden due to lack of any established guidelines.
There already exists some remarks on rpm.org (http://www.rpm.org/wiki/PackagerDocs/DependenciesOverview) from which I was apparently able to extract a reasonable understanding of how these weak dependencies ought to work.
In the bz above, I outlined a pretty clear, consistent, reasonably non-trivial, and apparently correct use of direct weak dependencies as well as some potential uses for reverse weak dependencies.
AFAIK, hawkey is just a simplified API for libsolv, which is a special purpose SAT solver. dnf does a lot of management tasks, but in terms of dependency resolution, it just passes on what hawkey/libsolv tells it.
Therefore, the issue is what libsolv does and how it resolves various kinds of dependencies. About this there may be some non-definedness, which may or may not be a necessary consequence of satisfiability solutions. But that need not hinder the establishment of guidelines.
I've come up with some slightly complicated questions, but I'm not even sure that they need to be resolved in order for the guidelines to be written. That's why I've put them at the bottom.
I believe that a bunch of use cases were discussed at DevConf this February. These could be gathered up to provided some guidance in drawing up guidelines.
============================================================================================
Slightly convoluted questions:
A requires B_1 or B_2
C suggest B_1
D requires B_2
E suggest B_2
The order of events: 1. C is installed (B_1 is not installed) 2. A is installed. Since B_1 and B_2 are equivalent, B_1 is installed because C suggests it? 3. D is installed. D requires B_2 so it must be installed. Does B_1 remain because it was installed already? Is dnf able to do a minimization or pruning activity?
Another possible order is: 1. C and E are installed first. 2. Now A is installed. A must have either B_1 or B_2, who among C or E gets its suggested dependency installed?
OR
C suggests B > 1.0
D suggest B > 2.0
A requires B
A is installed. Probably the higher version of B is preferred. But if it can not be satisfied, then lower version is still good. Or is tie resolved some other way?
Replying to [ticket:531 mulhern]:
Related: https://fedorahosted.org/fpc/ticket/492. Related: https://bugzilla.redhat.com/show_bug.cgi?id=1221363 Would like FPC to establish guidelines for use of weak dependencies in F23. The motivation is that it seems like they would be quite handy for some ongoing development, as described in bz above. Currently, use of weak dependencies appears to be forbidden due to lack of any established guidelines.
May I ask how did you come to that conclusion? I hope I didn't give you that impression :-) I certainly don't recall any decision on this front and there is nothing technically stopping you from using weak deps. They are in fact already used in a few packages in F22.
Actually, soft deps are really as simple as described in the document, at least their main intended use case.
Using soft deps to declare preference between packages is a little different though and as you can already see there are so many corner cases that I would discourage you from using them for this purpose unless absolutely necessary. It's like I always say: if your package declare it Requires B or C, both are interchangeable by definition and you should not care which one is going to be installed.
Reading all this, I'd like to ask the obvious question: what would you like to see in the guidelines? Or better yet, what should be the purpose of the guidelines in your opinion? We (as in the software management team) were planning to use such a document only to show maintainers what can they do with soft deps. Reading this post, maybe an FAQ is not a bad idea either :-) Some more extensive documentation might be required for soft deps being used to set preference between packages if we want to encourage people to use them like that.
============================================================================================ Slightly convoluted questions: A requires B_1 or B_2 C suggest B_1 D requires B_2 E suggest B_2 The order of events: 1. C is installed (B_1 is not installed) 2. A is installed. Since B_1 and B_2 are equivalent, B_1 is installed because C suggests it?
Slightly convoluted questions: A requires B_1 or B_2 C suggest B_1 D requires B_2 E suggest B_2
The order of events: 1. C is installed (B_1 is not installed) 2. A is installed. Since B_1 and B_2 are equivalent, B_1 is installed because C suggests it?
If C is not included in the transaction (which it doesn't appear to be), it doesn't have any effect on it I think. That means B_1 and B_2 are equivalent in that transaction any any of the two can be chosen.
D is installed. D requires B_2 so it must be installed. Does B_1 remain because it was installed already?
Yes, if it was already installed, it will remain installed.
Is dnf able to do a minimization or pruning activity?
There is no intention to do anything like that, as it would be nothing more than a wild guess of what the user wants to have on his system.
Again, neither of the two packages seem to be a part of the transaction so they don't influence it.
OR C suggests B > 1.0 D suggest B > 2.0 A requires B A is installed. Probably the higher version of B is preferred. But if it can not be satisfied, then lower version is still good. Or is tie resolved some other way?
C suggests B > 1.0 D suggest B > 2.0 A requires B
I don't see any tie here, B > 2.0 satisfies all three. But if there was a conflict in suggestions, I would expect them to cancel each other out and only the standard behavior of Requires to remain.
50% of the purpose of the guidelines in my mind is just to exist in the Fedora Packaging Guidelines in some rational form so that would-be users of the weak dependencies like myself do not get told that it is not currently allowed.
The other 50% is just to be useful. It should surely include the very general synopsis provided by http://www.rpm.org/wiki/PackagerDocs/DependenciesOverview.
It should probably include a remark indicating that the dependency solutions are calculated per transaction, so that the suggestions of previously installed packages are not taken into account for the solution to the current transaction. This will obviously cause some wierdness, as it seems that installing A, then B, will end up in the general case, with a different set of dependencies then installing A with B. But that fact is not at all clear in any of the documentation or remarks that I have seen and is certainly not clear to me as is evident from my questions.
Once I've assimilated that fact, though, it seems like a bunch of the dependencies are less useful than I had imagined. For example, if A supplements B and B is installed, A will not be installed unless it is also part of the transaction. It seems more intuitive, though scarier, that A will be automatically installed if B is installed and it is available.
It seems to me, therefore, that Supplements is not so useful for satisfying dependencies during an installation. It actually requires a 2-step process. Use dnf's repoquery to find out what packages supplement the package that you want to install. Select among those packages any that you might want to install and specifically request their installation.
It seems to me that there is not much point in the distinction between Supplements and Enhances except to completely fill the weak dependencies table.
Well, we wanted to be in sync with the other distros where this model works. As a user, I can see and appreciate the difference but I admit it needs support of the tooling for other people to fully appreciate it as well.
As a maintainer, you can see weak dependencies as something that you want to pull in by default but not having it around (user uninstalls it for instance) will not cause your application any serious problems.
On the other hand, you can perceive the very weak dependencies as a "see also", or "FYI" if you will. It will not be installed by default but it is nice to have an information that some extension exists that you can use.
Does that make sense?
As for the FAQ, I'd be glad if we could avoid this. I'd prefer to document solutions to some real use cases of weak dependencies rather then documenting how a solver may/should behave in an artificial situation. If there is a concern that a solver may choose a wrong solution to a request in a particular system state, let's focus on why is it a concern and document how the package dependencies should be used in such use case. I mean, the document should explain that "if you want to achieve this, you should use these dependencies" rather than "if a given combination of dependencies is passed to a solver, it should pick these packages". One reason for this is e.g. that because YUM documented how it picks providers, there are people that abuse these rules to achieve package preferences (e.g. by choosing a tweaked package name, or by using higher versions of virtual provides, etc.).
Replying to [comment:4 rholy]:
Ack, I could not have said it better :-)
As for the FAQ, I'd be glad if we could avoid this. I'd prefer to document solutions to some real use cases of weak dependencies rather then documenting how a solver may/should behave in an artificial situation.
FPC had discussed weak deps etc. a couple of weeks ago, IIRC, without conclusion. Our (at least my) primary concern had been weak deps are causing a hardly handable explosion of dimensions in complexity in dependency handling, leading to overall undefined and unclear behavior (and discussions).
That said, I'd actually prefer a pragmatic approach and would prefer to see dnf be equipped with a testsuite/demosuite, which demonstrates the dnf's devs vision of dnf's behavior in such cases.
Replying to [comment:3 jzeleny]:
Well, we wanted to be in sync with the other distros where this model works. As a user, I can see and appreciate the difference but I admit it needs support of the tooling for other people to fully appreciate it as well. As a maintainer, you can see weak dependencies as something that you want to pull in by default but not having it around (user uninstalls it for instance) will not cause your application any serious problems. On the other hand, you can perceive the very weak dependencies as a "see also", or "FYI" if you will. It will not be installed by default but it is nice to have an information that some extension exists that you can use. Does that make sense?
Thanks. But my question was about reverse weak dependencies, which seem hardly distinguishable from reverse very weak dependencies in the way that they would actually be used. You have to go out and look for packages that have either type of relationship to the package of interest and then, if you find them, you do AFAICT, the same thing, which is decide whether you are interested in installing them or not. You could say that Supplements vs. Enhances conveys some sort of information to me, when I'm deciding whether to install or not, but I would take that with a grain of salt.
Regarding FAQs and test suites:
It is always a mistake to rely on accidental behaviors of the library you're using. However, people will do it, and the absence of a FAQ will not prevent them. If the library is completely undocumented, users will have no way to distinguish between the accidental and the intended behaviors, making them blameless when they rely on the accidental behaviors. It's not necessary to explain how an algorithm works, except to those who have to modify it, but it is really useful to the users of a library to document what it guarantees and then make sure that it keeps fulfilling its guarantees.
Tests are good, almost always.
FAQs are often helpful, but not always sufficient.
I've put this on the meeting, so we can discuss it. But unless non-FPC people turn up, don't expect there to be any results from the discussion.
Replying to [comment:1 jzeleny]:
Replying to [ticket:531 mulhern]: Related: https://fedorahosted.org/fpc/ticket/492. Related: https://bugzilla.redhat.com/show_bug.cgi?id=1221363 Would like FPC to establish guidelines for use of weak dependencies in F23. The motivation is that it seems like they would be quite handy for some ongoing development, as described in bz above. Currently, use of weak dependencies appears to be forbidden due to lack of any established guidelines. May I ask how did you come to that conclusion? I hope I didn't give you that impression :-)
May I ask how did you come to that conclusion? I hope I didn't give you that impression :-)
This is from the FPC. There is no policy to use them, thus. the default is that they shouldn't be used.
We don't have any kind of enforcement though, so "forbidden" is the wrong word for anything related to policy.
The big unanswered question is why should packagers be using them. Documenting how they affect a transaction in specific situations is much easier, but if that's all we do then it'll be pretty random what packagers do with that information.
One obvious way forward is to keep everything else the same and have a set of problems/solutions using weak-deps. So if you have problem XYZ, then put these weak-deps. into your package to solve the problem. That's mostly a discussion between people with specific problems they think can be solved by weak-deps. and the dnf-devs who are willing to give a solution they'll support forever to solve that problem and not do random other damage.
The harder way forward means someone defining exactly why/how a package is requires/soft-requires/info-requires (or obsoletes+requires, reverse-soft-requires, reverse-info-requires), this will almost certainly result in a bunch of existing requires moving to soft-requires. I guess Fedora.next and dnf-devs would mostly lead this?
Of course the really easy way is to let chaos happen for a bit, and see what the results are. This might not be as bad as it seems, and is kind of what we've been doing by default, although it's much more likely to result in conflicting usecases (soft-requires should be on so FOO happens to package A, soft-requires should be off so BAR happens on package B).
I think the big problem here is that when soft-deps first came up they were talked about with rich-deps ... which might actually be able to solve the prefered-provider problem well. It's also complicated by the amount of grey area, some of which is very close to "XYZ should really be a requires, but can't be" and soft-requires might help that.
with my old yum developer hat on
Replying to [comment:8 mulhern]:
Regarding FAQs and test suites: It is always a mistake to rely on accidental behaviors of the library you're using. However, people will do it, and the absence of a FAQ will not prevent them. If the library is completely undocumented, users will have no way to distinguish between the accidental and the intended behaviors, making them blameless when they rely on the accidental behaviors. It's not necessary to explain how an algorithm works, except to those who have to modify it, but it is really useful to the users of a library to document what it guarantees and then make sure that it keeps fulfilling its guarantees.
The quoted text accurately explains why the compare-providers page was created and maintained (and we did modify it a few times). Before it was created there still existed rules and packagers still had to solve problems so that systems were reliably installed, and if those rules changed then it still broke installations. And those rules were passed along to new packagers via. word of mouth, if that. Just saying "It's not my fault you can't install CentOS-9 anymore" or "I'm not telling you how to make your install of CentOS-12 work, that's undocumented internals" is not going to make many people think good things about dnf.
I understand the desire to say "[I'd rather not be] documenting how a solver may/should behave in an artificial situation", but artificial situations are the only ones that can be reasoned about.
Replying to [comment:9 james]:
I've put this on the meeting, so we can discuss it. But unless non-FPC people turn up, don't expect there to be any results from the discussion. Replying to [comment:1 jzeleny]: Replying to [ticket:531 mulhern]: Related: https://fedorahosted.org/fpc/ticket/492. Related: https://bugzilla.redhat.com/show_bug.cgi?id=1221363 Would like FPC to establish guidelines for use of weak dependencies in F23. The motivation is that it seems like they would be quite handy for some ongoing development, as described in bz above. Currently, use of weak dependencies appears to be forbidden due to lack of any established guidelines. May I ask how did you come to that conclusion? I hope I didn't give you that impression :-) This is from the FPC. There is no policy to use them, thus. the default is that they shouldn't be used.
Let me quote mattdm: "I think in general we can't possibly consider lack of guidelines on something as a ban, because the world is big and moves quickly." In other words, this is hardly justifiable default.
@mulhern: reverse dependencies have exactly the same semantics as the forward ones, the only difference is where you specify it. But if you perceive packages and dependencies between them as a graph, the graph will be exactly the same, no matter if you use the forward or the backward one.
I like your proposal to state what the dependency model guarantees and then to make sure it really guarantees that. Such a definition can be project / solver independent. It will also increase our chance to avoid a similar discussion couple years from now.
@james: when is the next FPC meeting? I can think of two or three people from the team (including me) who could join the discussion
2 hours from now: https://lists.fedoraproject.org/pipermail/packaging/2015-May/010658.html
Replying to [comment:13 jzeleny]:
Ok. I'm thinking of it in those terms then. A supplements B is exactly equivalent to B recommends A. So, regardless of whether Supplements: B occurs in A's spec or Recommends: A occurs in B's spec.
if A is installed that does not cause B to be installed BUT
if B is installed that causes A to be installed (unless A can't be installed, in which case installation of B proceeds regardless).
that is something worth saying in any spec, since it halves the complexity of the problem ;)
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-05-21/fpc.2015-05-21-16.08.txt):
Ok, updated https://fedoraproject.org/wiki/PackagingDrafts/WeakDependencies. Comments welcome!
In meeting, somebody mentioned that package preferences should really be handled by rich dependencies, e.g., OR. So, maybe this sentence "They also allow packages to specify preference for specific providers while maintaining the flexibility of virtual provides - e.g. preferring ruby vs jruby, community-mysql vs mariadb" should be dropped.
Well, we do not have rich deps, yet. The policy is for the current state of rpm and dnf. We will update the policy as soon as rich deps get available. One step at a time.
Could be worth making clear that weak dependencies are the duct tape solution for this particular problem and that rich dependencies (when introduced) will be the preferred solution. Right now, it looks like preferred package problem is one of the major reasons for introducing weak dependencies.
Replying to [comment:20 mulhern]:
+1
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-06-04/fpc.2015-06-04-16.02.txt):
Written up (finally, sorry).
Announcement text:
Guidelines for making use of weak dependencies (Recommends:, Suggests:, etc.) have been added. https://fedoraproject.org/wiki/Packaging:WeakDependencies https://fedorahosted.org/fpc/ticket/531
Metadata Update from @mulhern: - Issue assigned to tibbs
Login to comment on this ticket.