Hi, there's been a discussion on python-devel@fp.o ML [1] about how to properly/easily package applications using the system Python in both Fedora and EPEL.
To sum up current state: - By "applications", I mean packages that don't have python- and python3- subpackages, but rather packages like Pidgin, that use system Python to provide some end-user capability. Packages that ship python- and python3- subpackages shouldn't be hit by this decision. - Since F22 we recommend building apps against Python 3, which has different set of macros than Python 2. - People want to create specs that look the same for Fedora and EPEL.
I've already been contacted by several people seeking guidance on proper packaging of applications that are built for both Fedora and EPEL. Their problem is that they have to define a new set of macros to point to Python 3 in Fedora >= 22 and to Python 2 in EPEL (and Fedora < 22). Or they sometimes redefine the unversioned {{{%__python/%python_sitelib/...}}} macros to point to Python 3 in Fedora (these point to Python 2 by default). I think we should acknowledge that this is a valid use case and create a guideline for it.
The linked discussion on python-devel has yielded 3 possible solutions and I'd like to ask for your opinion on which one seems to be the best: 1. We can define a set of new macros (the original proposal used "{{{%__default_python}}}", but "{{{%__syspython}}}" is shorter and probably better) to be used. Then we either provide a "{{{%make_syspython}}}" macro function, that will take either "python2" or "python3" as argument and define the {{{%__syspython/...}}} macros to correct values. OR we can negotiate adding these macros to minimal buildroots for both Fedora and EPEL, so that there's no need for {{{%make_syspython}}} and everything works out of the box (including macro'd BuildRequires, which need to be resolved in minimal buildroot). 2. Provide {{{%make_syspython}}} function to be used with unversioned {{{%__python/...}}} macros. Packagers would call {{{%make_syspython}}} on top of their specfile and then they'd keep using {{{%__python/...}}} on both Fedora and EPEL. 3. Just point Fedora's unversioned {{{%__python}}} macros to Python 3. This would initially break a lot of extension packages (the guidelines have for some time suggested using the versioned macros for all builds, but many old packages still use unversioned ones), but would work without any changes to application specfiles. I'm however unsure that this aligns with keeping /usr/bin/python point to python 2 and other decisions that we've made for the Python 3 switch.
I'll appreciate all comments on this from FPC.
Thanks.
[1] https://lists.fedoraproject.org/pipermail/python-devel/2015-January/000671.html
Why keeping /usr/bin/python pointing to python 2 then? I think whenever python 3 is the default, I expect /usr/bin/python to point to python3. Otherwise python2 is the "default" implementation with the only exception that applications are encouraged to use python3.
I see {{{%__python}}} as a nice macro for whatever is currently the default and don't see a reason to introduce yet another macro, such as {{{%__default_python}}} or so.
Sure, this will most likely break old packages, but is the cleanest solution possible. The other solutions make it only overly complicated.
Replying to [comment:1 tomspur]:
The main reason is that upstream has a PEP that deals with downstream packaging and recommends that for the time being, /usr/bin/python should point to Python 2 [1]. I think Fedora should follow the upstream recommendation (although I admit I'm not very happy about the recommendation itself).
I see {{{%__python}}} as a nice macro for whatever is currently the default and don't see a reason to introduce yet another macro, such as {{{%__default_python}}} or so. Sure, this will most likely break old packages, but is the cleanest solution possible. The other solutions make it only overly complicated.
I tend to agree, but I'm still unsure whether making /usr/bin/python and {{{%__python}}} diverge is a good thing to do. (Oh, that's actually why I opened this ticket in the first place, so thanks for your opinion :))
[1] https://www.python.org/dev/peps/pep-0394/#id14
Replying to [comment:2 bkabrda]:
Replying to [comment:1 tomspur]: Why keeping /usr/bin/python pointing to python 2 then? I think whenever python 3 is the default, I expect /usr/bin/python to point to python3. Otherwise python2 is the "default" implementation with the only exception that applications are encouraged to use python3. The main reason is that upstream has a PEP that deals with downstream packaging and recommends that for the time being, /usr/bin/python should point to Python 2 [1]. I think Fedora should follow the upstream recommendation (although I admit I'm not very happy about the recommendation itself).
That's why I recommend to not follow that PEP and see one of our [https://fedoraproject.org/wiki/Foundations foundations] in action: "First" :) (We won't first anyway as Arch already uses python3 as /usr/bin/python according that PEP.)
That PEP recommends to leave /usr/bin/python as python2 to not break "third party scripts", and wait until the majority of users are aware of python3 and transformed their scriptlets.
I see {{{%__python}}} as a nice macro for whatever is currently the default and don't see a reason to introduce yet another macro, such as {{{%__default_python}}} or so. Sure, this will most likely break old packages, but is the cleanest solution possible. The other solutions make it only overly complicated. I tend to agree, but I'm still unsure whether making /usr/bin/python and {{{%__python}}} diverge is a good thing to do. (Oh, that's actually why I opened this ticket in the first place, so thanks for your opinion :)) [1] https://www.python.org/dev/peps/pep-0394/#id14
Proposal for the meeting: Don't let /usr/bin/python and {{{%__python}}} diverge :)
+1 to tomspur's proposal. Where in the guidelines would I insert that if it passes?
How about replacing the {{{%{__python} deprecated}}} box at the end of the [http://fedoraproject.org/wiki/Packaging:Python#Macros macros section] with something like:
{{{ '''The generic %{__python} macro''' The unversioned macros, %{__python}, %{python_sitelib}, and %{python_sitearch} are generic macros that will always point to the default implementation. For now this "system runtime" is the python2 interpreter. }}}
The {{{now}}} can then be replaced with {{{Fedora 22 and older}}}, if we really switch to python3 later end of this year...
Fedora wouldn't be the first distro to change the python symlink to point to Python 3: that honour goes to Arch, and the fact they did that is the reason we wrote the upstream PEP in the first place.
The reason changing the symlink is currently still a bad idea has nothing to do with distro packages, and everything to do with custom scripts written by system administrators. Those are almost always lacking automated tests, and they're also usually rife with print statements. If Fedora changes the symlink target, we will break the world for at least some end user systems. There'll be a point in time when that's a defensible thing to do, but I don't think we've reached that point yet (I'm personally pegging it at the point where CentOS has Python 3 in the base package set).
Changing {%__python} had the potential to be a different story, as that only affects packaged software. It limits the impact of a definition change to RPMs that are built without nominating a specific Python version, rather than all Python scripts (whether packaged or not) that are executed on the system via the unversioned symlink. Unfortunately, I just realised that the core problem with the idea of letting the two definitions of the "default Python" diverge is that you wouldn't be able to use unversioned shebang lines to go with the unversioned RPM macros. Introducing a new {%__syspython} macro has the same problem: what do you put in the shebang lines of the packaged scripts?
{%__python}
{%__syspython}
So on reflection, I think tomspur is right: actually changing the default Python really does require changing the symlink, otherwise you can't do anything sensible with the shebang line in packaged scripts.
If we go with that perspective, then my near term recommendation would be to stick with explicitly opting in to Python 3 in Fedora for the time being, get Python 3 stacks into at least EPEL6 and EPEL7 (so Python 3 only packages will also "just work" in EPEL), and then revisit the question for Fedora 23.
Replying to [comment:6 ncoghlan]:
what do you put in the shebang lines of the packaged scripts?
You can always do this from spec:
{{{ find -name '*.py' | xargs sed -i '1s|^#!/usr/bin/python|#!%{__python}|' # (or syspython at the end) }}}
This would get replaced by either /usr/bin/python2 or /usr/bin/python3 and would work, regardless of where /usr/bin/python points to.
Also in library packages, you would do:
{{{ find %{py3dir} -name '.py' | xargs sed -i '1s|^#!/usr/bin/python|#!%{__python3}|' find . -name '.py' | xargs sed -i '1s|^#!/usr/bin/python|#!%{__python2}|' }}}
(Assuming Python 2 part of the package is created from . and Python 3 from %{py3dir}, as it is now.)
This also solves the problem, that maybe some day in the future /usr/bin/python will point to Python 3, because all of our packages would have specific version of shebang. We can even make it a MUST, so nothing in system uses usr/bin/python and we leave that only for users/sysadmins/dinosaurs.
This was discussed in this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-02-05/fpc.2015-02-05-17.02.txt):
...note that there was a lot of confusion here, esp. wrt. FPC ticket #327 and what default means.
So my personal feeling here is that we have at least three things it would be good to support:
User wants to write a script which runs on CentOS, Mac OSX and Fedora. Given Mac OSX doesn't have /usr/bin/python2 then we should keep /usr/bin/python as py2 for the next few years. This is also what upstream recommends.
Packager wants to just us the old _python/python_sitelib/etc. macros and just get the "default" implementation for the distro. At worst the code has to change anyway, but at least you can see specfile changes between F20 and F25 without horrific line noise.
Packager would like a simple way to say "what is the default python", if we change the _python macro this could be used as the test ... or could introduce a new macro _python_default_version or something.
If it's different, although that seems crazy to me, we could also have a "what is the python version I should build for" ... AIUI 3.5 is kind of py3 now, even though that'll bring in two pythons for everyone.
Toshio and me are currently discussing the options [https://etherpad.mozilla.org/2Uqk0ikCll online].
I think pretty much everything should be covered there except:
Replying to [comment:9 james]:
As @churchyard notes, we can just run the find/sed command to make sure all shebangs are pointing to /usr/bin/python3 - we'd just probably need a smarter pattern, that would replace stuff like "/usr/bin/python2" and not replace lines that already point to python3 (which would make them /usr/bin/python33) - but that's a technical detail. We could provide a macro in python3-devel, e.g. {{{%{py3_ensure_shebangs}}}} or similar.
Thanks @tomspur and @toshio for putting together the etherpad, it's very nice work and sums up the current state and the options very nice.
I'm starting to like "Unversioned macros are changed and that's it.", even though I know it's a bit controversial. There's one more fact to consider here, too: 3rd part repos providing RPMs for Fedora. We don't know how many of these are out there and we could cause some headaches to their maintainers. I'm not saying that this should stop us, we would just need to push the message out very loudly. Also, I think this would make sense to change for Fedora 23, not for F22, mostly due to the amount of work. Since "Python 3 as Default" was postponed, we could make this macro change a part of it.
It's true that this approach would make /usr/bin/python and {{{%__python}}} out of sync, but I don't think that's necessarily a problem, if explained properly. In short, we could say something like this: "Libraries" specfiles must use versioned macros - {{{%__python2}}}/{{{%__python3}}}, ... "Applications" that want to run with the system default Python should use unversioned macros - {{{%__python}}}, ... to run on the default system Python and may use versioned macros assuming that upstream code doesn't work with default system Python. * Due to upstream recommendation [1], {{{%__python}}} macro is not guaranteed to contain {{{/usr/bin/python}}} value. Notably, when Python 3 is the default Python this macro will point to {{{/usr/bin/python3}}}.
[1] https://www.python.org/dev/peps/pep-0394/#recommendation
Here is the diff for bringing back the unversioned python macros, so python apps can rely on them: https://fedoraproject.org/w/index.php?title=User%3ATomspur%2FUnversioned_Python_Macros&diff=403799&oldid=403797
Drawback: When packaging for Fedora and EPEL, those can be overwritten in each (Fedora) spec file to point to python3 or just wait, when the global interpreter is switched to python3. This means it is easier to just stay on python2 for F21 (at least) and wait for the global switch.
Replying to [comment:12 tomspur]:
Looks good. Some comments:
I'd even say we should only do this in F23, since this will require changes in massive ammount of old specfiles. F22 can stay as it is, IMO.
Instead of saying that these macros are"not for use with libraries" it would be better to say "these macros are only for use with applications that need to choose to use the system's default version of python." This better captures ncoghlan's categorization of packages that depend on python. There are a few applications that need to be built for both python 2 and python 3 and also applications that are not written to run on either version of python and so must explicitly use python 2 or python 3 which cannot use these macros.
Would also be good to explicitly define "systems default python" somewhere early on. And probably put the note about this being primarily about moving from python 2 to python 3 in there (otherwise inexperienced packagers might think this is just about python minor versions ie python 2.6 on rhel6 vs python 2.7 on fedora).
Last note if you're thinking of changing these macros globally instead of in the spec files you need to talk to the rpm maintainers to be sure this is acceptable (either in rpm or in redhat-rpm-config) as these macros are defined by the upstream python package. (And they may point out other places that need to be changed like the python byte compile script that tomspur noticed last week.)
@toshio thanks and +1, these are all good points. I think you meant "upstream rpm package", not "upstream python package", correct?
@panu you are correct. Typo on my part.
Thought of a few more things...
If the macros are changed globally rather than in the spec files then all python packages need to be modified to not use the unversioned python macros unless they are intended only to run on the system's default python. So that strategy probably needs a Fedora Change in place that says who is going to do that work.
I see some discussion of when these macros are going to change. They need to change at the same time as we're telling people to switch their packages to use python 3 instead of python 2 (currently f22). Otherwise we haven't solved the problem stated at the beginning of the ticket (single spec file for all fedora and epel versions). Given that changing the macros globally requires a Fedora Change, I think we either need to move that date forward to f23 or not change these macros globally.
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-02-19/fpc.2015-02-19-17.00.txt):
I'm not sure whether this suggestion belongs on this ticket or somewhere else, but with the change postponed to Fedora 23, would it be worth enabling Py3k warnings by default when the Fedora 22 system Python is accessed as "python" rather than the qualified "python2"?
The reason I ask is because I'm considering proposing we change the upstream recommendation for where /usr/bin/python points in conjunction with the Python 3.5 release, and that would be more feasible if there was an associated recommendation around implicitly enabling Py3k warnings in a transition period.
Written up. Announcement text:
The Python guidelines were modified to clarify the use of unversioned macros (%__python instead of %__python2 or %__python3, for example). * https://fedoraproject.org/wiki/Packaging:Python * https://fedoraproject.org/w/index.php?title=Packaging%3APython&diff=406053&oldid=405394 * https://fedorahosted.org/fpc/ticket/498
Also, Nick, I don't really think the question you raise in comment 19 has, well, really anything to do with the FPC. I guess that's something the python package maintainers would do, but then again I'd wager it's way to late to do anything like that in the F22 timeframe.
Metadata Update from @churchyard: - Issue assigned to tibbs
Login to comment on this ticket.