When Python packages follow https://fedoraproject.org/wiki/Packaging:Python#Naming it can result in them not working correctly. It seems that upstreams don't count with the distribution complicating things by not providing the unversioned executable for Python 3 build of the software.
Example is Sphinx. When you install the Python 3 version of the package it does not install unversioned version of 'sphinx-build'. This means that when you generate the documentation for your new project using 'sphinx-quickstart-3' the generated Makefile does not work (https://bugzilla.redhat.com/show_bug.cgi?id=1321413).
I don't think that modifying the project's Makefile due to Fedora is a good solution nor installation of Python 2 version of the package is a good solution.
I propose to revisit the Python naming guidelines so that the produced Python 3 packages in Fedora are actually usable.
Maybe allow both Python 2 and Python 3 version of packages to install unversioned script, but state that one of the Python versions is preferred. So if we state that Python 2 version is preferred, let it overwrite the unversioned script, but allow also Python 3 version to install it.
In the case of sphinx, does the functionality differ between the python2 and python3 versions? If so, how do you propose to deal with the fact that what /usr/bin/sphinx gets you depends on what happens to be installed at the time? And if not, why are two versions packaged?
As long as "python" is python2, then the principle of least surprise kind of dictates that every tool which cares about the python version would also have its unversioned executable be python2 version, and that's what our guidelines specify.
And also, if sphinx cares about the python version, then I would expect that it would take care to make sure that it does the right thing in the case that one version is installed and the other isn't.
Replying to [comment:1 tibbs]:
In the case of sphinx, does the functionality differ between the python2 and python3 versions?
Yes, Sphinx is configured with a Python script; "conf.py" is processed using the interpreter that runs sphinx-build.
Replying to [ticket:612 thozza]:
Example is Sphinx. When you install the Python 3 version of the package it does not install unversioned version of 'sphinx-build'. This means that when you generate the documentation for your new project using 'sphinx-quickstart-3' the generated Makefile does not work (https://bugzilla.redhat.com/show_bug.cgi?id=1321413). I don't think that modifying the project's Makefile due to Fedora is a good solution
I don't think that modifying the project's Makefile due to Fedora is a good solution
I believe this is an unfortunate consequence of Sphix's reliance on installed scripts being available. By calling "/usr/bin/sphinx-build" one can not specify the interpreter needed to run it. As long as some projects require Python 2 and some require Python 3, we simply can't have a good system-wide default. (Let alone synchronizing that default across Linux distros and OS X, so it can be safely put in a project's Makefile.)
I wonder what the Sphinx upstream thinks about this issue; do they have a solution? Do you know if/how other distros solve the problem?
nor installation of Python 2 version of the package is a good solution.
That's a reasonable point, though it does break the symmetry of /usr/bin/sphinx-quickstart using /usr/bin/python and /usr/bin/sphinx-quickstart-3 using /usr/bin/python3. Also, it would break all projects whose conf.py is for Python 2 only.
AFAICS this problem only affects software that launches (or creates scripts to launch) itself through a script. I'm not currently aware of anything except Sphinx that actually writes Makefiles like this. It's certainly not a case of many Python 3 packages in Fedora being unusable.
Where/how would you state this preference?
Replying to [comment:2 pviktori]:
Replying to [ticket:612 thozza]: Example is Sphinx. When you install the Python 3 version of the package it does not install unversioned version of 'sphinx-build'. This means that when you generate the documentation for your new project using 'sphinx-quickstart-3' the generated Makefile does not work (https://bugzilla.redhat.com/show_bug.cgi?id=1321413). I don't think that modifying the project's Makefile due to Fedora is a good solution I believe this is an unfortunate consequence of Sphix's reliance on installed scripts being available. By calling "/usr/bin/sphinx-build" one can not specify the interpreter needed to run it. As long as some projects require Python 2 and some require Python 3, we simply can't have a good system-wide default. (Let alone synchronizing that default across Linux distros and OS X, so it can be safely put in a project's Makefile.) I wonder what the Sphinx upstream thinks about this issue; do they have a solution? Do you know if/how other distros solve the problem?
I'm not sure, I hope the maintainer will approach the upstream with this question. I also didn't investigate sphinx's sources, but I would expect that it is not aware about the fact that we ship versioned binaries for Python 3.
nor installation of Python 2 version of the package is a good solution. That's a reasonable point, though it does break the symmetry of /usr/bin/sphinx-quickstart using /usr/bin/python and /usr/bin/sphinx-quickstart-3 using /usr/bin/python3. Also, it would break all projects whose conf.py is for Python 2 only.
Right, this is the tricky thing. I agree that having /usr/bin/sphinx-quickstart to use Python 3 may be confusing, but I believe that people install packages based on names (IOW python-sphinx or python3-sphinx) instead of relying on what provides /usr/bin/sphinx-quickstart.
I propose to revisit the Python naming guidelines so that the produced Python 3 packages in Fedora are actually usable. AFAICS this problem only affects software that launches (or creates scripts to launch) itself through a script. I'm not currently aware of anything except Sphinx that actually writes Makefiles like this. It's certainly not a case of many Python 3 packages in Fedora being unusable.
I think the problem is broader than that. Imagine someone running CI on top of Fedora for some Python project. It really complicates things that besides installing python- and python3- packages you have to also use different binary names when testing against Python 2 and Python 3.
e.g. when I use virtualenv-3.4 and activate the environment, the "python" in it actually points to the Python 3 interpreter. When I pull the official python:3.x Docker image from Docker Hub, the "python" points to the Python 3 interpreter.
So from my point of view not providing unversioned version of binaries when there is only Python 3 version of the package installed complicates things.
I understand that /usr/bin/python should always point to Python 2 interpreter, but applying the same logic to binaries using Python is at least in some cases questionable.
Maybe allow both Python 2 and Python 3 version of packages to install unversioned script, but state that one of the Python versions is preferred. So if we state that Python 2 version is preferred, let it overwrite the unversioned script, but allow also Python 3 version to install it. Where/how would you state this preference?
My wild idea would be something like this:
Do it by some predefined macros in the %post and %postun sections for python- and python3- sub-packages. The macro would check the unversioned binary symlink. In case it does not exist, it would be linked to the versioned binary that is being installed (or being left on the system in case of uninstall). In case it is linked to the Python 2 version of the binary it would be left as is.
Maybe this is not really doable, but with my current understanding of RPM I don't see why it wouldn't be. Maybe I would find out once I'd try to do it. :) Ideally it could be written in such a generic way, that in the future we could flip the preference to Python 3 version of binaries just by changing the macro.
Replying to [comment:3 thozza]:
Replying to [comment:2 pviktori]: Replying to [ticket:612 thozza]: Example is Sphinx. When you install the Python 3 version of the package it does not install unversioned version of 'sphinx-build'. This means that when you generate the documentation for your new project using 'sphinx-quickstart-3' the generated Makefile does not work (https://bugzilla.redhat.com/show_bug.cgi?id=1321413). I don't think that modifying the project's Makefile due to Fedora is a good solution I believe this is an unfortunate consequence of Sphix's reliance on installed scripts being available. By calling "/usr/bin/sphinx-build" one can not specify the interpreter needed to run it. As long as some projects require Python 2 and some require Python 3, we simply can't have a good system-wide default. (Let alone synchronizing that default across Linux distros and OS X, so it can be safely put in a project's Makefile.) I wonder what the Sphinx upstream thinks about this issue; do they have a solution? Do you know if/how other distros solve the problem? I'm not sure, I hope the maintainer will approach the upstream with this question. I also didn't investigate sphinx's sources, but I would expect that it is not aware about the fact that we ship versioned binaries for Python 3.
There's a question of what else we can do, since some projects want sphinx-build use to one Python 2, and others Python 3. The Python world is standardizing on provide packages runnable with -m, such as:
-m
{{{ python3 -m pip # instead of "pip" python3 -m venv # instead of "pyvenv" python3 -m pytest # instead of "pytest" }}}
This makes it quite straightforward to specify the interpreter, and I believe this is what sphinx-quickstart should provide and put in the Makefile (using the specific Python it was invoked with, of course). Or can it do this already?
In other words, I believe the Python ecosystem already has a solution that should work well in distros like Fedora.
nor installation of Python 2 version of the package is a good solution. That's a reasonable point, though it does break the symmetry of /usr/bin/sphinx-quickstart using /usr/bin/python and /usr/bin/sphinx-quickstart-3 using /usr/bin/python3. Also, it would break all projects whose conf.py is for Python 2 only. Right, this is the tricky thing. I agree that having /usr/bin/sphinx-quickstart to use Python 3 may be confusing, but I believe that people install packages based on names (IOW python-sphinx or python3-sphinx) instead of relying on what provides /usr/bin/sphinx-quickstart.
Really? I do things like "dnf install /usr/bin/sphinx-quickstart" woute often. (Apropos, what do you think "dnf install /usr/bin/sphinx-quickstart" should do?)
I propose to revisit the Python naming guidelines so that the produced Python 3 packages in Fedora are actually usable. AFAICS this problem only affects software that launches (or creates scripts to launch) itself through a script. I'm not currently aware of anything except Sphinx that actually writes Makefiles like this. It's certainly not a case of many Python 3 packages in Fedora being unusable. I think the problem is broader than that. Imagine someone running CI on top of Fedora for some Python project. It really complicates things that besides installing python- and python3- packages you have to also use different binary names when testing against Python 2 and Python 3. e.g. when I use virtualenv-3.4 and activate the environment, the "python" in it actually points to the Python 3 interpreter. When I pull the official python:3.x Docker image from Docker Hub, the "python" points to the Python 3 interpreter. So from my point of view not providing unversioned version of binaries when there is only Python 3 version of the package installed complicates things.
How is it complicated? When "python" points to Python 2, "sphinx-build" is run with Python 2. When "python" points to Python 3, "sphinx-build" points to Python 3. (This of course assumes that if you layer virtualenvs over the base system, you install all devel dependencies in that virtualenv. But this caveat can be eliminated by using the python -m.)
python -m
It's at least consistent: unversioned scripts are either run with Python 2, or they're version-agnostic. If that's wrong in "some cases", can you enumerate those cases?
Maybe allow both Python 2 and Python 3 version of packages to install unversioned script, but state that one of the Python versions is preferred. So if we state that Python 2 version is preferred, let it overwrite the unversioned script, but allow also Python 3 version to install it. Where/how would you state this preference? My wild idea would be something like this: Do it by some predefined macros in the %post and %postun sections for python- and python3- sub-packages. The macro would check the unversioned binary symlink. In case it does not exist, it would be linked to the versioned binary that is being installed (or being left on the system in case of uninstall). In case it is linked to the Python 2 version of the binary it would be left as is. Maybe this is not really doable, but with my current understanding of RPM I don't see why it wouldn't be. Maybe I would find out once I'd try to do it. :) Ideally it could be written in such a generic way, that in the future we could flip the preference to Python 3 version of binaries just by changing the macro.
It's doable, but fragile and confusing. Would this revert to the version if the package that currently provides the script uninstalled? Would a new version take ownership of the file if installed later? How would one tell which interpreter is currently used for which script?
If you can convince upstream to support and use python -m, we can solve the problem in a much cleaner way.
Replying to [comment:4 pviktori]:
Replying to [comment:3 thozza]: Replying to [comment:2 pviktori]: Replying to [ticket:612 thozza]: Example is Sphinx. When you install the Python 3 version of the package it does not install unversioned version of 'sphinx-build'. This means that when you generate the documentation for your new project using 'sphinx-quickstart-3' the generated Makefile does not work (https://bugzilla.redhat.com/show_bug.cgi?id=1321413). I don't think that modifying the project's Makefile due to Fedora is a good solution I believe this is an unfortunate consequence of Sphix's reliance on installed scripts being available. By calling "/usr/bin/sphinx-build" one can not specify the interpreter needed to run it. As long as some projects require Python 2 and some require Python 3, we simply can't have a good system-wide default. (Let alone synchronizing that default across Linux distros and OS X, so it can be safely put in a project's Makefile.) I wonder what the Sphinx upstream thinks about this issue; do they have a solution? Do you know if/how other distros solve the problem? I'm not sure, I hope the maintainer will approach the upstream with this question. I also didn't investigate sphinx's sources, but I would expect that it is not aware about the fact that we ship versioned binaries for Python 3. There's a question of what else we can do, since some projects want sphinx-build use to one Python 2, and others Python 3. The Python world is standardizing on provide packages runnable with -m, such as: {{{ python3 -m pip # instead of "pip" python3 -m venv # instead of "pyvenv" python3 -m pytest # instead of "pytest" }}} This makes it quite straightforward to specify the interpreter, and I believe this is what sphinx-quickstart should provide and put in the Makefile (using the specific Python it was invoked with, of course). Or can it do this already?
The solution you are describing sounds good to me. I don't know if sphinx can do it, I expected it to work out of the box.
In other words, I believe the Python ecosystem already has a solution that should work well in distros like Fedora. nor installation of Python 2 version of the package is a good solution. That's a reasonable point, though it does break the symmetry of /usr/bin/sphinx-quickstart using /usr/bin/python and /usr/bin/sphinx-quickstart-3 using /usr/bin/python3. Also, it would break all projects whose conf.py is for Python 2 only. Right, this is the tricky thing. I agree that having /usr/bin/sphinx-quickstart to use Python 3 may be confusing, but I believe that people install packages based on names (IOW python-sphinx or python3-sphinx) instead of relying on what provides /usr/bin/sphinx-quickstart. Really? I do things like "dnf install /usr/bin/sphinx-quickstart" woute often. (Apropos, what do you think "dnf install /usr/bin/sphinx-quickstart" should do?)
Naturally there are people who do that :) I would expect it to install the preferred Python version, but I'm afraid that it would be super-hard to do with YUM/DNF and RPM in general.
I propose to revisit the Python naming guidelines so that the produced Python 3 packages in Fedora are actually usable. AFAICS this problem only affects software that launches (or creates scripts to launch) itself through a script. I'm not currently aware of anything except Sphinx that actually writes Makefiles like this. It's certainly not a case of many Python 3 packages in Fedora being unusable. I think the problem is broader than that. Imagine someone running CI on top of Fedora for some Python project. It really complicates things that besides installing python- and python3- packages you have to also use different binary names when testing against Python 2 and Python 3. e.g. when I use virtualenv-3.4 and activate the environment, the "python" in it actually points to the Python 3 interpreter. When I pull the official python:3.x Docker image from Docker Hub, the "python" points to the Python 3 interpreter. So from my point of view not providing unversioned version of binaries when there is only Python 3 version of the package installed complicates things. How is it complicated? When "python" points to Python 2, "sphinx-build" is run with Python 2. When "python" points to Python 3, "sphinx-build" points to Python 3. (This of course assumes that if you layer virtualenvs over the base system, you install all devel dependencies in that virtualenv. But this caveat can be eliminated by using the python -m.)
This is exactly what is NOT complicated ;) I think you misunderstood my point. I was trying to describe that in virtualenv or in official Python Docker images, there is no such complication with versioned binary names as it is on distribution level. IOW there is always unversioned binary even if I use only Python 3.
I understand that /usr/bin/python should always point to Python 2 interpreter, but applying the same logic to binaries using Python is at least in some cases questionable. It's at least consistent: unversioned scripts are either run with Python 2, or they're version-agnostic. If that's wrong in "some cases", can you enumerate those cases?
Basically any case when I want to use an application (not a library) that is a black box for me as a user. Currently in such case I must care that the application is written in Python and have to distinguish which version I have installed and use the correct suffix with the binary name in case I have only Python 3 version installed.
Maybe allow both Python 2 and Python 3 version of packages to install unversioned script, but state that one of the Python versions is preferred. So if we state that Python 2 version is preferred, let it overwrite the unversioned script, but allow also Python 3 version to install it. Where/how would you state this preference? My wild idea would be something like this: Do it by some predefined macros in the %post and %postun sections for python- and python3- sub-packages. The macro would check the unversioned binary symlink. In case it does not exist, it would be linked to the versioned binary that is being installed (or being left on the system in case of uninstall). In case it is linked to the Python 2 version of the binary it would be left as is. Maybe this is not really doable, but with my current understanding of RPM I don't see why it wouldn't be. Maybe I would find out once I'd try to do it. :) Ideally it could be written in such a generic way, that in the future we could flip the preference to Python 3 version of binaries just by changing the macro. It's doable, but fragile and confusing.
It's doable, but fragile and confusing.
I agree that it may be fragile or hard to get right.
Would this revert to the version if the package that currently provides the script uninstalled?
Yes, as described in my previous comment.
Would a new version take ownership of the file if installed later?
Yes, if it is the sub-package built with the preferred version of Python.
How would one tell which interpreter is currently used for which script?
Should I really care as a user? If there are Python 2 and Python 3 versions of the binary, I would expect them to provide equivalent functionality. I know you can argue that there are cases like sphinx parsing conf.py and expecting it to conform to the Python syntax based on the interpreter that was used for invoking the application.
I as a user use an application that may be written in Python, but I use is as an black-box, not as a script. If I write a script, let me care about the version of Python, but if it is a software packaged in a distribution, I assume the maintainer made a conscious decision to build the package for specific version of Python or both. I expect that both versions will have equivalent functionality. e.g. when you use "fedpkg" do you really care which version of Python it uses to run? But honestly as a user, not Python enthusiast :)
I'll advise this to the maintainer, thanks.
Replying to [comment:5 thozza]:
Replying to [comment:4 pviktori]: Replying to [comment:3 thozza]: How would one tell which interpreter is currently used for which script? Should I really care as a user? If there are Python 2 and Python 3 versions of the binary, I would expect them to provide equivalent functionality. I know you can argue that there are cases like sphinx parsing conf.py and expecting it to conform to the Python syntax based on the interpreter that was used for invoking the application.
Replying to [comment:3 thozza]: How would one tell which interpreter is currently used for which script?
This argument conflates two cases into a single case which is leading to a bad idea. For any program where the user does not need to care about the interpreter used the guidelines currently say:
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only the Python 3 version of the executable should be packaged.
and in the Naming section:
If only one executable is to be shipped, then it owns its own slot and should use /usr/bin/python3 from Fedora 22 on.
The portion of the guidelines that you are trying to have changed now is only for programs where the choice of interpreter is not a part of the blackbox to the user. It's where the user cares which python version the script runs on because the program has different capabilities related to the python version when run under each. This typically means that the program is parsing python source files using python's facilities (for instance, ast.parse) for some purpose -- running tests, linting code, extracting documentation, etc.
I think that for sphinx the real problem is not going to be conf.py but autodoc. I think that autodoc relies on parsing the source code of a project to extract docstrings which are to be used in documentation. This parsing will fail if the python2 version is run on python3 code and vice-versa.
Changing what sphinx-build points to based on what packages are installed is a horrible solution as it means that users will have different behaviour depending on what's installed. So different users will have different behaviour and different "bugs".
The above mess is even more confusing than the problem you're complaining about now: having to change the binary that is referenced in the Makefiles. At least the current problem has the virtue of being the same for everyone and easy for people to understand (the program you want has a different name because the python2 version conflicts with it).
here's some alternatives for addressing this: Fix the upstream scripts. I don't think this is as easy as saying "use python -m" because the problem is detection, not invocation. I have to detect that the source code I'm going to operate on is python2 or python3 in order to know whether to run python2 -m sphinx or python3 -m sphinx. However, fixing things upstream will make things better for everyone (other distros will have this same problem). If the real complaint here is that Fedora is default python3 but the python3 packages really need the python2 packages in order to operate (or the python3 programs are hard to find in comparison to the python2 programs in the case of sphinx-quickstart) then we should consider making the unversioned program the python3 version. Instead of making "what is /usr/bin/python" the gating factor make "what is the default python on this version of the distro" the gate. To me this makes sense as we're pushing the scripts which do not need two versions to install from the python3 package as part of python3 being default. So it makes sense that the unversioned script name in sphinx would also come from the python3 package.
Replying to [comment:6 toshio]:
Fix the upstream scripts. I don't think this is as easy as saying "use python -m" because the problem is detection, not invocation. I have to detect that the source code I'm going to operate on is python2 or python3 in order to know whether to run python2 -m sphinx or python3 -m sphinx. However, fixing things upstream will make things better for everyone (other distros will have this same problem).
This should not be hard: Sphinx should choose the "python2" or "python3" based on the interpreter that sphinx-quickstart runs under. (This won't work under Windows which don't have python2/python3 commands, but the problem is in creating scripts/Makefiles, and those are unix-specific anyway.)
If the real complaint here is that Fedora is default python3 but the python3 packages really need the python2 packages in order to operate (or the python3 programs are hard to find in comparison to the python2 programs in the case of sphinx-quickstart) then we should consider making the unversioned program the python3 version.
I don't think that's the complaint.
If it's just a problem with sphinx-quickstart then it is indeed pretty easy to create and propose a fix for upstream. However, fixing quickstart only pushes the problem into the makefile. Since the makefile may be run on another distro or for python2 instead of python 3 (if the source is dual python 2 and python 3) it needs to also have a way to decide whether to run sphinx-build under python 2 or python 3.
Replying to [comment:9 toshio]:
As per [https://www.python.org/dev/peps/pep-0394/ PEP 394], if Sphinx uses something like: {{{ python2 -m sphinx.build python3 -m sphinx.build python -m sphinx.build # (for documentation that is buildable with both) }}}
it should be portable across all Unix-like systems.
Note that I wouldn't recommend just "python" unless the project's tests actually involve building the documentation with both Python versions. It's much easier to stick to one for new projects. (And sphinx-quickstart only applies to new projects.)
Also, individual users of the Makefile can already set $SPHINXBUILD to use a custom command, in case the project-wide defaults are not appropriate.
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2016-03-31/fpc.2016-03-31-16.00.txt):
...just discussed it, doesn't seem likely we'd do any big changes.
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2016-05-05/fpc.2016-05-05-16.00.txt):
Metadata Update from @pviktori: - Issue assigned to james
Login to comment on this ticket.