Ticket #238 (closed defect: fixed)

Opened 16 months ago

Last modified 13 months ago

Permission to build sagemath with bundled pexpect

Reported by: pcpa Owned by: spot
Priority: minor Milestone:
Component: Bundled Library Exception Version:
Keywords: Cc: alexlan, tomspur
Blocked By: Blocking:

Description

Sagemath is not yet approved in fedora, but waiting for a reviewer to take the bug for some time.

It should work with system ipython soon based on http://trac.sagemath.org/sage_trac/ticket/12719 but I would prefer to have a limited time bundling permission granted, so that sagemath could be added to fedora, and this way, have more people interested on it and in having it working with system ipython.

Bundled pexpect should be used. I opened this trac 3 years ago http://trac.sagemath.org/sage_trac/ticket/6900 after significant debugging I could not make it work with a newer version. Upstream sagemath plans for very long is to switch to a more robust approach to talk via pipe to different applications, but this did not yet happen.

Cython should be the most tricky one, because both, cython and sagemath are too fast moving targets, and if system cython is updated, most times it causes sagemath to not be rebuildable for significant time, or a rebuild not reliable, so it is preferable to use the version bundled in sagemath. Once sagemath is in Fedora, I hope for it to get more attention, as several upstream sagemath developers are Fedora users, this should help in at some point no longer need the bundling exception.

The Fedora sagemath review request is at https://bugzilla.redhat.com/show_bug.cgi?id=877651

Change History

comment:1 follow-up: ↓ 3 Changed 16 months ago by toshio

Bundling of either ipython and cython make me wince. They're both large code bases that do some complex things. It'll take a lot of convincing to make think those are good ideas. The bug report about pexpect looks like both the bundled version and the new version have bugs, they're just different bugs. With the amount of information given in this ticket so far, I'd be inclined to vote against that too.

Some more in depth questions:

  • What's the timeline for having ipython unbundled?
  • Have you looked into the pexpect problem yourself? Have you tried the new pexpect package (based on pexpect-u) that's slated for Fedora-18 inclusion? The import statements will need some trivial patching but it may fix the issues you're seeing (then again, it may not...)
  • cython's updates -- this just means that we need coordination. Each Fedora release, the cython packagers are going to need to decide which version of cython they're going to ship. They'll need to get that packaged. Then we'll need to rebuild sagemath and other cython-consuming packages using it, upgrading issues that we encounter. The cython package will then need to avoid incompatible changes during that release. This is no different than when we upgrade python or gcc for a new Fedora release.

comment:2 Changed 16 months ago by alexlan

  • Cc alexlan added

comment:3 in reply to: ↑ 1 Changed 16 months ago by pcpa

Replying to toshio:

Bundling of either ipython and cython make me wince. They're both large code bases that do some complex things. It'll take a lot of convincing to make think those are good ideas. The bug report about pexpect looks like both the bundled version and the new version have bugs, they're just different bugs. With the amount of information given in this ticket so far, I'd be inclined to vote against that too.

Some more in depth questions:

  • What's the timeline for having ipython unbundled?

I believe at least 6 months. I should be able to carry patches for some time if absolutely not allowed. The reason for asking for bundling ipython and cython is to have it as an option to get something done. I want to work hard on not needing to bundle; I integrated sagemath in Fedora, using roughly 300 system packages that sage bundles...

  • Have you looked into the pexpect problem yourself? Have you tried the new pexpect package (based on pexpect-u) that's slated for Fedora-18 inclusion? The import statements will need some trivial patching but it may fix the issues you're seeing (then again, it may not...)

It does not work. I confess after spending a few days debugging it very long ago I just did settle for bundling it. Sorry for a possible long example, but anybody reading here can test it on x86_64 rawhide :-) Just install the packages at http://pcpa.fedorapeople.org/sagemath/RPMS/x86_64 After that, run:

$ cd /usr/lib64/sagemath/site-packages
$ sudo mkdir x
$ sudo mv `ls | grep -iv ipython` x
mv: cannot move ‘x’ to a subdirectory of itself, ‘x/x’
$ sage
----------------------------------------------------------------------
| Sage Version 5.4.1, Release Date: 2012-11-15                       |
| Type "notebook()" for the browser-based notebook interface.        |
| Type "help()" for help.                                            |
----------------------------------------------------------------------
sage: notebook()
The notebook files are stored in: sage_notebook.sagenb
**************************************************
*                                                *
* Open your web browser to http://localhost:8081 *
*                                                *
**************************************************
Executing twistd  --pidfile="sage_notebook.sagenb/sagenb.pid" -ny "sage_notebook.sagenb/twistedconf.tac"
2012-12-15 14:23:07-0200 [-] Log opened.
2012-12-15 14:23:07-0200 [-] twistd 12.2.0 (/usr/bin/python 2.7.3) starting up.
2012-12-15 14:23:07-0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2012-12-15 14:23:07-0200 [-] QuietSite starting on 8081
2012-12-15 14:23:07-0200 [-] Starting factory <__builtin__.QuietSite instance at 0x7fd93f3ebc68>

In firefox, click "Help" -> "Tutorial" -> "Basic Algebra and Calculus" then in the first example, when clicking the "evaluate" button, instead of it returning the result it will show something like:

Python 2.7.3 (default, Dec 11 2012, 20:49:32) 
[GCC 4.7.2 20121109 (Red Hat 4.7.2-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
[?1034h>>> 
>>> import os;os.chdir("/tmp/tmprvwTvh");
>>> execfile("_sage_input_1.py")
START1

import os;os.chdir("/tmp/tmpQ7Qshu");
execfile("_sage_input_2.py")
__SAGE__
__SAGE__import os;os.chdir("/tmp/tmpQ7Qshu");

and it gets completely out of sync. To revert it to a working setup, ^C in the sage prompt...

^C2012-12-15 14:30:31-0200 [-] Quitting all running worksheets...
2012-12-15 14:30:32-0200 [-] Saving notebook...
2012-12-15 14:30:32-0200 [-] Notebook cleanly saved.
2012-12-15 14:30:32-0200 [-] (TCP Port 8081 Closed)
2012-12-15 14:30:32-0200 [-] Stopping factory <__builtin__.QuietSite instance at 0x7f76c4f2b8c0>
2012-12-15 14:30:32-0200 [-] Main loop terminated.
2012-12-15 14:30:32-0200 [-] Server Shut Down.
sage: quit
$ ls
IPython  ipython-0.10.2-py2.7.egg-info  x
sudo mv x/* .
sudo rmdir x
  • cython's updates -- this just means that we need coordination. Each Fedora release, the cython packagers are going to need to decide which version of cython they're going to ship. They'll need to get that packaged. Then we'll need to rebuild sagemath and other cython-consuming packages using it, upgrading issues that we encounter. The cython package will then need to avoid incompatible changes during that release. This is no different than when we upgrade python or gcc for a new Fedora release.

Volker Braun pointed me to his upstream sagemath cython patch at http://trac.sagemath.org/13740 and my last srpm for review request uses it. But I prefer to still not lift the cython bundling request, well, unless Volker Braun wants to be a comaintainer and assures it will always work :-) I ask for bundling due to experience packaging sagemath for more than 3 years in Mandriva. There are windows of time that without such bundling the package would stay broken for a very long time, and could have the fate of https://bugs.launchpad.net/ubuntu/+source/sagemath/+bug/510521

comment:4 Changed 16 months ago by tomspur

  • Cc tomspur added

comment:5 Changed 16 months ago by spot

Proposal:

  • No exception for cython (coordination with system cython should be sufficient)
  • Temporary (6 month) exception for ipython
  • No exception for pexpect, unless it is impossible to get the system pexpect fixed such that it can be used by sagemath

FPC members, due to lack of quorum at the end of today's meeting, please vote on this proposal in this trac ticket.

(I vote +1 here)

comment:6 Changed 16 months ago by limb

+1 to spot's proposal.

comment:7 Changed 16 months ago by rdieter

+1 (though I'd prefer a more permissive approach wrt cython and pexpect too)

comment:8 Changed 16 months ago by toshio

+1. We can revisit pexpect if there's more information about why the patch can't be applied.

comment:9 Changed 16 months ago by spot

  • Status changed from new to assigned
  • Owner set to spot

With a vote of (+1:5, 0:0, -1:0), the proposed exceptions in comment #5 are approved.

comment:10 Changed 15 months ago by pcpa

There is no patch for pexect. I tried to contact upstream, but only one email so far, and got no response. There were several comments in the sagemath-devel thread but the information was basically that pexpect was completely rewritten in later versions and sagemath never updated to work with it. I did expend significant time long ago attempting to get it to work in Mandriva and gave up due to too many other "more important" things to correct to get the package working with system wide packages, and unfortunately this is yet the case in fedora, so, I am trying to break the barrier and hopefully get upstream working better with sagemath as a standard package, but should still take significant time, as the approach they use is the only viable approach if packaging a huge amount of software that runs in several distros and different distro versions, as well as having several patches, what frequently is not trivial to get upstream to accept...

comment:11 Changed 13 months ago by limb

Please update this ticket regarding its continued relevance, providing any information requested. If this is not done within the next two weeks, this ticket may be closed due to inactivity. Thank you!

comment:12 Changed 13 months ago by pcpa

  • Summary changed from Permission to build sagemath with bundled cython, ipython and pexpect to Permission to build sagemath with bundled pexpect

comment:13 Changed 13 months ago by pcpa

Only pexpect need to be bundled. Sagemath 5.8 works with ipython-0.13. Sagemath 5.4.1 was updated to use a patch to work with system cython and sagemath 5.5 did not need the patch.

About pexpect I am still in the same initial stage. There is a small thread about my request for information at https://groups.google.com/forum/#!topic/sage-devel/_tEyhHq7Vwg and I did not receive any reply when asking for guidance from pexpect upstream.

comment:14 Changed 13 months ago by spot

  • Resolution set to fixed
  • Status changed from assigned to closed

Exception to bundle pexpect in sagemath is granted (+1:7, 0:2, -1:0)

Note: See TracTickets for help on using tickets.