We don't have reproducer. It happens rarely.
(We found this during DNSSEC work, but we don't use PySSL in DNSSEC)
Oct 19 01:37:45 example.com ipactl[14145]: Starting ipa-dnskeysyncd Service Oct 19 01:37:51 example.com systemd-coredump[14049]: Process 13333 (ipa-upgradeconf) of user 0 dumped core. Stack trace of thread 13333: #0 0x00007f0acf3e5de4 PySSL_sslwrap (_ssl.so) #1 0x00007f0ad89ddade PyEval_EvalFrameEx (libpython2.7.so.1.0) #2 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #3 0x00007f0ad896a7bd function_call (libpython2.7.so.1.0) #4 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #5 0x00007f0ad89547e5 instancemethod_call (libpython2.7.so.1.0) #6 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #7 0x00007f0ad899c927 slot_tp_init (libpython2.7.so.1.0) #8 0x00007f0ad899b65f type_call (libpython2.7.so.1.0) #9 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #10 0x00007f0ad89db615 PyEval_EvalFrameEx (libpython2.7.so.1.0) #11 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #12 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #13 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #14 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #15 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #16 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) Stack trace of thread 13333: #0 0x00007f0acf3e5de4 PySSL_sslwrap (_ssl.so) #1 0x00007f0ad89ddade PyEval_EvalFrameEx (libpython2.7.so.1.0) #2 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #3 0x00007f0ad896a7bd function_call (libpython2.7.so.1.0) #4 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #5 0x00007f0ad89547e5 instancemethod_call (libpython2.7.so.1.0) #6 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #7 0x00007f0ad899c927 slot_tp_init (libpython2.7.so.1.0) #8 0x00007f0ad899b65f type_call (libpython2.7.so.1.0) #9 0x00007f0ad89458d3 PyObject_Call (libpython2.7.so.1.0) #10 0x00007f0ad89db615 PyEval_EvalFrameEx (libpython2.7.so.1.0) #11 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #12 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #13 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #14 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #15 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #16 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #17 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #18 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #19 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #20 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #21 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #22 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #23 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #24 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #25 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #26 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #27 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #28 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #29 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #30 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #31 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #32 0x00007f0ad89dcc76 PyEval_EvalFrameEx (libpython2.7.so.1.0) #33 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #34 0x00007f0ad89dcb74 PyEval_EvalFrameEx (libpython2.7.so.1.0) #35 0x00007f0ad89de490 PyEval_EvalCodeEx (libpython2.7.so.1.0) #36 0x00007f0ad89de589 PyEval_EvalCode (libpython2.7.so.1.0) #37 0x00007f0ad89f793f run_mod (libpython2.7.so.1.0) #38 0x00007f0ad89f8b62 PyRun_FileExFlags (libpython2.7.so.1.0) #39 0x00007f0ad89f9d77 PyRun_SimpleFileExFlags (libpython2.7.so.1.0) #40 0x00007f0ad8a0b30a Py_Main (libpython2.7.so.1.0) #41 0x00007f0ad7c32fe0 __libc_start_main (libc.so.6) #42 0x000000000040071e _start (python2.7) Oct 19 01:37:53 example.com /usr/share/ipa/ipadnssecd/ipa-ods-exporter.py[14535]: Kerberos principal: ipa-ods-exporter/example.com
Probably this is reason why we are getting segmentation fault.
There is null pointer dereference inside python standard library
#0 0x00007f3c4a66dde4 in newPySSLObject (ciphers=0x7f3c4544eeb4 "DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2", cacerts_file=<optimized out>, proto_version=PY_SSL_VERSION_SSL23, certreq=<optimized out>, socket_type=<optimized out>, cert_file=0x0, key_file=0x0, Sock=0x7f3c3fbafc30) at /usr/src/debug/Python-2.7.8/Modules/_ssl.c:317 317 self->ctx->options &= ~(SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3); (gdb) p self->ctx $1 = (SSL_CTX *) 0x0 bt #0 0x00007f3c4a66dde4 in newPySSLObject (ciphers=0x7f3c4544eeb4 "DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2", cacerts_file=<optimized out>, proto_version=PY_SSL_VERSION_SSL23, certreq=<optimized out>, socket_type=<optimized out>, cert_file=0x0, key_file=0x0, Sock=0x7f3c3fbafc30) at /usr/src/debug/Python-2.7.8/Modules/_ssl.c:317 #1 PySSL_sslwrap (self=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-2.7.8/Modules/_ssl.c:455 #2 0x00007f3c53c65ade in call_function (oparg=<optimized out>, pp_stack=<optimized out>) at /usr/src/debug/Python-2.7.8/Python/ceval.c:4110 #3 PyEval_EvalFrameEx (f=<unknown at remote 0x2e10960>, throwflag=1085658968) at /usr/src/debug/Python-2.7.8/Python/ceval.c:2753 #4 0x00007f3c53c66490 in PyEval_EvalCodeEx (co=0x7f3c45436730, globals=0x0, locals=< at remote 0x23e20a0>, locals@entry=0x0, args=0x2, ....
This is called at ipa-upgradeconfig:1457 ca.restart(...), where the check if CA is up is, which uses https (ipapython.dogtag.ca_status).
status, reason, headers, body = unauthenticated_https_request( ca_host, ca_port, '/ca/admin/ca/getStatus')
tail of ipaupgrade.log:
2014-10-23T23:42:35Z DEBUG stderr= 2014-10-23T23:42:35Z DEBUG wait_for_open_ports: localhost [8080, 8443] timeout 300 2014-10-23T23:42:38Z DEBUG Waiting until the CA is running 2014-10-23T23:42:38Z DEBUG request 'https://vm.example.com:443/ca/admin/ca/getStatus' 2014-10-23T23:42:38Z DEBUG request body ''
Fortunately it is last step of upgrading process, so ipa was upgraded correctly
Thanks for assessment. We will need the Bugzilla filed, issue fixed&Requires bumped...
Core dump core.22983.tar.gz
Fedora bug:
http://bugs.python.org/issue22717
Fedora BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1156519
This hits only F21.
Problem fixed in python-2.7.8-5.fc21
No need to create downstream Bugzilla, this is fixed.
Metadata Update from @mbasti: - Issue assigned to mbasti - Issue set to the milestone: FreeIPA 4.1.1
Login to comment on this ticket.