https://bugzilla.redhat.com/show_bug.cgi?id=903758 (Red Hat Enterprise Linux 6)
+++ This bug was initially created as a clone of Bug #902474 +++ Description of problem: When upgrading IPA from 2.2 (RHEL6.3) to 3.0 (from RHEL6.4 repos), I'm seeing certmonger errors: Updating : ipa-server-3.0.0-22.el6.x86_64 49/89 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n ocspSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n subsystemCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n Server-Cert cert-pki-ca -c dogtag-ipa-renew-agent -P XXXXXXXX' returned non-zero exit status 1 Unable to find certmonger request ID for auditSigning Cert Updating : ipa-server-selinux-3.0.0-22.el6.x86_64 50/89 Before the update I pre-updated certmonger to be sure that the dogtag-ipa-renew-agent CA was there. [root@rhel6-5 ~]# yum update certmonger ... Updated: certmonger.x86_64 0:0.61-3.el6 Dependency Updated: libtalloc.x86_64 0:2.0.7-2.el6 libtevent.x86_64 0:0.9.17-1.el6 Complete! [root@rhel6-5 ~]# getcert list-cas CA 'SelfSign': is-default: no ca-type: INTERNAL:SELF next-serial-number: 01 CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit CA 'certmaster': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/certmaster-submit CA 'dogtag-ipa-renew-agent': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/dogtag-ipa-renew-agent-submit I also checked the state of some directories before the ipa-server update: [root@rhel6-5 ~]# ls -ld /var/lib/pki-ca drwxrwx---. 11 pkiuser pkiuser 4096 Jan 21 12:59 /var/lib/pki-ca [root@rhel6-5 ~]# ls -ld /var/lib/pki-ca/alias/ drwxrwx---. 2 pkiuser pkiuser 4096 Jan 21 12:59 /var/lib/pki-ca/alias/ [root@rhel6-5 ~]# ls -ld /var/lib/pki-ca/alias drwxrwx---. 2 pkiuser pkiuser 4096 Jan 21 12:59 /var/lib/pki-ca/alias Yet, I see this in ipaupgrade.log: 2013-01-21T18:17:13Z DEBUG args=/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-p ki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX 2013-01-21T18:17:13Z DEBUG stdout=The location "/var/lib/pki-ca/alias" must be a directory. 2013-01-21T18:17:13Z DEBUG stderr= 2013-01-21T18:17:13Z ERROR certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 Version-Release number of selected component (if applicable): RHEL6.3 IPA 2.2 -> 3.0 upgrade: ipa-server-3.0.0-22.el6.x86_64 certmonger-0.61-3.el6.x86_64 How reproducible: always Steps to Reproduce: 1. Install IPA on RHEL6.3 server 2. add RHEL6.4 repos 3. yum -y update certmonger 4. yum -y update ipa-server Actual results: shows error above. Expected results: no error. Additional info: --- Additional comment from RHEL Product and Program Management on 2013-01-21 14:03:39 EST --- Since this bug report was entered in bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Scott Poore on 2013-01-21 15:40:01 EST --- running additional tests working with Rob on this: 1. pre-update certmonger, restart messagebus, restart certmonger, update ipa-server: failed with same error. 2. pre-update certmonger and messagebus, restart messagebus, restart certmonger, update ipa-server: failed with same error. --- Additional comment from Rob Crittenden on 2013-01-21 17:06:56 EST --- I notice that downgrading to certmonger-0.56-1 and running 'getcert list-cas' still lists dogtag-ipa-renew-agent as an available CA type even though its underlying provider file is gone. I wasn't able to make this go away via various restarts of certmonger/messagebus. --- Additional comment from Rob Crittenden on 2013-01-21 17:33:55 EST --- Found it in a file in /var/lib/certmonger/cas --- Additional comment from Rob Crittenden on 2013-01-21 18:36:40 EST --- selinux-policy-3.7.19-193.el6.noarch Looks like SELinux exceptions: type=AVC msg=audit(1358810171.195:87): avc: denied { search } for pid=9243 comm="certmonger" name="pki-ca" dev=dm-0 ino=5659 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir type=AVC msg=audit(1358810171.195:87): avc: denied { getattr } for pid=9243 comm="certmonger" path="/var/lib/pki-ca/alias" dev=dm-0 ino=5660 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir type=AVC msg=audit(1358810171.201:88): avc: denied { getattr } for pid=9793 comm="certmonger" path="/var/lib/pki-ca/alias/cert8.db" dev=dm-0 ino=5977 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=file type=AVC msg=audit(1358810171.201:89): avc: denied { read } for pid=9793 comm="certmonger" name="cert8.db" dev=dm-0 ino=5977 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=file type=AVC msg=audit(1358810171.201:89): avc: denied { open } for pid=9793 comm="certmonger" name="cert8.db" dev=dm-0 ino=5977 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=file type=AVC msg=audit(1358810171.213:90): avc: denied { write } for pid=9794 comm="certmonger" name="cert8.db" dev=dm-0 ino=5977 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=file --- Additional comment from Scott Poore on 2013-01-21 18:44:07 EST --- And I saw similar for a beaker test job: ---- time->Sun Jan 20 11:14:57 2013 type=SYSCALL msg=audit(1358698497.969:606): arch=c000003e syscall=4 success=no exit=-13 a0=2507660 a1=7fff722d10f0 a2=7fff722d10f0 a3=3011139180 items=0 ppid=1 pid=13659 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certmonger" exe=2F7573722F7362696E2F636572746D6F6E676572202864656C6574656429 subj=unconfined_u:system_r:certmonger_t:s0 key=(null) type=AVC msg=audit(1358698497.969:606): avc: denied { search } for pid=13659 comm="certmonger" name="pki-ca" dev=dm-0 ino=2624285 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir ---- time->Sun Jan 20 11:14:57 2013 type=SYSCALL msg=audit(1358698497.940:605): arch=c000003e syscall=4 success=no exit=-13 a0=2507660 a1=7fff722d10f0 a2=7fff722d10f0 a3=7fff722d0e70 items=0 ppid=1 pid=13659 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certmonger" exe=2F7573722F7362696E2F636572746D6F6E676572202864656C6574656429 subj=unconfined_u:system_r:certmonger_t:s0 key=(null) type=AVC msg=audit(1358698497.940:605): avc: denied { search } for pid=13659 comm="certmonger" name="pki-ca" dev=dm-0 ino=2624285 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir ---- time->Sun Jan 20 11:14:57 2013 type=SYSCALL msg=audit(1358698497.995:607): arch=c000003e syscall=4 success=no exit=-13 a0=2507660 a1=7fff722d10f0 a2=7fff722d10f0 a3=3011139180 items=0 ppid=1 pid=13659 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certmonger" exe=2F7573722F7362696E2F636572746D6F6E676572202864656C6574656429 subj=unconfined_u:system_r:certmonger_t:s0 key=(null) type=AVC msg=audit(1358698497.995:607): avc: denied { search } for pid=13659 comm="certmonger" name="pki-ca" dev=dm-0 ino=2624285 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir ---- time->Sun Jan 20 11:14:58 2013 type=SYSCALL msg=audit(1358698498.084:608): arch=c000003e syscall=4 success=no exit=-13 a0=2509680 a1=7fff722d10f0 a2=7fff722d10f0 a3=3011139180 items=0 ppid=1 pid=13659 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certmonger" exe=2F7573722F7362696E2F636572746D6F6E676572202864656C6574656429 subj=unconfined_u:system_r:certmonger_t:s0 key=(null) type=AVC msg=audit(1358698498.084:608): avc: denied { search } for pid=13659 comm="certmonger" name="pki-ca" dev=dm-0 ino=2624285 scontext=unconfined_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:pki_ca_var_lib_t:s0 tclass=dir --- Additional comment from Miroslav Grepl on 2013-01-22 01:19:13 EST --- We label it in Fedora as # matchpathcon /var/lib/pki-ca/alias/cert8.d /var/lib/pki-ca/alias/cert8.d system_u:object_r:pki_tomcat_cert_t:s0 and call optional_policy(` pki_rw_tomcat_cert(certmonger_t) ') --- Additional comment from Matthew Harmsen on 2013-01-22 18:11:19 EST --- alee checked the following into the 'IPA_v2_RHEL_6_ERRATA_BRANCH': * commit ca5fa67a0d0797d1f4c54bbd4d9db3661eaeb8c9 Author: Ade Lee <alee@redhat.com> Date: Tue Jan 22 11:15:16 2013 -0800 Resolves #902474 - upgrading IPA from 2.2 to 3.0 sees certmonger errors --- Additional comment from errata-xmlrpc on 2013-01-22 18:33:12 EST --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHSA-2012:13959-06 http://errata.devel.redhat.com/errata/show/13959 --- Additional comment from Scott Poore on 2013-01-23 15:56:21 EST --- hmm...I no longer see the AVCs but, do still see the error messages when I do an initial upgade. Because of how my tests work, I install, upgrade, uninstall/downgrade, and start over to test something else. Well, on subsequent tests, I do not see the error. So, testing to confirm if the certmonger errors are SELinux related, I 1. rebuilt a VM to rhel6.3 2. install 2.2 version of IPA 3. pointed to 6.4 repos 4. setenforce 0 5. semodule -DB # disable don't audit rules to pick up more 6. yum -u update ipa-server Then I see same errors as originally posted. Will post logs too. --- Additional comment from Scott Poore on 2013-01-23 15:58:45 EST --- Created attachment 686265 audit log from ipa upgade after selinux set to permissive and disabling don't audit --- Additional comment from Scott Poore on 2013-01-23 16:00:13 EST --- Created attachment 686267 ipa upgrade log with certmonger failures --- Additional comment from Scott Poore on 2013-01-23 16:53:18 EST --- Ok, from fresh install: [root@rhel6-2 ~]# rpm -q ipa-server ipa-server-2.2.0-16.el6.x86_64 [root@rhel6-2 ~]# ipa-getcert list-cas CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit [root@rhel6-2 ~]# getcert list-cas CA 'SelfSign': is-default: no ca-type: INTERNAL:SELF next-serial-number: 01 CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit CA 'certmaster': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/certmaster-submit add repos and the update just dbus and certmonger: [root@rhel6-2 ~]# yum -y update dbus certmonger Loaded plugins: product-id, security, subscription-manager Updating certificate-based repositories. Unable to read consumer identity This machine has not been registered and therefore has no access to security and other critical updates. Please register using subscription-manager. Repository 'rhel63-optional' is missing name in configuration, using id beaker-client | 1.3 kB 00:00 beaker-client/primary | 7.2 kB 00:00 beaker-client 35/35 mytestrepo1 | 3.9 kB 00:00 mytestrepo1/primary_db | 3.1 MB 00:02 mytestrepo2 | 3.7 kB 00:00 mytestrepo2/primary_db | 1.3 MB 00:01 mytestrepo3 | 1.3 kB 00:00 mytestrepo3/primary | 3.6 kB 00:00 mytestrepo3 7/7 mytestrepo4 | 1.3 kB 00:00 mytestrepo4/primary | 4.3 kB 00:00 mytestrepo4 13/13 mytestrepo5 | 3.9 kB 00:00 mytestrepo5/primary_db | 3.2 MB 00:02 rhel63-optional | 3.8 kB 00:00 rhel63-optional/primary_db | 1.3 MB 00:01 rhel63z | 2.2 kB 00:00 rhel63z/primary_db | 4.0 MB 00:03 Setting up Update Process Resolving Dependencies --> Running transaction check ---> Package certmonger.x86_64 0:0.56-1.el6 will be updated ---> Package certmonger.x86_64 0:0.61-3.el6 will be an update --> Processing Dependency: libtevent.so.0(TEVENT_0.9.9)(64bit) for package: certmonger-0.61-3.el6.x86_64 --> Processing Dependency: libtalloc.so.2(TALLOC_2.0.2)(64bit) for package: certmonger-0.61-3.el6.x86_64 --> Running transaction check ---> Package libtalloc.x86_64 0:2.0.1-1.1.el6 will be updated ---> Package libtalloc.x86_64 0:2.0.7-2.el6 will be an update ---> Package libtevent.x86_64 0:0.9.8-8.el6 will be updated ---> Package libtevent.x86_64 0:0.9.17-1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved =============================================================================== ======================== Package Arch Version Repository Size =============================================================================== ======================== Updating: certmonger x86_64 0.61-3.el6 mytestrepo1 280 k Updating for dependencies: libtalloc x86_64 2.0.7-2.el6 mytestrepo1 20 k libtevent x86_64 0.9.17-1.el6 mytestrepo1 24 k Transaction Summary =============================================================================== ======================== Upgrade 3 Package(s) Total download size: 324 k Downloading Packages: (1/3): certmonger-0.61-3.el6.x86_64.rpm | 280 kB 00:00 (2/3): libtalloc-2.0.7-2.el6.x86_64.rpm | 20 kB 00:00 (3/3): libtevent-0.9.17-1.el6.x86_64.rpm | 24 kB 00:00 ------------------------------------------------------------------------------- ------------------------ Total 273 kB/s | 324 kB 00:01 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : libtalloc-2.0.7-2.el6.x86_64 1/6 Updating : libtevent-0.9.17-1.el6.x86_64 2/6 Updating : certmonger-0.61-3.el6.x86_64 3/6 Cleanup : certmonger-0.56-1.el6.x86_64 4/6 Cleanup : libtevent-0.9.8-8.el6.x86_64 5/6 Cleanup : libtalloc-2.0.1-1.1.el6.x86_64 6/6 mytestrepo1/productid | 1.7 kB 00:00 mytestrepo5/productid | 1.7 kB 00:00 Installed products updated. Verifying : libtevent-0.9.17-1.el6.x86_64 1/6 Verifying : certmonger-0.61-3.el6.x86_64 2/6 Verifying : libtalloc-2.0.7-2.el6.x86_64 3/6 Verifying : certmonger-0.56-1.el6.x86_64 4/6 Verifying : libtalloc-2.0.1-1.1.el6.x86_64 5/6 Verifying : libtevent-0.9.8-8.el6.x86_64 6/6 Updated: certmonger.x86_64 0:0.61-3.el6 Dependency Updated: libtalloc.x86_64 0:2.0.7-2.el6 libtevent.x86_64 0:0.9.17-1.el6 Complete! [root@rhel6-2 ~]# getcert list-cas CA 'SelfSign': is-default: no ca-type: INTERNAL:SELF next-serial-number: 01 CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit CA 'certmaster': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/certmaster-submit CA 'dogtag-ipa-renew-agent': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/dogtag-ipa-renew-agent-submit [root@rhel6-2 ~]# rpm -q ipa-server ipa-server-2.2.0-16.el6.x86_64 [root@rhel6-2 ~]# service messagebus restart Stopping system message bus: [ OK ] Starting system message bus: [ OK ] [root@rhel6-2 ~]# service certmonger restart Stopping certmonger: [ OK ] Starting certmonger: [ OK ] [root@rhel6-2 ~]# ls /var/lib/certmonger/cas/ 20130107174444 20130107174445 20130107174445-1 [root@rhel6-2 ~]# cat /var/lib/certmonger/cas/* id=SelfSign ca_is_default=0 ca_type=INTERNAL:SELF ca_internal_serial=01 id=IPA ca_is_default=0 ca_type=EXTERNAL ca_external_helper=/usr/libexec/certmonger/ipa-submit id=certmaster ca_is_default=0 ca_type=EXTERNAL ca_external_helper=/usr/libexec/certmonger/certmaster-submit SO...at this point, getcert listcas does show dogtag-ipa-renew-agent but, there doesn't appear to be a file for it in /var/lib/certmonger/cas. Now, to note, if I upgrade, downgrade, upgade again, that file is left behind...I'm testing what happens if it's removed before initial install and before upgade. will post update when done --- Additional comment from Scott Poore on 2013-01-23 17:12:50 EST --- ok, I uninstalled/downgraded and cleaned up /var/lib/certmonger/cas/ by deleting the 4 files that matched the CAs. Then I did install and finally upgrade. Now I also see the errors I was seeing on initial install only. Will test more before upgrade to see what's there and what can be done. --- Additional comment from Scott Poore on 2013-01-23 17:41:04 EST --- Ok, twice in a row now when I pre-update certmonger and dbus (like in comment #13), I no longer see that error. Testing on a freshly installed server instead of re-running from reverted virsh snapshot to see if I see the same. --- Additional comment from Scott Poore on 2013-01-23 22:07:40 EST --- Ok, I've run tests several different times and so far, now it does look like I no longer see those errors is if upgrade certmonger and dbus first.
attachment freeipa-rcrit-1082-certmonger-master.patch
master: 41d11f4[[BR]] ipa-3-1: 771624b[[BR]] ipa-3-0: 2b6ea84
Metadata Update from @rcritten: - Issue assigned to rcritten - Issue set to the milestone: FreeIPA 3.0.3 (bug fixing)
Login to comment on this ticket.