I have a FreeIPA server running on CentOS 7 and AFAICT everything is working fine.
When I try to upgrade a current client (that works fine as a client) using the "ipa-replica-install --setup-ca" command it fails at step 28/44 (restarting directory server):
Run connection check to master Connection check OK Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd Done configuring NTP daemon (ntpd). Configuring directory server (dirsrv). Estimated time: 1 minute [1/44]: creating directory server user [2/44]: creating directory server instance [3/44]: updating configuration in dse.ldif [4/44]: restarting directory server [5/44]: adding default schema [6/44]: enabling memberof plugin [7/44]: enabling winsync plugin [8/44]: configuring replication version plugin [9/44]: enabling IPA enrollment plugin [10/44]: enabling ldapi [11/44]: configuring uniqueness plugin [12/44]: configuring uuid plugin [13/44]: configuring modrdn plugin [14/44]: configuring DNS plugin [15/44]: enabling entryUSN plugin [16/44]: configuring lockout plugin [17/44]: configuring topology plugin [18/44]: creating indices [19/44]: enabling referential integrity plugin [20/44]: configuring certmap.conf [21/44]: configure autobind for root [22/44]: configure new location for managed entries [23/44]: configure dirsrv ccache [24/44]: enabling SASL mapping fallback [25/44]: restarting directory server [26/44]: creating DS keytab [27/44]: retrieving DS Certificate [28/44]: restarting directory server ipa : CRITICAL Failed to restart the directory server (Command '/bin/systemctl restart dirsrv@DOMAIN-TLD.service' returned non-zero exit status 1). See the installation log for details. [29/44]: setting up initial replication [error] error: [Errno 111] Connection refused Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
I only got an error prior to this about the reverse DNS for the master IPs. They are correct, just not manager by the master.
This is what I got in ipareplica-install.log:
Jan 13 19:03:04 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:04.755486530 +0100] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 Jan 13 19:03:04 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:04.784878621 +0100] slapd shutting down - closing down internal subsystems and plugins Jan 13 19:03:04 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:04.858081378 +0100] Waiting for 4 database threads to stop Jan 13 19:03:05 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:05.605052250 +0100] All database threads now stopped Jan 13 19:03:05 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:05.666811570 +0100] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects Jan 13 19:03:06 server.domain.tld ns-slapd[17469]: [13/Jan/2017:19:03:06.219145820 +0100] slapd stopped. Jan 13 19:03:06 server.domain.tld systemd[1]: Starting 389 Directory Server DOMAIN-TLD.... Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.138743278 +0100] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.175397815 +0100] SSL alert: Security Initialization: Enabling default cipher set. Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.200221890 +0100] SSL alert: Configured NSS Ciphers Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.226045955 +0100] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.251131160 +0100] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.276228181 +0100] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.301297476 +0100] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.326403016 +0100] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.359744769 +0100] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.385073945 +0100] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.410158085 +0100] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.435231257 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.460304673 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.485394415 +0100] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.513958682 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.543913140 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.572448221 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.602398075 +0100] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.640473775 +0100] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.669424626 +0100] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.697966131 +0100] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.723012705 +0100] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.748109523 +0100] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.773182437 +0100] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.798282788 +0100] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.825546446 +0100] SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.856847055 +0100] SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.906991932 +0100] SSL failure: None of the cipher are valid Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.940428842 +0100] ERROR: SSL2 Initialization Failed. Disabling SSL2. Jan 13 19:03:07 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:07.966297470 +0100] 389-Directory/1.3.5.10 B2016.341.2222 starting up Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.046608261 +0100] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.107589296 +0100] Can't find certificate Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.245799593 +0100] Can't get private key from cert Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.283142981 +0100] Error: unable to initialize attrcrypt system for userRoot Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.304841024 +0100] start: Failed to start databases, err=-1 BDB0092 Unknown error: -1 Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.329860505 +0100] Failed to start database plugin ldbm database Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.361971071 +0100] WARNING: ldbm instance userRoot already exists Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.388361388 +0100] ldbm_config_read_instance_entries: failed to add instance entry cn=userRoot,cn=ldbm database,cn=plugins,cn=config Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.418084240 +0100] ldbm_config_load_dse_info: failed to read instance entries Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.438704240 +0100] start: Loading database configuration failed Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.463769102 +0100] Failed to start database plugin ldbm database Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.542292136 +0100] Error: Failed to resolve plugin dependencies Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.563914469 +0100] Error: betxnpreoperation plugin 7-bit check is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.588999281 +0100] Error: preoperation plugin Account Usability Plugin is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.614076116 +0100] Error: accesscontrol plugin ACL Plugin is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.639166316 +0100] Error: preoperation plugin ACL preoperation is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.664276432 +0100] Error: betxnpreoperation plugin Auto Membership Plugin is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.689333682 +0100] Error: object plugin Class of Service is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.718022500 +0100] Error: preoperation plugin deref is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.739687142 +0100] Error: preoperation plugin HTTP Client is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.764745221 +0100] Error: preoperation plugin IPA DNS is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.789831664 +0100] Error: object plugin IPA Lockout is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.814924508 +0100] Error: betxnpostoperation plugin IPA MODRDN is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.840000826 +0100] Error: object plugin IPA Topology Configuration is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.865097652 +0100] Error: preoperation plugin IPA UUID is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.890193162 +0100] Error: preoperation plugin ipa-winsync is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.915258590 +0100] Error: extendedop plugin ipa_enrollment_extop is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.940352038 +0100] Error: preoperation plugin ipaUniqueID uniqueness is not started Jan 13 19:03:08 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.976370717 +0100] Error: preoperation plugin krbCanonicalName uniqueness is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:08.999034730 +0100] Error: preoperation plugin krbPrincipalName uniqueness is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.024108816 +0100] Error: database plugin ldbm database is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.049189342 +0100] Error: object plugin Legacy Replication Plugin is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.074291530 +0100] Error: betxnpreoperation plugin Linked Attributes is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.095427749 +0100] Error: betxnpreoperation plugin Managed Entries is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.120517023 +0100] Error: betxnpostoperation plugin MemberOf Plugin is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.149568254 +0100] Error: object plugin Multimaster Replication Plugin is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.170724261 +0100] Error: preoperation plugin netgroup uniqueness is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.195816459 +0100] Error: betxnpostoperation plugin referential integrity postoperation is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.232047955 +0100] Error: object plugin Roles Plugin is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.254502905 +0100] Error: preoperation plugin sudorule name uniqueness is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.279550874 +0100] Error: object plugin USN is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.304637234 +0100] Error: object plugin Views is not started Jan 13 19:03:09 server.domain.tld ns-slapd[17601]: [13/Jan/2017:19:03:09.329714831 +0100] Error: extendedop plugin whoami is not started Jan 13 19:03:09 server.domain.tld systemd[1]: dirsrv@DOMAIN-TLD.service: main process exited, code=exited, status=1/FAILURE Jan 13 19:03:09 server.domain.tld systemd[1]: Failed to start 389 Directory Server DOMAIN-TLD..
It loos really similar to https://fedorahosted.org/freeipa/ticket/5561 but this is already fixed (and the fix is in CentOS)
about the NSS errors, may want to review some certutil outputs (change the EXAMPLE-COM string to your domain)
certutil -K -d /etc/dirsrv/slapd-EXAMPLE-COM/ -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt certutil -L -d /etc/dirsrv/slapd-EXAMPLE-COM/ certutil -L -d /etc/dirsrv/slapd-EXAMPLE-COM/ -n "Server-Cert"
what is the replica version before and after the upgrade? was ipa-server-upgrade ran?
This is all run on the Master:
certutil -K -d /etc/dirsrv/slapd-EXAMPLE-COM/ -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx NSS Certificate DB:Server-Cert
certutil -L -d /etc/dirsrv/slapd-EXAMPLE-COM/
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
EXAMPLE.COM IPA CA CT,C,C Server-Cert u,u,u
certutil -L -d /etc/dirsrv/slapd-EXAMPLE-COM/ -n "Server-Cert"
Certificate: Data: Version: 3 (0x2) Serial Number: 8 (0x8) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=EXAMPLE.COM" Validity: Not Before: Sat Jan 07 00:48:09 2017 Not After : Tue Jan 08 00:48:09 2019 Subject: "CN=ipa.example.com,O=EXAMPLE.COM" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: c9:69:f3:89:9b:e8:4d:18:b4:1f:ed:b2:dc:59:25:0a: eb:51:33:3c:33:e1:08:60:c1:64:56:45:b4:b1:69:c5: 23:c0:75:0f:87:b4:b8:6c:1d:d4:f3:6a:c6:dc:a7:1a: b7:0d:cd:37:85:15:b5:73:97:1f:2e:c8:6d:fc:8d:ef: d2:51:3d:f8:ea:dc:12:8b:5e:fe:f3:58:3f:35:0f:d7: eb:cf:ae:1a:1e:6b:3c:f1:9d:3a:90:f3:71:1e:08:bb: 68:eb:55:28:2d:93:5c:5b:81:b5:d7:a7:8d:78:fa:25: 25:61:16:14:f6:b7:89:64:aa:d6:5f:fe:e0:0d:82:70: 86:d0:66:5d:c8:eb:98:be:d9:6b:54:2d:2b:f3:15:8c: e3:73:3b:70:b3:7d:e1:d5:b8:6f:2c:1f:07:d2:d6:80: a6:fe:ba:bf:89:b1:a9:0b:d8:ba:1d:7d:11:a9:a9:1e: 07:59:b5:82:67:87:9d:c0:4d:fc:ac:e0:2e:9d:b8:3a: 9e:90:7c:30:19:e3:b0:35:4a:e9:c5:96:92:6c:7a:42: 17:18:61:f2:ff:80:ac:e6:15:8a:47:44:16:00:d9:48: d9:ad:bb:0a:af:99:3a:fb:8a:db:2b:e2:24:82:b8:10: da:a4:43:b2:2a:99:98:68:ab:bb:a6:4f:43:9e:c7:09 Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: 7a:2b:97:fd:00:92:2b:c6:72:6c:96:25:ba:3c:5d:bb: c6:e7:0c:46
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.example.com/ca/ocsp" Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment Name: Extended Key Usage TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate Name: CRL Distribution Points Distribution point: URI: "http://ipa-ca.example.com/ipa/crl/MasterCRL.bin" CRL issuer: Directory Name: "CN=Certificate Authority,O=ipaca" Name: Certificate Subject Key ID Data: 85:c1:b3:b4:f4:bf:64:0c:32:a5:fe:2a:41:db:c6:88: ec:f6:3c:38 Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 2a:93:41:92:45:a6:61:fd:99:87:3b:ea:b4:25:74:9b: 55:0e:86:50:f9:6a:2a:3c:95:78:50:ee:f2:43:e9:c1: d9:b0:1f:f6:8f:62:8b:92:a5:1c:67:26:44:d2:e9:74: a5:d0:ce:e9:25:7b:a2:82:f7:76:af:d5:f6:96:be:b3: 01:87:6b:d3:03:b8:52:11:8e:87:c0:1f:06:f6:e7:84: 41:03:1a:88:7a:c9:66:d1:9f:2e:f4:75:35:99:36:22: 4f:00:05:d4:c8:67:28:ff:12:32:3b:2f:8d:90:ef:ba: cd:0a:bb:b6:8f:26:80:fa:cc:d4:71:5e:ab:d3:55:cc: bc:8b:8e:89:83:5c:29:7f:02:3d:ce:d0:e0:20:32:9b: ff:9a:4c:d4:ad:06:14:66:26:96:f4:c4:5a:41:a1:4f: 82:f7:94:76:fa:ba:af:a5:d1:7e:a0:07:84:f6:c9:93: 31:b8:28:44:60:db:46:5f:ee:0a:88:b6:3e:f3:e5:bd: a6:3a:0e:43:12:c8:f8:43:ca:0c:f3:8c:27:58:ad:a5: 66:e7:2a:61:d4:02:20:10:86:d2:2b:20:05:7f:01:69: 74:30:43:3e:af:0a:0e:ee:34:68:a5:73:4f:19:a9:9d: 92:17:90:f2:4e:46:fb:8d:48:02:c7:7f:c8:b6:0f:21 Fingerprint (SHA-256): D9:B1:C6:02:87:A9:05:E6:8A:8D:AD:5D:9B:9F:F4:AC:E5:FD:F4:FE:49:BF:D1:86:89:36:54:9A:B7:87:A7:7D Fingerprint (SHA1): 1D:F1:DF:06:1C:81:11:58:2A:4C:F7:B7:0D:DE:E4:9B:E8:06:C9:70 Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User
nsSSLPersonalitySSL: Server-Cert
Replica was never setup (ipa-replica-install always fails), it's just a client for now. (master was never upgraded)
it is a 2K RSA key, cert and key seem ok, so not the issue on the replica, if strace or gdb are available, may be try to troubleshoot the dirsrv early start-up like this (it may not be the best procedure, some of my peers may chime in) systemctl stop dirsrv@EXAMPLE-COM.service systemctl status dirsrv@EXAMPLE-COM.service vi /etc/systemd/system/dirsrv.target.wants/dirsrv\@EXAMPLE-COM.service ... [Service] Environment=DEBUG_SLEEP=30 ... systemctl daemon-reload systemctl start dirsrv@EXAMPLE-COM.service & systemctl status dirsrv@EXAMPLE-COM.service and either strace -Tttvfs 1024 -o /tmp/strace.out -p `pidof ns-slapd` and check for open("/etc/dirsrv/slapd-EXAMPLE-COM/secmod.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/cert8.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/key3.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/pin.txt", O_RDONLY or: gdb -p `pidof ns-slapd` (gdb) cont
OK, I think I found the error. On the logs I get something like this before the failing dirsrv restart:
2017-01-14T03:41:28Z DEBUG [27/44]: retrieving DS Certificate 2017-01-14T03:41:28Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2017-01-14T03:41:28Z DEBUG Starting external process 2017-01-14T03:41:28Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-EXAMPLE-COM/ -L -n EXAMPLE.COM IPA CA -a 2017-01-14T03:41:28Z DEBUG Process finished, return code=255 2017-01-14T03:41:28Z DEBUG stdout= 2017-01-14T03:41:28Z DEBUG stderr=certutil: Could not find cert: EXAMPLE.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
So, when the process stopped, I run the command again:
# /usr/bin/certutil -d /etc/dirsrv/slapd-EXAMPLE-COM/ -L -n EXAMPLE.COM IPA CA -a certutil: Could not find cert: EXAMPLE.COM : PR_FILE_NOT_FOUND_ERROR: File not found
and thought "wait... something is missing there":
# /usr/bin/certutil -d /etc/dirsrv/slapd-EXAMPLE-COM/ -L -n "EXAMPLE.COM IPA CA" -a -----BEGIN CERTIFICATE----- <strip> -----END CERTIFICATE-----
So, could this be the problem?
you may need to replace the string EXAMPLE-COM with the realm/domain used in the replica's environment, it may be DOMAIN-TLD like seen in the first post, but then we see EXAMPLE-COM, may be there is some confusion? on replica and master, review the realm and domain from /etc/ipa/default.conf
No, the realm/domain is fine, i'm just replacing them to hide it. Got enough ssh brute forces by now :P
Replying to [comment:4 msauton]:
{{{ strace -Tttvfs 1024 -o /tmp/strace.out -p pidof ns-slapd and check for open("/etc/dirsrv/slapd-EXAMPLE-COM/secmod.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/cert8.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/key3.db", O_RDONLY open("/etc/dirsrv/slapd-EXAMPLE-COM/pin.txt", O_RDONLY }}}
pidof ns-slapd
Everything looks fine to me :(
28812 17:21:32.554696 open("/etc/dirsrv/slapd-EXAMPLE-COM/secmod.db", O_RDONLY) = 11 <0.000050> 28812 17:21:32.554836 fstat(11, {st_dev=makedev(9, 2), st_ino=134186, st_mode=S_IFREG|0600, st_nlink=1, st_uid=389, st_gid=0, st_blksize=4096, st_blocks=24, st_size=16384, st_atime=2017/01/14-17:11:30, st_mtime=2017/01/14-17:05:49, st_ctime=2017/01/14-17:05:49}) = 0 <0.000050> 28812 17:21:32.555041 close(11) = 0 <0.000037> 28812 17:21:32.555169 open("/etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif.startOK", O_RDONLY) = 11 <0.000051> 28812 17:21:32.555311 fstat(11, {st_dev=makedev(9, 2), st_ino=134193, st_mode=S_IFREG|0600, st_nlink=2, st_uid=389, st_gid=389, st_blksize=4096, st_blocks=184, st_size=91010, st_atime=2017/01/14-17:05:48, st_mtime=2017/01/14-17:05:46, st_ctime=2017/01/14-17:05:50}) = 0 <0.000032> 28812 17:21:32.555489 close(11) = 0 <0.000037> 28812 17:21:32.555616 open("/etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif", O_RDONLY) = 11 <0.000049> 28812 17:21:32.555757 fstat(11, {st_dev=makedev(9, 2), st_ino=134122, st_mode=S_IFREG|0600, st_nlink=1, st_uid=389, st_gid=389, st_blksize=4096, st_blocks=184, st_size=91010, st_atime=2017/01/14-17:21:01, st_mtime=2017/01/14-17:05:50, st_ctime=2017/01/14-17:05:50}) = 0 <0.000032> 28812 17:21:32.555957 close(11) = 0 <0.000037> 28812 17:21:32.556090 open("/etc/dirsrv/slapd-EXAMPLE-COM/pin.txt", O_RDONLY) = 11 <0.000052> 28812 17:21:32.556235 fstat(11, {st_dev=makedev(9, 2), st_ino=134192, st_mode=S_IFREG|0400, st_nlink=1, st_uid=389, st_gid=389, st_blksize=4096, st_blocks=8, st_size=66, st_atime=2017/01/14-17:05:49, st_mtime=2017/01/14-17:05:44, st_ctime=2017/01/14-17:05:44}) = 0 <0.000033> 28812 17:21:32.556414 close(11) = 0 <0.000038> 28812 17:21:32.556541 open("/etc/dirsrv/slapd-EXAMPLE-COM/key3.db", O_RDONLY) = 11 <0.000049> 28812 17:21:32.556679 fstat(11, {st_dev=makedev(9, 2), st_ino=134188, st_mode=S_IFREG|0600, st_nlink=1, st_uid=389, st_gid=0, st_blksize=4096, st_blocks=24, st_size=16384, st_atime=2017/01/14-17:05:49, st_mtime=2017/01/14-17:05:41, st_ctime=2017/01/14-17:05:49}) = 0 <0.000032> 28812 17:21:32.556856 close(11) = 0 <0.000052> 28812 17:21:32.557003 open("/etc/dirsrv/slapd-EXAMPLE-COM/cert8.db", O_RDONLY) = 11 <0.000050> 28812 17:21:32.557145 fstat(11, {st_dev=makedev(9, 2), st_ino=134187, st_mode=S_IFREG|0600, st_nlink=1, st_uid=389, st_gid=0, st_blksize=4096, st_blocks=104, st_size=65536, st_atime=2017/01/14-17:05:49, st_mtime=2017/01/14-17:05:41, st_ctime=2017/01/14-17:05:49}) = 0 <0.000032> 28812 17:21:32.557322 close(11) = 0 <0.000036>
A little more debugging:
on the replica-to-be server there is no "Server-Cert" in the nssdb:
EXAMPLE.COM IPA CA CT,C,C
that would be a problem for ns-slapd to enable SSL, the installation seem incomplete, but I do not know why.
the installer on the replica uses certmonger to do cert operations, check on the master and replica ipa-getcert list and the status of the CA, is it running, and if the certs valid...
ipa-getcert list (replica)
Number of certificates and requests being tracked: 1. Request ID '20170114212250': status: CA_UNREACHABLE ca-error: Server at https://ipa.EXAMPLE.COM/ipa/xml failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Unable to communicate with CMS (503)). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM//pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: track: yes auto-renew: yes
ipa-getcert list (master)
Number of certificates and requests being tracked: 8. Request ID '20170107004809': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.EXAMPLE.COM,O=EXAMPLE.COM expires: 2019-01-08 00:48:09 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM track: yes auto-renew: yes Request ID '20170107005822': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.EXAMPLE.COM,O=EXAMPLE.COM expires: 2019-01-08 00:58:21 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
OK, after debugging the CA_UNREACHABLE error, httpd on the master was complaining that he could talk to 127.0.0.1:8009. After asking on IRC, this was DogTag (pki-tomcatd@pki-tomcat.service)
The server was listening to the port:
ss -tapn | grep 8009 LISTEN 0 100 ::1:8009 :::* users:(("java",pid=1775,fd=83))
Testing with telnet, reveals that it really only binds to IPv6 localhost (::1), but not IPv4 localhost.
I edited /etc/hosts to add an IPv6 localhost (only 127.0.0.1 was mapped), and now it can sucessfully get the Server-Cert certificate.
Redid ipa-replica-install and now it works fine! :) Thanks msauton for helping!
Thanks for providing updates until the solution was found. Is this a situation specific to a particular system (this replica only?), or is this a repeatable scenario? If there is no defect, we may want to close this ticket, otherwise we would need to figure out to reproduce the failed replica install/config. Thanks again for the perseverance!
Replying to [comment:14 msauton]:
I'm gonna try and replicate the situation, but it should be easily reproducible as long as there is no hosts entry with a "localhost ::1" mapping (default on CentoOS). CentOS's default has two mappings: 127.0.0.1 to localhost and ::1 to ip6-localhost which would trigger this behaviour.
I had done a little more debugging on the issue, because it was failing creating the CA replica now.
I changed /var/lib/pki/pki-tomcat/conf/server.xml to listen on 127.0.0.1 instead of ::1
# grep 8009 server.xml <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" />
It was previously (per default instalation) set to ::1 and after this change the replica install process ended successfully, with just this error:
ipa : ERROR unable to resolve host name ipa-replica.example.com. to IP address, ipa-ca DNS record will be incomplete
but that was solved by ipa dns-update-system-records.
So, I guess the problem is really on talking to the AJP server running on port 8009 while it's only bind is to port 8009 on IPv6, which was the case. Changing that to IPv4 localhost solved all the problems.
This is a duplicate of https://fedorahosted.org/freeipa/ticket/6575.
We're working on a fix. As mentioned in comment:16, changing ::1 to localhost or 127.0.0.1 in /var/lib/pki/pki-tomcat/conf/server.xml in 'address' field of AJP/1.3 Connector fixes the issue.
Metadata Update from @r3pek: - Issue assigned to someone - Issue set to the milestone: 0.0 NEEDS_TRIAGE
Login to comment on this ticket.