https://bugzilla.redhat.com/show_bug.cgi?id=845691 (Red Hat Enterprise Linux 6)
Description of problem: Occassionally ipa-client-install fails KRB pre-auth and rolls back like this: # ipa-client-install -U --domain=testrelm.com --realm=TESTRELM.COM -p admin -w PASSWORD --server=qe-blade-13.testrelm.com Failed to obtain host TGT. Discovery was successful! Hostname: qe-blade-04.testrelm.com Realm: TESTRELM.COM DNS Domain: testrelm.com IPA Server: qe-blade-13.testrelm.com BaseDN: dc=testrelm,dc=com Synchronizing time with KDC... Enrolled in IPA realm TESTRELM.COM Created /etc/ipa/default.conf New SSSD config will be created. Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm TESTRELM.COM Installation failed. Rolling back changes. This is primarily being seen when trying to install clients (in automated test) when the server topology is a branched tree. Topology: M / \ R1 R2 / /\ R3 R4-R5 C1 - C6 are balances across M, R1 - R5 where: C1 -> M C2 -> R1 ... C6 -> R5 In this particular case, it was C4 -> R3 but, I've also seen C1 -> M fail the same way. Version-Release number of selected component (if applicable): RHEL6.3 server ipa-client-2.2.0-16.el6.x86_64 certmonger-0.56-1.el6.x86_64 krb5-workstation-1.9-33.el6.x86_64 How reproducible: Unknown but is occurring noticably on automated tests. Steps to Reproduce: 1. <May need to build a similar server topology> 2. ipa-client-install -U --domain=$DOMAIN --realm=$RELM -p admin -w $ADMINPW --server=$SERVER Actual results: error as above. Expected results: no error and client now joined to IPA domain. Additional info: /var/log/ipaclient-install.log: ... 2012-08-03T15:45:59Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/qe-blade-04.testrelm.com@TESTRELM.COM 2012-08-03T15:45:59Z DEBUG stdout= 2012-08-03T15:45:59Z DEBUG stderr=kinit: Preauthentication failed while getting initial credentials 2012-08-03T15:45:59Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -L -d /etc/pki/nssdb -n IPA CA 2012-08-03T15:45:59Z DEBUG stdout=Certificate: ...cert info here....let me know if needed. 2012-08-03T15:45:59Z DEBUG stderr= 2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -D -d /etc/pki/nssdb -n IPA CA 2012-08-03T15:45:59Z DEBUG stdout= 2012-08-03T15:45:59Z DEBUG stderr= 2012-08-03T15:45:59Z DEBUG args=/sbin/service messagebus start 2012-08-03T15:45:59Z DEBUG stdout=Starting system message bus: 2012-08-03T15:45:59Z DEBUG stderr= 2012-08-03T15:45:59Z DEBUG args=/sbin/service certmonger start 2012-08-03T15:45:59Z DEBUG stdout= 2012-08-03T15:45:59Z DEBUG stderr= 2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -L -d /etc/pki/nssdb -n IPA Machine Certificate - qe-blade-04.testrelm.com 2012-08-03T15:45:59Z DEBUG stdout= 2012-08-03T15:45:59Z DEBUG stderr=certutil: Could not find cert: IPA Machine Certificate - qe-blade-04.testrelm.com : File not found ... Now it shows log entries for undoing config. IPA SERVER httpd/access_log: [root@qe-blade-13 log]# grep <IPADDRESS> httpd/access_log |cut -f2- -d' ' - - [03/Aug/2012:11:45:54 -0400] "GET /ipa/config/ca.crt HTTP/1.0" 200 1321 "-" "Wget/1.12 (linux-gnu)" - - [03/Aug/2012:11:45:54 -0400] "GET /ipa/config/ca.crt HTTP/1.0" 200 1321 "-" "Wget/1.12 (linux-gnu)" - - [03/Aug/2012:11:45:55 -0400] "POST /ipa/xml HTTP/1.1" 401 1856 - admin@TESTRELM.COM [03/Aug/2012:11:45:55 -0400] "POST /ipa/xml HTTP/1.1" 200 1925 - - [03/Aug/2012:11:46:01 -0400] "POST /ipa/xml HTTP/1.1" 401 1856 - host/qe-blade-04.testrelm.com@TESTRELM.COM [03/Aug/2012:11:46:01 -0400] "POST /ipa/xml HTTP/1.1" 200 427 IPA SERVER httpd/error_log: [Fri Aug 03 11:45:56 2012] [error] ipa: INFO: admin@TESTRELM.COM: join(u'qe-blade-04.testrelm.com', nshardwareplatform=u'x86_64', nsosversion=u'2.6.32-279.el6.x86_64'): SUCCESS [Fri Aug 03 11:46:02 2012] [error] ipa: INFO: host/qe-blade-04.testrelm.com@TESTRELM.COM: host_disable(u'qe-blade-04.testrelm.com'): SUCCESS
Can you take a network trace while this happens ? It would allow me to rule in/out an hypothesis I have about why this may be happening.
The cause seems to be that we don't always contact the same KDC, because it's looked up in DNS several times.
From Simo's mail:
We should probably use a fixed overriding krb5.conf during setup that disables KJDC and Realm DNS resolution and has only the specific server we are working against in the [realm] section. Once the machine account is setup and sssd has the locator plugin working then we can revert top the normal krb5.conf file.
We should probably use a fixed overriding krb5.conf during setup that disables KJDC and Realm DNS resolution and has only the specific server we are working against in the [realm] section.
Once the machine account is setup and sssd has the locator plugin working then we can revert top the normal krb5.conf file.
It's due to the client using DNS to find a KDC that hasn't yet replicated info about the new client, and then asking it for a TGT. How to reproduce:
- Install a "master" and "replica" - Change the Kerberos DNS entries to only point to the replica: for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 $REPLICA_HOSTNAME" done ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 $MASTER_HOSTNAME" ipa dnsrecord-find $DOMAIN # check - Sever communication between the hosts, disabling replication: on master: # iptables -A INPUT -j DROP -p all --source $REPLICA_IP - On client machine, put master as nameserver in /etc/resolv.conf & install client
master: 1f83139
ipa-3-0: 25ddbfc
Additional change to fix new installs:
master: c87ac6b
ipa-3-0: a649d27
Additional fix to make ipa-client-install use private Kerberos CCACHE:
master: 79b90d1[[BR]] ipa-3-0: 29a5d16
Metadata Update from @dpal: - Issue assigned to pviktori - Issue set to the milestone: FreeIPA 3.0 RC1
Login to comment on this ticket.