https://bugzilla.redhat.com/show_bug.cgi?id=845405 (Red Hat Enterprise Linux 6)
Description of problem: During ipa-replica-install, sometimes httpd restart fails: Configuring the web interface: Estimated time 1 minute [1/12]: disabling mod_ssl in httpd [2/12]: setting mod_nss port to 443 [3/12]: setting mod_nss password file [4/12]: enabling mod_nss renegotiate [5/12]: adding URL rewriting rules [6/12]: configuring httpd [7/12]: setting up ssl [8/12]: publish CA cert [9/12]: creating a keytab for httpd [10/12]: configuring SELinux for httpd [11/12]: restarting httpd [12/12]: configuring httpd to start on boot done configuring httpd. creation of replica failed: Command '/sbin/service httpd restart ' returned non-zero exit status 1 I've seen this more than once now but, not yet able to reproduce. Version-Release number of selected component (if applicable): ipa-server-2.1.3-9.el6.i686 httpd-2.2.15-15.el6.i686 How reproducible: Unknown Steps to Reproduce: 1. <Set up RHEL6.2 IPA Master server> 2. On Master: ipa-replica-prepare -p $ADMINPW --ip-address=$SLAVE_IP $SLAVE_S.$DOMAIN 3. On Replica: sftp $MASTER:/var/lib/ipa/replica-info-$SLAVEFQDN.gpg /dev/shm 4. On Replica: ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$SLAVEFQDN.gpg Actual results: Error from description above. Expected results: No error. Additional info: From ipareplica-install.log: 2012-08-01 15:19:40,811 DEBUG stdout=pk12util: PKCS12 IMPORT SUCCESSFUL 2012-08-01 15:19:40,812 DEBUG stderr= 2012-08-01 15:19:52,763 DEBUG args=/sbin/service httpd restart 2012-08-01 15:19:52,770 DEBUG stdout=Stopping httpd: [ OK ] Starting httpd: [FAILED] 2012-08-01 15:19:52,770 DEBUG stderr=(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs 2012-08-01 15:19:52,799 DEBUG Command '/sbin/service httpd restart ' returned non-zero exit status 1 File "/usr/sbin/ipa-replica-install", line 482, in <module> main() File "/usr/sbin/ipa-replica-install", line 447, in main ipaservices.knownservices.httpd.restart() File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py", line 47, in restart ipautil.run(["/sbin/service", self.service_name, "restart", instance_name], capture_output=capture_output) File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 273, in run raise CalledProcessError(p.returncode, args) From http/error_log: [root@ipaqavmb httpd]# cat error_log [Wed Aug 01 15:19:38 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0 [Wed Aug 01 15:19:38 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Wed Aug 01 15:19:40 2012] [notice] Digest: generating secret for digest authentication ... [Wed Aug 01 15:19:40 2012] [notice] Digest: done [Wed Aug 01 15:19:40 2012] [warn] mod_wsgi: Compiled for Python/2.6.2. [Wed Aug 01 15:19:40 2012] [warn] mod_wsgi: Runtime using Python/2.6.6. [Wed Aug 01 15:19:41 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_auth_kerb/5.4 mod_nss/2.2.15 NSS/3.12.9.0 mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations [Wed Aug 01 15:19:42 2012] [error] Exception KeyError: KeyError(-1216170160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored [Wed Aug 01 15:19:43 2012] [error] Exception KeyError: KeyError(-1216170160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored [Wed Aug 01 15:19:45 2012] [error] ipa: INFO: *** PROCESS START *** [Wed Aug 01 15:19:45 2012] [error] ipa: INFO: *** PROCESS START *** PS showed httpd processes still running like they were orphaned when service httpd restart ran the stop: [root@ipaqavmb ~]# ps -ef|grep http root 25610 1 0 16:35 ? 00:00:00 /usr/sbin/nss_pcache 983043 off /etc/httpd/alias apache 25637 1 8 16:35 ? 00:00:02 /usr/sbin/httpd apache 25639 1 7 16:35 ? 00:00:02 /usr/sbin/httpd apache 25640 1 1 16:35 ? 00:00:00 /usr/sbin/httpd apache 25642 1 1 16:35 ? 00:00:00 /usr/sbin/httpd apache 25644 1 1 16:35 ? 00:00:00 /usr/sbin/httpd apache 25645 1 1 16:35 ? 00:00:00 /usr/sbin/httpd apache 25646 1 1 16:35 ? 00:00:00 /usr/sbin/httpd apache 25647 1 1 16:35 ? 00:00:00 /usr/sbin/httpd root 25936 22439 1 16:35 pts/6 00:00:00 grep http So, it looks like a problem with the httpd stop from the init script?
We will probably end up filing a bug against httpd or systemd for this, but we may also want to consider using stop/start instead of restart.
Bug 867467 has been closed as NEXTRELEASE. You can check Joe Orton's response with reasoning there. Bottomline is that with sysV scripts cannot guarantee that httpd will always restart successfully when restarted in such a short notice. This should be fixed with systemd.
As for sysV platforms, this will instead lead to a patch to enforce a wait+fallback in case httpd restart fails.
attachment freeipa-mkosek-324-add-fallback-for-httpd-restarts.patch
Patch freeipa-mkosek-324-add-fallback-for-httpd-restarts.patch sent for review
master: 9126b18
ipa-3-0: 5f95fbf
Metadata Update from @dpal: - Issue assigned to mkosek - Issue set to the milestone: FreeIPA 3.0.1 (bug fixing)
Login to comment on this ticket.