#2379 2010 September Updates and Reboots
Closed: Fixed None Opened 13 years ago by smooge.

There will be an outage starting at 2010-09-10 00:00:00 UTC, which will last approximately 2 hours.

To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:

date -d '2010-09-10 00:00:00 UTC'

Affected Services: Buildsystem CVS / Source Control Database DNS Fedora Hosted Fedora People Fedora Talk Mail Mirror System Torrent Websites

Unaffected Services:

None

Reason for outage:

Every month we get new updates and many months we get new kernels.. this is one of those months. We will be updating all hosts and rebooting them on Thu 2010-09-10 starting at 00:00. Outages will be rolling so few services will be down for longer than 15 minutes at a time.

Contact Information:

Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.


Lessons learned:

Updates went much better this time than previous releases. Hosts with broken update process are limited to bapp01/app01.stg and some publictest boxes. func-yum did the rest

Rebooting this time crashed horribly:

  1. It was started an hour early because of a mistyped date command by lead rebooted ( Stephen Smoogen)
  2. Nagios changes did not take effect as planned. This caused a storm of pager tickets that didn't need to occur.
  3. DNS changes were not made for proxy servers in correct order. This was fixed later by Mike Mcgrath
  4. Several servers did not come up correctly due to non-existant or broken symlinks in /etc/auto.
  5. Fas servers (RHEL5.90.) did not reboot correctly. This seems to be caused by an oops in the kernel dealing with swap.

Login to comment on this ticket.

Metadata