Ticket #2571 (closed outage: fixed)

Opened 3 years ago

Last modified 3 years ago

PHX2 Network outage - 2011-01-25 02:00 UTC

Reported by: smooge Owned by: smooge
Priority: blocker Milestone: Fedora 15
Component: Systems Version: Production
Severity: The Sky Is Falling Keywords:
Cc: Blocked By:
Blocking: Sensitive:

Description (last modified by smooge) (diff)

phenomenon

We are moving from older netapp to newer one due to hardware limitations and aging hardware. Due to change in disk architectures and such, we will be having a total outage of Fedora services.

reason

recommendation

Outage: PHX2 Network outage - 2011-01-25 02:00 UTC

There will be an outage starting at 2011-01-25 02:00 UTC, which will last approximately 3 hours.

To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:

date -d '2011-01-25 02:00 UTC'

Reason for outage:

Moving from

Affected Services:

BFO - http://boot.fedoraproject.org/
Bodhi - https://admin.fedoraproject.org/updates/
Buildsystem - http://koji.fedoraproject.org/
CVS / Source Control
DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
Docs - http://docs.fedoraproject.org/
Email system
Fedora Account System - https://admin.fedoraproject.org/accounts/
Fedora Community - https://admin.fedoraproject.org/community/
Fedora Hosted - https://fedorahosted.org/
Fedora People - http://fedorapeople.org/
Fedora Talk - http://talk.fedoraproject.org/
Main Website - http://fedoraproject.org/
Mirror List - https://mirrors.fedoraproject.org/
Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
Package Database - https://admin.fedoraproject.org/pkgdb/
Smolt - http://smolts.org/
Spins - http://spins.fedoraproject.org/
Start - http://start.fedoraproject.org/
Torrent - http://torrent.fedoraproject.org/
Translation Services - http://translate.fedoraproject.org/
Wiki - http://fedoraproject.org/wiki/

Unaffected Services:

NONE

Contact Information:

Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage.

Change History

comment:1 Changed 3 years ago by smooge

  • Description modified (diff)

comment:2 Changed 3 years ago by smooge

  • Status changed from new to assigned

comment:3 Changed 3 years ago by smooge

DNS will not be affected.

BFO may be affected partially.

comment:4 Changed 3 years ago by smooge

  • Description modified (diff)

The outage will be for 3 hours. fixing in ticket

comment:5 Changed 3 years ago by smooge

Systems were updated to 5.6 but only ones using nfs/iscsi were rebooted

Found out that nfsv4 on systems was touch and go. Some systems were able to mount without problems and others were getting idmapd issues that could not be resolved. Decided to deal with those later as nfsv3 worked fine.

Most systems were able to mount the new iscsi without issues however the build servers had problems with not keeping a persistant cache. This caused them not to be able to bring up systems after reboot. Need to rearchitect and figure out.

comment:6 Changed 3 years ago by ricky

Some remaining issues:

weird denial of obsolete secondary01 mounts on puppet01
nfs4 issues
xen07, bxen03, bxen04 requiring manual vgchange -ay

comment:7 Changed 3 years ago by smooge

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.