Ticket #2753 (closed outage: fixed)

Opened 3 years ago

Last modified 3 years ago

Outage: Fedorapeople.org outage 2011-05-04

Reported by: kevin Owned by: kevin
Priority: major Milestone: Fedora 15
Component: General Version:
Severity: Normal Keywords:
Cc: lmacken, jnovy Blocked By:
Blocking: Sensitive:

Description

fedorapeople.org and the xen host it's on are not reachable via the network.

Outage started around 21:50UTC.

A ticket has been filed with bu support to investigate.

Change History

comment:1 follow-up: ↓ 4 Changed 3 years ago by kevin

Note that fedpkg is affected as it's trying to check jkeating.fedorapeople.org to see if there are new branches setup. ;(

comment:2 Changed 3 years ago by kevin

This issue has been assigned an ID of: #86098 on the enghelp@… ticketing system. So far there's no ETA or information on cause of the outage. ;(

We do have a new instance that was being prepped to replace this one, and if the outage continues we could restore a backup to the new instance.

comment:3 Changed 3 years ago by kevin

Additional information indicates there is a massive power outage on site. ;(

Hopefully more information on when power and network may be restored will be coming in the morning.

comment:4 in reply to: ↑ 1 Changed 3 years ago by nphilipp

Replying to kevin:

Note that fedpkg is affected as it's trying to check jkeating.fedorapeople.org to see if there are new branches setup. ;(

I've let my local webserver serve an empty /reposconverted file and added an entry with its IP to /etc/hosts for jkeating.fedorapeople.org to work around this.

comment:5 Changed 3 years ago by kevin

The site has restored power now, but the xen host that fedorapeople.org is running on is not coming up correctly yet. On site folks are working on it.

comment:6 Changed 3 years ago by lmacken

  • Cc lmacken added

comment:7 Changed 3 years ago by kevin

We have moved jkeating.fedorapeople.org over to allow fedpkg to work. As soon as you have refreshed DNS information this should start working normally now.

We are also syncing over a backup of the data to a new instance. This will allow us to bring up an instance thats available, but a day out of date.

comment:8 Changed 3 years ago by kevin

We are still syncing backup data to the new instance.

There's no further news on the fate of the old machine at this point, so we are moving forward with the idea that we will bring up the new one as soon as the restore is done, sadly losing a day of changes.

Hopefully we can bring up the new instance tomorrow once data has finished syncing to it.

comment:9 Changed 3 years ago by jnovy

  • Cc jnovy added

This affects TeX Live 2010 and 2011 repositories, making their users quite uncomfortable:

http://www.linux.cz/pipermail/texlive/2011-May/000141.html

comment:10 Changed 3 years ago by kevin

  • Status changed from new to closed
  • Resolution set to fixed

fedorapeople is back up and operating now finally as of 09:20UTC or so.

We are going to look at migrating it to the new instance soon, but for now, the old one is back up.

Note: See TracTickets for help on using tickets.