#2647 pkgs01 remounts /srv/cache/lookaside read-only, preventing uploads.
Closed: Fixed None Opened 13 years ago by eseyman.

= phenomenon =
fedpkg does not allow me to upload the source tarball for perl-Math-Random-ISAAC yet replies that it did. Nils Philippsen confirmed this on #fedora-devel.
= reason =
Probably not a very good one.


Disk error; volume is currently marked read-only.

Things should be back to normal now.

Leaving this open to note root cause and how to prevent it moving forward. :)

This just happened again.

Errors like:

sd 0:0:0:0: timing out command, waited 180s
sd 0:0:0:0: [sdb] Unhandled error code
sd 0:0:0:0: [sdb] Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK
sd 0:0:0:0: [sdb] CDB: Read(10): 28 00 2b c1 15 18 00 00 08 00
end_request: I/O error, dev sdb, sector 734074136
...
Aborting journal on device dm-1-8.
connection1:0: detected conn error (1020)
EXT4-fs error (device dm-1): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (dm-1): Remounting filesystem read-only

in dmesg.

Mount is a iscsi from a equilogic san.

It would be good if the upload CGI script detected this and returned a suitable HTTP error status so that curl/fedpkg would know that the upload hadn't actually succeeded.

There was another 'event' tonight. This time with some xen traces...

we have rebooted the instance into a newer kernel and had it do a full fsck on the lookaside filesystem.

Cause investigation is still ongoing...

Looks like it's happened again; my upload of Perl-Critic-1.115.tar.gz is reported to have worked, but it didn't.

sadly so. Fixed again. Please re-upload.

ok. We have moved pkgs01 to a new kvm rhel6 host.
Hopefully this will fix this from happening again.

I am going to go ahead and close this ticket now, feel free to re-open if this issue ever re-occurs.

Login to comment on this ticket.

Metadata