Buildsystem is hosed right now:
418533 build (dist-f9, devel:mboxgrep-0_7_9-5_fc9): open (xenbuilder4.fedora.phx.redhat.com) -> FAILED: Fault: <Fault 1: "exceptions.OSError: [Errno 31] Too many links: '/mnt/koji/work/tasks/418535'">
Furthermore please clean up http://koji.fedoraproject.org/koji/buildinfo?buildID=35211 http://koji.fedoraproject.org/koji/buildinfo?buildID=35547 and http://koji.fedoraproject.org/koji/buildinfo?buildID=35554. They've been running for more than a day.
Additional info it may or may not help.
http://koji.fedoraproject.org/koji/taskinfo?taskID=418705
Okay, I've done something which should provide a short-term band-aid to get builds working again.
Basically what's happening is that we're overflowing the maximum files (32k) in a directory for /mnt/koji/work/tasks. This directory was in the past being gc'd I think, but not sure where from. For right now I created ../tasks.old and moved ~1200 of the oldest directories there to get us some breathing room. But we should figure out why the garbage collection isn't happening since the move to the new storage
the old storage was on a netapp. new storage is a ext3 filesystem nfs exported. We need to be and are now more agressive in cleaning up.
Login to comment on this ticket.