mmcgrath and warthog9 looked at this but couldn't pin it down.

It is a problem with caching so gitweb-caching suffers from it but vanilla gitweb does not. Reproducing has been an issue in diagnosing further.

We looked further and thought that this was a problem with Apache RewriteRules in puppet/configs/web/git.fedorahosted.org.conf. However, when attempting to reproduce on hosted02 I found some discrepancies between what we have setup on pkgs01 and hosted01 and hosted02. I ended up upgrading gitweb-caching on both hosted boxes and removing a perl module that wasn't in rpm or puppet (and not needed after the gitweb update).

At this point in time I cannot reproduce the problem.

What I'd like is for us to monitor this problem and if it comes back, see if we can reproduce it/reproduce it after a restart. If so we'll look at the RewriteRules at that time. If not, we'll also look at the RewriteRules but it'll be less easy to tell if we've actually fixed the issue.

https://fedorahosted.org/fedora-infrastructure/ticket/2600

This behavior is sporadic? Because I am getting the correct response now.

When the freeze is over, we need to give a careful look to our gitweb/gitweb-caching setup. It's possible that we're hitting stock gitweb unintentionally at times, which might explain the random issues. We may have gitweb-caching mis-configured as well.

In case I drop out of sight, here are two patches against puppet that I worked on when I looked at this a few weeks ago:

http://tmz.fedorapeople.org/patches/0001-gitweb-caching-Configure-httpd-to-use-git.patch
http://tmz.fedorapeople.org/patches/0002-gitweb-caching-Use-etc-gitweb.conf-for-modified-conf.patch

We also need to fix the sobby module to not require the stock gitweb package. Not having stock gitweb installed should make it much simpler to see if and when we're not using gitweb-caching as we intend.

Fixed this by dropping gitweb-caching and switching to cgit.