I just pushed out a new version of bodhi into production with the dumb fix of appending the trailing slash on the update link. This is just a band-aid as to not prevent anyone from submitting updates, but we really need to figure out why this is happening.

At a quick glance, this problem might be happening in the CSRF/identity layer. The links on the page are fine, so the tg.url/fedora.tg.util.csrf_url don't seem to be the culprit. On the way to CherryPy from the request, something seems to be prepending an additional /updates/ to the URL, thus triggering the 404.

Toshio, do you think this change could be causing this behavior?

{{{
commit 877c758d861cd0cd7b5bd7090f50427c36ad2764
Author: Toshio くらとみ toshio@puppet1.fedora.phx.redhat.com
Date: Fri Sep 18 22:42:12 2009 +0000

Turn off base_url_filter.use_x_forwarded_host so these will work in staging.

diff --git a/modules/bodhi/templates/bodhi-epel-masher.cfg.erb b/modules/bodhi/t
index 4c23710..b39c115 100644
--- a/modules/bodhi/templates/bodhi-epel-masher.cfg.erb
+++ b/modules/bodhi/templates/bodhi-epel-masher.cfg.erb
@@ -26,7 +26,7 @@ server.socket_queue_size = 30

session_filter.on = False
base_url_filter.on = True
-base_url_filter.use_x_forwarded_host = True
+base_url_filter.use_x_forwarded_host = False
base_url_filter.base_url = 'https://admin.fedoraproject.org/updates'

tg.strict_parameters = True

}}}

What's weird is that it doesn't always happen.

Could it be that Bodhi runs on several servers and that not all are rewriting the URL in the same way? (different configurations?)

Is there a redirect involved in this?

Just checked rpm -q bodhi-server , rpm -V bodhi-server, md5sum of /etc/bodhi/bodhi.cfg on app1-7. All of the versions and checksums match. So the randomness doesn't seem to be because one server is out of sync.

bocheca was able to reproduce with the base_url_filter config changes. We still don't know why it sometimes works.

Here's hte problematic code: python-cherrypy-2.x:
cherrypy/_cphttptools.py

in def mapPathToObject(self, objectpath):

{{{
# Try the current leaf.
if callable(candidate) and getattr(candidate, 'exposed', False):
if i == len(objectTrail) - 1:
# We found the extra ".index". Check if the original path
# had a trailing slash (otherwise, do a redirect).
self.is_index = True
if not objectpath.endswith('/'):
atoms = self.browser_url.split("?", 1)
newUrl = atoms.pop(0) + '/'
if atoms:
newUrl += "?" + atoms[0]
raise cherrypy.HTTPRedirect(newUrl)
}}}

Looking at this, one thing that sticks out is that we are examining the path that is passed into this function (which happens to be '/new') but when we go to reconstruct it, we go back to self.browser_url. This is what's causing us issues but what precisel it should do needs a more awake toshio to figure out.

bochecha suggested removing the path from the baseurl in the config. That needs to be approached carefully as it may break other code. However, this particular piece of code should combine baseurl with server.webpath (earlier, in the TG code that calls it). So it should fix this. I've updated the configs for staging. If we test this and it works we'll want to cherrypick 3dce2478caa48e9bbf1d94630235b732e003f98a into production.

Note that app2.stg is currently locked for puppet and I don't know who is working on that. So I updated pkgdb's config manually on app2.stg. The pkgdb no longer suffers from this issue there. We need to update the other configs/run puppet on app2.stg and then test pretty heavily. A grep of source code for in-app use of base_url_filter.base_url would probably also be good.

app2.stg has been updated. Test, test test. (bodhi looks like it's correct as does pkgdb.) elections, mirrormanager, fas, need to be checked over. I'll grep source for those today.

I wonder if we're not back to the situation you described in your email last friday.

If I try to open http://admin.stg.fedoraproject.org/updates , I'm redirected to https://admin.fedoraproject.org/updates , whereas it should be https://admin.stg.fedoraproject.org/updates , right ?

That's always going to do that due to the way the staging environment is setup. What should not be happening is that a link inside one of the web apps takes you to the http:// version of the page (and hence, you get redirected to admin.fp.o instead of staying on admin.stg.fp.o)

grepped source for mirrormanager, fas, bodhi, python-fedora, and pkgdb for occurrences of base_url. One fix needed for pkgdb: https://fedorahosted.org/fedora-infrastructure/ticket/1683

Still having a strange issue with fas/python-fedora -- If you log into https://admin.stg.fedoraproject.org/accounts then remove the CSRF token (ie: go back to https://admin.stg.feoraproject.org/accounts/ ) You get the button to verify your login. When you click on that you end up at admin.fp.o instead of admin.stg.fp.o. Trying to track that down.

Okay, this wasa configuration error as well -- this time because we have configuration information showing in the wsgi files. I've fixed this in puppet and deployed to staging. Please test the apps in the staging env to make sure things work.

Especially important is that you do not get redirected to admin.fedoraproject.org as that is a sign that the problem the original config change was made for is not fixed with this setup. If we don't find anything we can push this live tomorrow.

Lightly tested known problem cases in fas, pkgdb, mirrormanager. Checked that stg.smolt.org seems to work. I don't know how much of bodhi is expected to work since it can't talk to the production koji. Le me know tomorrow if you're satisfied and we'll push this out.

After doing some basic testing of bodhi in staging, it seems that everything is working fine.

Pushed to production.