Traceback from one of the rhel6 app servers:

{{{

[cherrypy.msg] INFO 2012-03-01 22:34:03,766 HTTP: Page handler: <bound method Root.save of <bodhi.co ntrollers.Root object at 0xafc9d3cc>>
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/CherryPy-2.3.0-py2.6.egg/cherrypy/_cphttptools.py", line 12
1, in _run
self.main()
File "/usr/lib/python2.6/site-packages/CherryPy-2.3.0-py2.6.egg/cherrypy/_cphttptools.py", line 26
4, in main
body = page_handler(virtual_path, self.params)
File "<string>", line 3, in save
File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 361, in expose
*args, kw)
File "<generated code>", line 0, in run_with_transaction
File "/usr/lib/python2.6/site-packages/peak/rules/core.py", line 153, in call
return self.body(args, kw)
File "/usr/lib/python2.6/site-packages/turbogears/database.py", line 410, in so_rwt
retval = func(*args, kw)
File "<generated code>", line 0, in _expose
File "/usr/lib/python2.6/site-packages/peak/rules/core.py", line 153, in call
return self.body(args, kw)
File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 390, in <lambda>
fragment, options, args, kw)))
File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 425, in _execute_func
output = errorhandling.try_call(func, *args, kw)
File "/usr/lib/python2.6/site-packages/turbogears/errorhandling.py", line 77, in try_call
return func(self, args, kw)
File "<string>", line 2, in save
File "/usr/lib/python2.6/site-packages/bodhi/util.py", line 494, in json_redirect
return f(*args, kw)
File "<string>", line 3, in save
File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 207, in validate
return errorhandling.run_with_errors(errors, func, args, kw)
File "/usr/lib/python2.6/site-packages/turbogears/errorhandling.py", line 118, in run_with_errors
return func(self, *args, kw)
File "<string>", line 3, in save
File "/usr/lib/python2.6/site-packages/turbogears/identity/conditions.py", line 249, in require
return fn(self, args, *kwargs)
File "/usr/lib/python2.6/site-packages/bodhi/controllers.py", line 605, in save
koji = buildsys.get_session()
File "/usr/lib/python2.6/site-packages/bodhi/buildsys.py", line 218, in get_session
session = koji_login()
File "/usr/lib/python2.6/site-packages/bodhi/buildsys.py", line 210, in koji_login
koji_session.ssl_login(client, clientca, serverca)
File "/usr/lib/python2.6/site-packages/koji/init.py", line 1670, in ssl_login
sinfo = self.callMethod('sslLogin', proxyuser)
File "/usr/lib/python2.6/site-packages/koji/init.py", line 1715, in callMethod
return self._callMethod(name, args, opts)
File "/usr/lib/python2.6/site-packages/koji/init.py", line 1740, in _callMethod
return proxy.getattr(name)(args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in call
return self.__send(self.__name, args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib/python2.6/xmlrpclib.py", line 1235, in request
self.send_content(h, request_body)
File "/usr/lib/python2.6/xmlrpclib.py", line 1349, in send_content
connection.endheaders()
File "/usr/lib/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib/python2.6/site-packages/koji/ssl/SSLCommon.py", line 108, in connect
self.sock.connect((self.host, self.port))
File "<string>", line 1, in connect
error: [Errno 110] Connection timed out
}}}

This seems to only be happening on the rhel6 app servers, so it's likely some behavior change from rhel5.

FWIW, I'm hitting this issue when trying to (un)push F15/16 and EL5/6 packages.

ok, it turns out it was just app01.

I found a incorrect route on it that may have been causing all this.

Can everyone test now and see if they can see the issue?

I spoke too soon. Something is still messed up route wise.
Hold any testing for a few.

ok, NOW it should be fixed.

I'm going to close this now, I have seen no more tracebacks and I think everything is back to normal.

Please reopen if you see any further issues.