#2686 Gracefully handle FAS outages in our TurboGears apps
Closed: Fixed None Opened 12 years ago by lmacken.

Right now when FAS has an outage, our other TG apps do not gracefully fail, and merely return 500 errors. This is due to the complexity of the TurboGears identity/visit manager, and not being able to simply catch the error and redirect.

I think the best solution right now for our TG1 apps is to write a 'catch all' cherrypy filter, like this:

http://docs.turbogears.org/1.0/ErrorReporting#method-3-application-wide-catch-all-with-cherrypy-filter

Our filter should at least do the following:

  • Catch exceptions from our apps
    • Determine if it is an outage (ServerError), or an app error (Exception)
  • Email the maintainers the traceback
  • Redirect to a nice error/outage page

Here is a typical traceback that occurs when our identity layer has trouble contact the account system:

{{{
[turbogears.access] INFO 2011-06-19 04:02:33,451 - - "GET /updates/nagios HTTP/1.0" 500 791 "" ""
[cherrypy.msg] INFO 2011-06-19 04:02:33,985 HTTP: Page handler: 'transfer closed with 119 bytes remaining to read'
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 119, in _run
applyFilters('before_main')
File "/usr/lib/python2.4/site-packages/cherrypy/filters/init.py", line 151, in applyFilters
method()
File "/usr/lib/python2.4/site-packages/turbogears/visit/api.py", line 244, in before_main
plugin.record_request(visit)
File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 181, in record_request
identity = self.identity_from_request(visit.key)
File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 92, in identity_from_request
identity = source(visit_key)
File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 139, in identity_from_visit
return self.provider.load_identity(visit_key)
File "/usr/lib/python2.4/site-packages/fedora/tg/identity/jsonfasprovider2.py", line 433, in load_identity
ident = JsonFasIdentity(visit_key)
File "/usr/lib/python2.4/site-packages/fedora/tg/identity/jsonfasprovider2.py", line 107, in init
self.login(using_ssl)
File "/usr/lib/python2.4/site-packages/fedora/tg/identity/jsonfasprovider2.py", line 322, in login
self.send_request('', auth=True)
File "/usr/lib/python2.4/site-packages/fedora/tg/identity/jsonfasprovider2.py", line 125, in send_request
req_params=req_params, auth=auth, retries=3)
File "/usr/lib/python2.4/site-packages/fedora/client/baseclient.py", line 344, in send_request
auth_params=auth_params, retries=retries)
File "/usr/lib/python2.4/site-packages/fedora/client/proxyclient.py", line 380, in send_request
request.perform()
error: 18
}}}

More basic information on this since I was approached by someone who was thinking about working on it but they're working on something else instead. May help someone new to Fedora if they want to take up this particular ticket.

Right now we authenticate against the username and password stored in fas. (https://admin.fedoraproject.org/accounts/ )
fas has a way to query it via http and it returns data in json format. so, when someone logs into another web app like the packagedb (https://admin.fedoraproject.org/pkgdb/ ) we query fas to actually log them in. If FAS Is down this won't work but that's to be expected.

The real problem is in how fas maintains a session for a user. fas has a 20 minute timeout value for a session. To keep it active, you need to hit urls in fas. So our fas auth for the other apps hits fas with every request that you make to them. ie: I browse to http://admin.fedoraproject.org/pkgdb/ ; pkgdb forwards a request to fas ; fas records that I looked at a page to keep my session active.

Since TurboGear1 uses this system for both visit tracking and identity, this request is forwarded even if you are not authenticated. The same requests happen (just with a session not associated with a user) even if I'm not logged in. So. If fas goes down, all requests to the other apps start to fail because they try to hit a fas url and fail.

There's several thing that need to change to fully solve this:

  • make it so that unauthenticated requests do not talk to fas.
  • if we're authenticated and on one of the other apps (like pkgdb), either allow the user to stay authenticated and catch the exception from fas or display an error and de-auth them.

Hey look, no more turbogears apps. ;)

Login to comment on this ticket.

Metadata