= problem =
We are using varnish to cache only a very small number of our web apps. We'd get better response times if we cached more.
= analysis =
Our setup uses apache, varnish, haproxy, and a farm of application servers running web apps. For the wiki, it looks like this:
request => apache => varnish => haproxy => app server (response goes back up the chain).
In this setup, varnish does our caching.
We have other web applications besides the wiki. Most of those are not using varnish. So their setup looks like this:
request => apache => haproxy => app server (response goes back up the chain).
= enhancement recommendation =
We'd like to start using varnish with these so that we can cache more of our resources. To make this happen, we need to modify apache to send requests to varnish and (if not already written) create varnish config to get the data from haproxy when needed. We then need to test this on the app servers for several potential problems:
As an optimization, we want to test for the following as well:
This should be deployed and tested in the staging environment one app at a time and then merged to production when it seems to work.
Hi, I like to work on this one. Please give me some instruction, such as how to access these machines. Thank you.
Are there any changes? Can I help with this issue?
Replying to [comment:3 kubo]:
I have not made any change. Please go ahead, and you can change the owner if you want
thanks,
Edit: Nevermind the last comment. I was looking only into varnish and haproxy confs when I should be also (and mainly) looking into apache confs...
original varnish config file default.vcl
the patch file for varnish config file default.vcl.patch
I added a patch of default.vcl. I am not sure if tg-visit cookie should be cached. Instead of fpo-mediawiki_en_session, fpo-mediawiki_en_UserID and fpo-mediawiki_en_UserName, only fpo-mediawiki_en_UserID is cached. Put static files(such as js,css,png) into cache, too. Changes are mostly applied to wiki and pkgdb app, as the most of the traffic is directed to these two sites.
Moving all currently open easyfix tickets to the HANDYWAVY-FUTURE milestone.
I'm clearing the assigned status on all easyfix tickets.
If you are an apprentice actively working on this ticket, feel free to reassign to yourself. Otherwise let a new apprentice have a look.
Hi, I would like to take this bug.
I'll take this one on. I was looking for a project with Varnish to tackle. Great coincidence.
Hello, can I have access to the machines so I can look around and maybe help with varnish.
Ansible httpd patch for https://apps.fedoraproject.org/nuancier/ httpd___proxies-reverseproxy.yml.patch
Ansible varnish patch for https://apps.fedoraproject.org/nuancier/ varnish___proxy.vcl.j2.patch
I started with something smaller. Here are a patches for caching https://apps.fedoraproject.org/nuancier/static/* and https://apps.fedoraproject.org/nuancier/cache/*.
For the nuancier patch, the sub-urls wouldn't work. You are matching against: if (req.url ~ "^/static/") { .
The problem is that this would match against https://apps.fp.o/static/, since ^ means "start of string here". What you likely want is "^/nuancier/static/" (and same with /cache/).
Ansible varnish patch for https://apps.fedoraproject.org/nuancier/ - repaired proxy.vcl.j2.patch
Yes, you are right, nuancier part is missing in match condition. I made a new patch. Im sorry.
This patch looks sane to me. Will merge it.
Please note that this does not close the ticket, there are lots of other services that should be switched to Varnish.
Ansible patch for caching https://apps.fedoraproject.org/github2fedmsg github2fedmsg.patch
I know there is a lot of work to do :) There is a new patch for caching https://apps.fedoraproject.org/github2fedmsg/static
Ansible patch for caching https://apps.fedoraproject.org/notifications notifications.patch
Patch for caching https://apps.fedoraproject.org/notifications. Because I found a some whitespace characters at end of lines in roles/varnish/templates/proxy.vcl.j2 in nuancier section, this patch will remove this unnecessary characters also.
Please group applications for the same host in the same component (so only keep one comparison with apps.fedoraproject.org). Otherwise, this will incur a big performance penalty.
Also, I just remembered why we didn't have that for apps. previously: it breaks staging. Please either make it an or between apps.fp.o || apps.stg.fp.o, or just remove the if for forwarded-host.
Since these applications are only on apps.fp.o, that should be just fine.
Ansible patch for caching https://apps.fedoraproject.org/github2fedmsg + groups apps.fp.o github2fedmsg-with-group.patch
I group all application for apps.fp.o into one condition with logical OR for staging. Its seems to me it will be faster this way than without forwarder-host condition regarding other hosts.
Patch will also add caching for github2fedmsg app.
Iam looking at caching apps.fp.o/packages. There is already section in varnish config file, but apache forwards requests directly to packages backend. I suppose there was problem with that. So, do anybody remember why you bypass varnish for this app?
Hi, thanks for the patch, the grouping looks great!
Please note however that caching github2fedmsg is useless since there are no non-authenticated pages and the amount of CSS and image files it uses is very small, and it may break the github webhook if we're not careful.
I don't remember exactly from the top of my head why /packages isn't cached, but I think that it was related to some bugs in the older code. Though I'd like to ask @ralph about his opinion.
I don't remember why it was disabled, but we could try it and see how it works. Can we try it in staging first?
Oh, this is probably relevant: the httpd config for the app itself has some cache control settings that are worth looking at: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/packages3/web/files/packages-httpd.conf
Thanks Ralph for your notes. Packages app is using pretty well a browser caching. I am thinking about benefits of caching this app by Varnish and I dont find it useful for now. Varnish should be used only for first request to static content. Or Iam wrong?
Ansible Varnish patch for grouping apps.fp.o apps_fp_o_grouping.patch
In this patch I group all application for apps.fp.o into one condition with logical OR for staging.
I attach similar patch with caching github2fedmsg app, but it will be easier to work on caching of another apps if there will be this grouping done.
Patch does not change any caching, only groups conditions in logical way.
If you like it, please push it to staging. Thanks
I am very, very sorry that it took so long to get back to this patch, thank you for bearing with me. The patch itself look great! I'm going to apply it on staging now, and we'll see how it goes!
This is live now. ;)
If you can think of any new improvements please file a new ticket or discuss with patrick.
Login to comment on this ticket.