Good catch, this is an SELinux issue that we hit in production too. We've temporarily set SELinux to permissive on production for this, so I did the same on fas01.stg for now and started memcached.

Luke, any idea what kind of SELinux policy we need for this? I think it's not being allowed to bind to tcp/udp 11211 or something:

{{{
node=10.5.126.86 type=AVC msg=audit(1261698519.891:3390): avc: denied { name_bind } for pid=25569 comm="memcached" src=11211 scontext=user_u:system_r:memcached_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket
node=10.5.126.86 type=AVC msg=audit(1261698519.891:3391): avc: denied { name_bind } for pid=25569 comm="memcached" src=11211 scontext=user_u:system_r:memcached_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=udp_socket
}}}

I'm not seeing this issue happening in production anymore, however I am seeing it in staging:

{{{

============= memcached_t ==============

allow memcached_t http_cache_port_t:tcp_socket name_bind;
allow memcached_t http_cache_port_t:udp_socket name_bind;
allow memcached_t self:netlink_route_socket { write getattr read bind create nlmsg_read };
}}}

CCing dwalsh -- Dan, what's the best fix for this? Should we just add these rules to the custom FAS policy module? or is there a simple boolean we can flip?

I have a partial fix in staging

Closing -- it looks like SELinux + memcached are playing nicely in staging now.