Are we sure that the switch that xen2 and the nfs server are plugging into aren't overloaded? I believe that at least some of the systems in PHX are plugged into a Cisco Catalyst 2950G-48, which only has 13Gb/s of forwarding capacity (at least that's what Cisco advertises, real-world performance is probably less). Depending on how many of the ports in use we could be hitting the performance limits of the switch. What would really be nice would be to get a separate switch that we could designate as a storage network and would carry only iSCSI and NFS traffic. Second best would be to directly connect xen2 and the NFS server on 2nd ports, however that does not scale for obvious reasons.

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps628/product_data_sheet09186a00801cfb64.html

Also, it would be really nice to have a diagram of how the servers in PHX are connected network-wise. Since RH is using Cisco switches, running the following command on the physical hosts should give us enough information to start a fairly accurate map:

{{{
/usr/sbin/tshark -i eth0 -a duration:120 -c 1 -V ether dst 01:00:0c:cc:cc:cc 2>&1 | /bin/mail -s "CDP from $(hostname)" jcollie@fedoraproject.org
}}}

(will need to replace eth0 with peth0 if this is a xen host).

To be honest, its been a while since I've worked with any catalysts. I can put in a ticket request for the soc, can you tell me exactly what info you want?

Well, to start with it'd just be nice to have a map of where things plug into physically on the swicthes. Perhaps it's because IRL I'm a network engineer (it says so on my business cards so it must be true) but I find it helpful to know these things. We can ask RH for the info, or we can gather some of the information on our own (see the command I posted above). Basically, what I'd like to know is what switch, what port on that switch, and what VLAN was assigned to the port for all of the Fedora machines in PHX. Knowing where the PHX PIX plugs in would be good to know too. If the Fedora switches plug into several different switches having some sort of idea how the switches were interconnected would be good too. Also intertesting information to have would be to know what non-Fedora systems are using the same switches.

xen2 and xen5 now have a dedicated link to eachother.

Issues on the actual switch have been identified but not root cause. Eventually we'll be replacing the switch.