Adding adrian here for input...

Thanks for the detailed report. It sounds interesting indeed. The code you had a look at seems to be MirrorManager1. Fedora is currently running on MirrorManager2. That part of the code, however, has not changed much and does the same. I am pretty sure that that part of the code is no problem. That is part of the peer ASN handling which both of the princeton mirrors do not use.

Looking at the code of MirrorManager2 the host_id is used in the example you cite. So this looks all correct.

There could be of course an error in the ASN handling but I tested it on two other mirrors in the same ASN without a problem:

{{{
$ curl -s "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=x86_64&ip=185.134.84.1" | head -3

repo = epel-7 arch = x86_64 Using ASN 553 Using ASN 553 country = DE country = BG country = BY country = RU country = RO country = GR country = GB country = PT country = PL country = ES country = MD country = IL country = FR country = FI country = NL country = NO country = CH country = CZ country = SK country = SE country = DK country = TR country = LT country = IS country = AM country = AT country = UA

https://ftp-stud.hs-esslingen.de/pub/epel/7/x86_64/
http://ftp.uni-stuttgart.de/epel/7/x86_64/
}}}

Both mirrors have ASN 553 listed in their configuration. This seems to work correctly.

Looking at EPEL for for princeton I get both mirrors:

{{{
$ curl -s "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=x86_64&ip=128.112.172.21" | head -10

repo = epel-7 arch = x86_64 Using preferred netblock Using preferred netblock Using ASN 88 Using ASN 88 Using Internet2 country = US country = CA

http://mirror.math.princeton.edu/pub/epel/7/x86_64/
http://mirror.cs.princeton.edu/pub/mirrors/fedora-epel/7/x86_64/
}}}

Looking closer at the rsync URL of mirror.cs.princeton.edu it looks wrong. It seems the crawler is disabling the host and report_mirror is enabling the host. It works most of the time as the HTTP URL is correct. I fixed the RSYNC URL for mirror.cs.princeton.edu 'Fedora Linux' and it should return both mirrors 'soon'. As it works for EPEL I guess/hope it will also work correctly for 'Fedora Linux'.

In the beginning you said that the mirrors are in different datacenters. If you both specify the same netblocks and the same ASN you can never control who gets which mirror.

I started a manual crawl of mirror.cs.princeton.edu to see if it will be enabled in about an hour. Closing for now as everything seems work as designed (and wanted).

Now something interesting has happened. mirrorfedora.math.princeton.edu has no longer any directories listed as being up to date. Re-opening as this is strange. I cannot yet explain what is going on. According to the report_mirror timestamp no new check-in has happened since the last crawl. The last crawl has marked many directories as up to date. But somehow they disappeared. Maybe there are some internal database links wrong or broken. I have to look closer at those sites and hosts.

I see there was a check-in at 2016-08-30 18:15:27.

Could it be that your report_mirror configuration is broken? Can run it and check what it returns?

You can also look at https://admin.fedoraproject.org/mirrormanager/host/1016/category/3973 to see if it is empty. Right now it lists a lot of up to date directories.

Hello,

Thanks for your time, and I think now we are getting somewhere. After running report_mirror, all of the entries in the link you provided disappeared.

Attached is the output of my report_mirror run with debugging enabled. The command was:

{{{
report_mirror -d -c /etc/mirrormanager-client/report_mirror_fedora.conf > fedora_mm.output.20160830.1449.txt
}}}

The output can be found here:

http://mirror.math.princeton.edu/pub/fedora_mm.output.20160830.1449.txt

Will post the config file shortly as an attachment.

Thanks,
Ben

report_mirror config for main fedora repo
report_mirror_fedora.conf

Please change

path=/var/www/html/pub/fedora

to

path=/var/www/html/pub/fedora/linux

The paths with our different mirror modules are confusing and you are missing one directory level.

Thanks, made that adjustment and reran the report_mirror job. Now the list is populated in the link you sent above. However, running the same curl as above on the metalink target yields the same results:

{{{
[root@test ~]# curl "https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64" 2>/dev/null | grep -i princeton
<url protocol="http" type="http" location="US" preference="78" >http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
<url protocol="rsync" type="rsync" location="US" preference="78" >rsync://mirror.cs.princeton.edu/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
}}}

Perhaps this can take a little while to update correctly.

Ben

Hi Adrian,

Thanks much for all your help. Things seem to be working as expected now:

{{{
[root@test ~]# curl "https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64" 2>/dev/null | grep -i princeton
<url protocol="rsync" type="rsync" location="US" preference="98" >rsync://mirror.math.princeton.edu/pub/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
<url protocol="http" type="http" location="US" preference="98" >http://mirror.math.princeton.edu/pub/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
<url protocol="http" type="http" location="US" preference="94" >http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
<url protocol="rsync" type="rsync" location="US" preference="94" >rsync://mirror.cs.princeton.edu/fedora/linux/releases/24/Everything/x86_64/os/repodata/repomd.xml</url>
}}}

The difference is night & day with the amount of traffic being transmitted off of mirror.math. Currently averaging over half a gigabit per sec for just Fedora alone. This issue can probably be closed, I'll reopen or open a new ticket if anything crops up again.

Thanks again,
Ben

Ah, maybe I can close it myself :-)