if these two machines run linux-bridges for the "extra" IPs reported, then we can filter-out interfaces with type bridge along with their IPs in the fetch-ssh-keys script.

kindly, run ansible setup on both machines and attach it to the ticket to have a look.

attachment
fed-cloud02

attachment
serverbeach09

Both output attached

One way I can think about this is that: to build the known_hosts file, and distributed to the servers, one needs the primary IP of the machine; the one it uses to access the outside world. accordingly, we could replace the {{ansible_all_ipv4_addresses}} fact with {{ansible_default_ipv4}}.address.

further investigation, reveals that tunnel IPs might be needed, as for example, cron scripts ssh to the tunnel IP itself.

the latter can be dealt with as an exception, by looking into a interface-with-type-tunnel fact, and adding its corresponding address.

to get rid of the exception case, one needs to inventory the use of tunnel end-point IP usage, and possible point that to the main ip of the machine.

Here is my attempt on the script.

kindly, test and feedback

I seem to be getting a traceback in testing:

{{{
Traceback (most recent call last):
File "./fetch-ssh-keys", line 73, in <module>
names.append(non_link_local["address"])
TypeError: list indices must be integers, not str
}}}

Replying to [comment:6 kevin]:

I seem to be getting a traceback in testing:

{{{
Traceback (most recent call last):
File "./fetch-ssh-keys", line 73, in <module>
names.append(non_link_local["address"])
TypeError: list indices must be integers, not str
}}}

fixed. kindly run and feedback.

Looking better. ;)

It seems to be picking up however on eth1 ip's on some hosts. Some phx2 hosts have a 10.5.127.x ip on eth1 to use for storage. We likely don't need this in known_hosts at all. I can't think of any reason we would ssh over the storage network.

for example:

{{{
wiki01.phx2.fedoraproject.org,wiki01,10.5.127.43,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
}}}

So, I guess we either need to filter eth1 out, or 10.5.127.x ip's or something...

wiki01
wiki01

Replying to [comment:8 kevin]:

Looking better. ;)

It seems to be picking up however on eth1 ip's on some hosts. Some phx2 hosts have a 10.5.127.x ip on eth1 to use for storage. We likely don't need this in known_hosts at all. I can't think of any reason we would ssh over the storage network.

for example:

{{{
wiki01.phx2.fedoraproject.org,wiki01,10.5.127.43,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
}}}

So, I guess we either need to filter eth1 out, or 10.5.127.x ip's or something...

makes sense. although I asked around and found that this machine has a specific net config that might differ from others.

wiki01 has its net setup as inbound from iface eth0 (126 net) and default outbound through eth1 (127 net), which is as you mentioned used as well for storage. I was not able to find out the reason behind this, though, and will be interested to know why.

so, here we have two alternatives:
1- take the machines net config as is, and deal with it and others with similar config as corner cases. then I need the list of machines you encountered this behavior on.

2- if there is no harm, change the default interface to be eth0 (126) and assign it a gateway (same for others with similar case), and the script will do its job, without dealing with corner cases.

awaiting feedback.

Thanks,

ok, we fixed wiki* (and virthost10) which had default 10.5.127 routes when they shouldn't have.

One last issue I now see with the output...

We are adding in the 'short' name of a host also, which is great, but in the case of stg hosts it will cause overlap. For example we get:

{{{
wiki01.phx2.fedoraproject.org,wiki01,10.5.126.63,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
wiki01.stg.phx2.fedoraproject.org,wiki01,10.5.126.60 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5XlcxpCOQiq7WT/RA3OWDBdw7LUXDhdq5vIRUH9vtvX+i72vS+DSc/l0VI9Y7JgiEoyGTWN2ncFMuxp6SeW7WGxBs9mDflutLqZWrbTBR2KBPA16kAaE1GjkvlxTy4ubvmHpqL4Iyxh0N3jcFZbak9LHCAH3iycuFX1KLe7YT7wlrE0Ng3GVV1AVwWbINq2tmotsRmiiqiiU4+y/R2eLy2FhJfoAUXqh2Cn5jqWRsAZWj7WNzLWz/sT+T4v6Y1hlna7K6o4hWxJlI9BJ/s6RwNMPBFVPYOlceXiNCC33CXLD13xKkiHs/nPbkS8RMjY5LnYicABjUykBnVh/X9yoVQ== wiki01.stg.phx2.fedoraproject.org
}}}

So both end up with 'wiki01'. I guess we need a case for stg and always add it's short name with a .stg after it? This would allow for 'ssh wiki01.stg' to work for example.

Replying to [comment:10 kevin]:

ok, we fixed wiki* (and virthost10) which had default 10.5.127 routes when they shouldn't have.

One last issue I now see with the output...

We are adding in the 'short' name of a host also, which is great, but in the case of stg hosts it will cause overlap. For example we get:

{{{
wiki01.phx2.fedoraproject.org,wiki01,10.5.126.63,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
wiki01.stg.phx2.fedoraproject.org,wiki01,10.5.126.60 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5XlcxpCOQiq7WT/RA3OWDBdw7LUXDhdq5vIRUH9vtvX+i72vS+DSc/l0VI9Y7JgiEoyGTWN2ncFMuxp6SeW7WGxBs9mDflutLqZWrbTBR2KBPA16kAaE1GjkvlxTy4ubvmHpqL4Iyxh0N3jcFZbak9LHCAH3iycuFX1KLe7YT7wlrE0Ng3GVV1AVwWbINq2tmotsRmiiqiiU4+y/R2eLy2FhJfoAUXqh2Cn5jqWRsAZWj7WNzLWz/sT+T4v6Y1hlna7K6o4hWxJlI9BJ/s6RwNMPBFVPYOlceXiNCC33CXLD13xKkiHs/nPbkS8RMjY5LnYicABjUykBnVh/X9yoVQ== wiki01.stg.phx2.fedoraproject.org
}}}

So both end up with 'wiki01'. I guess we need a case for stg and always add it's short name with a .stg after it? This would allow for 'ssh wiki01.stg' to work for example.

Added the exception to deal with stg.

kindly, test and feedback.

Sadly, now it's including .stg on all hosts. ;(

{{{
+wiki01.phx2.fedoraproject.org,wiki01.stg,10.5.126.63,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
+wiki01.stg.phx2.fedoraproject.org,wiki01.stg,10.5.126.60 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5XlcxpCOQiq7WT/RA3OWDBdw7LUXDhdq5vIRUH9vtvX+i72vS+DSc/l0VI9Y7JgiEoyGTWN2ncFMuxp6SeW7WGxBs9mDflutLqZWrbTBR2KBPA16kAaE1GjkvlxTy4ubvmHpqL4Iyxh0N3jcFZbak9LHCAH3iycuFX1KLe7YT7wlrE0Ng3GVV1AVwWbINq2tmotsRmiiqiiU4+y/R2eLy2FhJfoAUXqh2Cn5jqWRsAZWj7WNzLWz/sT+T4v6Y1hlna7K6o4hWxJlI9BJ/s6RwNMPBFVPYOlceXiNCC33CXLD13xKkiHs/nPbkS8RMjY5LnYicABjUykBnVh/X9yoVQ== wiki01.stg.phx2.fedoraproject.org
}}}

Replying to [comment:12 kevin]:

Sadly, now it's including .stg on all hosts. ;(

{{{
+wiki01.phx2.fedoraproject.org,wiki01.stg,10.5.126.63,192.168.1.129 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0c5WAVH4jrNNnMvK4Cyj+NH5NQa+s5pwois0W05dl/38kRHpwmLM7YshO5pUyL3H290zCkB27alBAtS4IYduoKNZ+v5z0KFFux2T/qY1YLpw/FbdfviF/GdXqS/uUHTNQp9kISZUkKcbmnu1qQfZq+GAVEdbQH/7H/TrWxqCV0NsUzRlCzj4VMM7NWdG/RqLkdK+ielq5F5Qry8BRWV+m+qUkTYLr+7wVsQvQ50CsIQ5pYePqzLEaIrWH7jzTIv1AgixhmfgRfQtmOpI1xlBmiBkgUCIYS8KYwZCxQks7L2XmjaXsEficzNBtxHmOx1UjfPK00sVb9FPoH+z/NbrIw== wiki01.phx2.fedoraproject.org
+wiki01.stg.phx2.fedoraproject.org,wiki01.stg,10.5.126.60 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5XlcxpCOQiq7WT/RA3OWDBdw7LUXDhdq5vIRUH9vtvX+i72vS+DSc/l0VI9Y7JgiEoyGTWN2ncFMuxp6SeW7WGxBs9mDflutLqZWrbTBR2KBPA16kAaE1GjkvlxTy4ubvmHpqL4Iyxh0N3jcFZbak9LHCAH3iycuFX1KLe7YT7wlrE0Ng3GVV1AVwWbINq2tmotsRmiiqiiU4+y/R2eLy2FhJfoAUXqh2Cn5jqWRsAZWj7WNzLWz/sT+T4v6Y1hlna7K6o4hWxJlI9BJ/s6RwNMPBFVPYOlceXiNCC33CXLD13xKkiHs/nPbkS8RMjY5LnYicABjUykBnVh/X9yoVQ== wiki01.stg.phx2.fedoraproject.org
}}}

sorry. please test now and feedback.

attachment
fetch-ssh-keys

ok, looks like we have a winner. ;)

I've commited this and pushed out a ssh_known_hosts using it. Please look it over and let me know if you see anything else to change.