Original symptom:
github2fedmsg01's clock is about 4 minutes behind. This causes fedmsg messages from github to look like they are always arriving other places "late". They just have a screwy timestamp.
I looked into our ansible repo for our ntp config, and there are some tasks at the root level for cloud and sign_setup that reference ntp, but there's nothing in roles/base that I could find. This might need a closer look by someone who knows what they're doing with ntpd.
To summarize, two issues:
Got github2fedmsg01 working with 'ntpdate -u ntp1.tummy.com'. I'll leave the other issue for another sysadmin to look into or close as notabug.
I think relrod fixed this up with some changes to base.
We are going to have all non virt machines run ntpd and see if any virt machines drift. If they do, we will just install it on all machines.
Just noticed that github2fedmsg01 is about 2 minutes behind again.
ok, looks like guests won't keep time with the hosts.
I've pushed a change that will enable ntp on the guests too.
Re-opening to look back into this after the freeze is up. github2fedmsg01 has ntpd running on it now, but it's still about 2 minutes behind from what I can tell.
I've never had good luck with ntpd -- never seemed to get it configured right and it would always go out of sync. Maybe worth trying chronyd?
This was caused I think by an old ntp.conf. New one was put in place, but ntpd didn't get restarted.
Things should be ok now, please re-open if you see this again.
Login to comment on this ticket.