#2783 Check file changes against puppet for validity
Closed: Fixed None Opened 12 years ago by kevin.

We would like to go and check the output of 'rpm -Va' on all machines against puppet to see what files really should be out of sync with the rpms.

  1. If the file is produced by puppet thats fine.
  2. If the file is modified by install or kickstart, thats fine.
  3. If the file is not in puppet, and not modified on install normally, we want to investigate why it is modified and fix it/resync it to the rpm.

This is a great task for fi-apprentices to do as it lets you see how puppet is setup and how each of our machines in turn are configured, etc.

Multiple people could work on this task. Just be sure and note what machine you are currently working on so there is less duplication of effort. In the end we want a list of 'these files are modified from the rpm, but should not be' that we can then investigate. If there are files you cannot read, ask someone in a more privileged group to check for you (irc would be fine for this).

A list of 'rpm -Va' output from all hosts is available on puppet01 in /tmp/rpm-va/


I got the script puppet01:/home/fedora/janfrode/puppet-checksums.py from skvidal, which will take a state.yaml file (i.e. /var/lib/puppet/state/state.yaml) as argument and print the filenames and checksums of files puppet is managing according to the state.yaml.

But to run it we need access to these files. Could maybe the apprentice group get access to run "sudo /bin/cat /var/lib/puppet/state/state.yaml" to read these on all relevant machines? Or are there other suggestions for how to find out which files are puppet managed ?

Full list of files changed according to rpm -Va but not managed by puppet:

puppet01:/home/fedora/janfrode/changed-files-not-in-puppet.txt

(generated by "/home/fedora/janfrode/compare-rpm-vs-puppet.py $hostname" which compares /tmp/rpm-va/$hostname.output to /tmp/state-yaml/$hostname/state.yaml)

== Report for value01.phx2.fedoraproject.org ==

Unable to check these (access denied):

S.5....T c /etc/securetty

S.5....T /etc/cron.d/smolt

S.5....T c /etc/inittab

......G. /etc/audit

Otherwise:

/etc/login.defs has "ENCRYPT_METHOD MD5" which I believe can be removed, and then login.defs would be same as default.

/etc/pam.d/system-auth changed on all systems. Probably happens during kickstarts that pam is installing this as a file, but then authconfig changes it to a symlink to system-auth-ac.

/etc/sysctl.conf has the following changes (compared to puppet01):

            kernel.sysrq = 0 (1 on puppet01)

            kernel.shmmax = 4294967295 (1073741824 on puppet01)

            kernel.msgmnb = 65536 (not set on puppet01)

            kernel.msgmax = 65536 (not set on puppet01)

            kernel.shmall = 268435456 (not set on puppet01)

/etc/logrotate.d/zarafa. No /etc/logrotate.d/zarafa in puppet. Should probably be configured there.

/etc/idmapd.conf -- only timestamp difference (.......T), should be reset to default timestamp.

Replying to [comment:4 janfrode]:

== Report for value01.phx2.fedoraproject.org ==

Unable to check these (access denied):

S.5....T c /etc/securetty

Looks like we add (or the install does):
xvc0
(The xen console). So, this likely will show up on any machine that is a xen guest. ;(

S.5....T /etc/cron.d/smolt

This file will sadly never verify. ;( It's generated by a scriptlet (see 'rpm -q smolt --scripts) to make it run at a random time (so all clients don't check in at the same time and swamp the server). So, this is ok I suppose.

S.5....T c /etc/inittab

This is again the xen console:
co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav

Can be ignored and probibly will be the case on any other xen guest machine. ;(

......G. /etc/audit

We change the group here to 'sysadmin-noc' to allow noc people to look at audit rules I suppose. The group is listed as a numeric in puppet. Perhaps to get around issues with new machines that don't have the groups yet?

Otherwise:

/etc/login.defs has "ENCRYPT_METHOD MD5" which I believe can be removed, and then login.defs would be same as default.

Our kickstart seems to have:
authconfig --enableshadow --enablemd5

I think we should/could look at removing the enable there and clean these up as you indicate.

/etc/pam.d/system-auth changed on all systems. Probably happens during kickstarts that pam is installing this as a file, but then authconfig changes it to a symlink to system-auth-ac.

Yep. This will happen on them all. ;(

/etc/sysctl.conf has the following changes (compared to puppet01):

            kernel.sysrq = 0 (1 on puppet01)

            kernel.shmmax = 4294967295 (1073741824 on puppet01)

            kernel.msgmnb = 65536 (not set on puppet01)

            kernel.msgmax = 65536 (not set on puppet01)

            kernel.shmall = 268435456 (not set on puppet01)

We should get these synced up. I wonder why puppet is not setting the ones that are set above? Needs more investigation.

/etc/logrotate.d/zarafa. No /etc/logrotate.d/zarafa in puppet. Should probably be configured there.

It looks to me like we can just go back to using the one in the package.
Moved back in place. It looks like we had some output from the job in the past
and changed it to /dev/null the output. I don't think this is needed anymore.

/etc/idmapd.conf -- only timestamp difference (.......T), should be reset to default timestamp.

Fixed.

Thanks for working on this!

rpm -Va output does not exist in /tmp/rpm-va/ on puppet01, may have been cleaned-out?

I've checked/fixed puppet01 (before we moved it to lockbox01), so we can cross that one off.

We can get the output setup again later today.

I looked at updating this today. Looks like we need the syck package for epel6.
I filed a bug to get that. If they don't respond soon, I will build it for us.

ok, some notes:

  • We need to refresh the files in order to look at this again. Will try and do so soon.

  • puppet01 no longer exists, all files will be on lockbox01.

I'd like to tackle this one. Have there been any recent updates on this?

Replying to [comment:9 kevin]:

ok, some notes:

  • We need to refresh the files in order to look at this again. Will try and do so soon.

  • puppet01 no longer exists, all files will be on lockbox01.

Kevin, can you let me know what files need to be "refreshed" ?

We need to regenerate the changed-files-not-in-puppet.txt file.

This will show changes on hosts where files are different from their rpm state, but yet not changed by puppet.

I am regenerating that now. Will update in a few.

ok. On lockbox01 in /tmp/changed-files-not-in-puppet.txt

@jclark48, are you working on this or can I take it over?

@skotter, yes you can go ahead and take this over. I won't have time to contribute much since starting my new gig, so go for it.

May I help in this ticket?

Sure. Just note what hosts you are looking at and let us know if you need any files or have questions...

let me research on this and I will come to what I need.

today, I discussed about this ticket with skvidal, and he pointed out that there is a plugin in yum to do similar thing. let me know if this still need to be worked on.

Yeah, we talked some more about it, and I think we are going to look at doing a ansible playbook to do this in an automated way.

It would need to go to each machine, rpm -Va and then check that against config files it knows about and only output the ones that are not changed by ansible/puppet. This is likely down the road a bit as we have some machines using puppet and some ansible, so it could be confusing right now.

Lets hold off for now on this...

Moving all currently open easyfix tickets to the HANDYWAVY-FUTURE milestone.

I'm clearing the assigned status on all easyfix tickets.

If you are an apprentice actively working on this ticket, feel free to reassign to yourself. Otherwise let a new apprentice have a look.

Closing this one now.

Login to comment on this ticket.

Metadata