With the new way we are doing staging configuration (see http://infrastructure.fedoraproject.org/infra/docs/staging.txt ) we want to run a weekly script that does the following:
runs on lockbox01 checks out puppet repo to a tmp dir looks in the modules-staging directory for any subdirectories. if none, exit with no output. for each that exists: diff -Nur modules/$directory modules-staging/$directory (ie, diff against the non staging directory) mail a report listing directories and diffs out so we know whats being worked on in staging.
This script would need to be added to puppet under modules/scripts/
attachment staging_monitoring.sh
Will something like staging_monitoring.sh work?
I am not confortable with using a hardcoded directory in /tmp/, as this is a vector of various type of attack ( symlink attack, etc ).
{{{ tmpdir=/tmp/staging_monitoring
...
cleanup(){ rm -rf "$tmpdir" } }}}
I would also say there is more elegant way to make git silent than using /dev/null ( like -q )
And for mail filtering purpose, what about adding a header to the mail ( like X-Fedora-Script or something else ? )
This looks like a pretty good first cut. ;) We are in freeze right now, so we can't apply anything, but we can work on making this all ready to commit.
In addition to the above comments, I'd suggest using 'mktemp' for the temp file creation.
We could add a header to filter, but hopefully once a week these emails shouldn't be too much trouble.
Also, I'd say we should sent to the infrastructure list.
Finally if you want to work on a git diff to add this script into the modules/scripts/ area and add a cron job for it once a week on lockbox01, running as the nobody user that would be great. Happy to expand on that.
attachment 0001-Added-script-to-monitor-staging-modules.patch
I added a patch with the modifications suggested to the script (not the X-Fedora-Script header though, I need to research how to do that...).
I guess next step is adding it to cron :)
Yep.
If you could make a cron.d file that runs this script once a week that would be great.
You should be able to add it to the above patch and I can commit the entire thing at once. :)
.cron-file which calls the script tuesday at 06:00 diff
Sorry, forgot to type more information :)
The cron-file calls the script tuesday at 06:00, if it's provided in /usr/local/bin. I assumed the username was 'nobody' like in other .cron-files in /modules/scripts/files
ok, sorry for the delay here. ;)
I finally got to looking at this script. Running it here gives me:
modules/scripts/files/staging_monitoring.sh: line 40: syntax error near unexpected token <' modules/scripts/files/staging_monitoring.sh: line 40:done < <(find ./ -mindepth 1 -maxdepth 1 -type d)'
<' modules/scripts/files/staging_monitoring.sh: line 40:
Would you like to fix it up? Or shall I?
new version staging_monitoring.2.sh
Try this Version. I've changed line 40 from:
{{{ done < <(find ./ -mindepth 1 -maxdepth 1 -type d) }}}
to
{{{ done < (find ./ -mindepth 1 -maxdepth 1 -type d) }}}
(find ./ -mindepth 1 -maxdepth 1 -type d)
Hope that gets the intended result.
{{{
$ sh -x staging_monitoring.sh ++ mktemp -d + tmpdir=/tmp/tmp.0UJq2TlfB4 + report=/tmp/tmp.0UJq2TlfB4/report + unset i list + cd /tmp/tmp.0UJq2TlfB4 + git clone /git/puppet -q + cd puppet/modules-staging ++ find ./ -mindepth 1 -maxdepth 1 -type d ++ find ./ -mindepth 1 -maxdepth 1 -type d staging_monitoring.sh: line 40: ./fedmsg ./packages ./mediawiki ./git ./yum ./bodhi ./gitolite: No such file or directory + [[ 0 == 0 ]] + cleanup + rm -rf /tmp/tmp.0UJq2TlfB4 + exit 0 }}}
attachment staging_monitoring.3.sh
Hi,
please try again. I think the while-loop used before went through the directories twice. I've changed that to a simpler for-loop.
btw this task definitely screams for python :)
Yes, it does work now. ;)
Some further thoughts:
The output is a bit muddled. Perhaps we could use diffstat to just show stats instead of the full diff?
If you can re-write it in python and make it cleaner, thats ok with me.
It would be nice to seperate the modules more, currently they are all in one big diff and it's hard visually to see whats different.
thanks!
attachment staging_monitoring.4.sh
HI,
changed the output to diffstat instead of the full diff. Currently there is no diff between modules and modules-stating, so I can't seeh if that's already enough seperation.
I will work on the python-version, but I don't know when I will get around to it (waiting for my next night-shift :) ), so I hope it's ok to use the bash-version until then.
I seem to get empty diff output now. ;(
}}}
(sorry for the long delay here)
That's what I meant with "no diff". I had to change some lines in the Readme-Files to get diff output. I thought currently there was no diff between modules and staging.
Anyway I started on the python version and ran into a stranger behaviour, when using this code:
{{{ TMPDIR=tempfile.mkdtemp() subprocess.Popen( [ '/usr/bin/git', 'clone', '/git/puppet', '-q' ], cwd=TMPDIR, stdout=subprocess.PIPE) STAGING=os.path.join(TMPDIR, "puppet/modules-staging/") MODULES=os.path.join(TMPDIR, "puppet/modules/") for project in os.listdir(STAGING): staging=os.path.join(STAGING, project) modules=os.path.join(MODULES, project) ... }}}
I get this error:
{{{ Traceback (most recent call last): File "/home/fedora/ingmar/staging_monitoring.py", line 16, in <module> for project in os.listdir(STAGING): OSError: [Errno 2] No such file or directory: '/tmp/tmp0cL3vd/puppet/modules-staging/' }}}
But when I check the directory and it's contents are there. Any Ideas about that?
I think you have a race condition there.
Perhaps use Popen.wait() to make sure the clone has finished?
attachment 0001-add-staging-monitoring-script.patch
I add a patch with the part that is needed.
Cool. So that generates the report fine here... I wonder what we could do to make it more readable.
I fear the diffs might get too large to be easily readable.
So, perhaps we should just go back to diffstat only for now?
Or do you have any other ideas to make it more readable? I suppose we could add the diffs as attachments, but that may be not easy to do.
send attach report 0001-add-staging-monitoring-script.2.patch
That says so?. Other option is try with fpaste.
well, fpaste expires and would be more links to have to click on.
Lets go with just the diffstat to start with... ;)
Of course we are in freeze right now, so we will need a freeze break request to commit this. I will submit one...
ok, the last patch sends diffstat and attached a file with diff-Nur
Moving all currently open easyfix tickets to the HANDYWAVY-FUTURE milestone.
I'm clearing the assigned status on all easyfix tickets.
If you are an apprentice actively working on this ticket, feel free to reassign to yourself. Otherwise let a new apprentice have a look.
Closing this now. We are moving to ansible and will need to revisit how to handle this there.
Login to comment on this ticket.