wiki:NewReleaseTesting
Last modified 6 years ago Last modified on 10/08/08 11:46:33

Testing New Release

These are tests we perform when doing a new release. Should probably be automatized, at least to some extent (detect a traceback, check for $?, ...)

Library testing

In the git repo, there is a selftest.py file (in src/scheduler) which is not being distributed with rpm. Run it with python selftest.py and make sure all the test cases pass.

Sanity testing

Test the most basic functions to make sure sectool is able to start, etc.

  1. sectool --help
  1. sectool --version
    • make sure that the version string is correct
  2. sectool --list
    • check that the level numbers are sane
    • test globbing: sectool --list \*home\*
  3. sectool --info <test>

Running the tests

Test that sectool is able to run tests.

  1. test running a single test in default level - sectool --run bootloader
  1. test running a single test in non-default level - sectool --level 5 --run bootloader
    • bootloader is a good candidate since it should stay silent in the default level and ERROR in level 5
  2. test running multiple levels in default/nondefault mode
    • sectool --run bootloader home_files
    • sectool --level 5 --run bootloader home_files
    • test globbing: sectool --run \*home\*

Running a level

  1. Run a level sectool --level 5
    • level 5 might be good if you have enough time, level 1 is probably OK for quick smoke testing
  2. Run a level via its nickname sectool --level Desktop
    • the valid levels are: Naive, Desktop, Network, Server, Paranoid
  3. Exclude a test from its level sectool --level 1 --exclude integrity
    • check that all the tests in level 1 except for integrity ran
  4. Include a test into another level sectool --level 1 --include path
    • check that all the tests in level 1 and 'path' ran

Display filters

  1. check that sectool display additional info when ran with --debug
    • a good candidate is home_files - sectool --run home_files --debug
  2. check that sectool displays hints when ran with --hint
    • a good candidate is mountopt - sectool --run home_files --debug

The auto action

  1. Running a whole level via --action
    • set the [ACTION] section of /etc/sectool/sectool.conf as follows:
          [ACTION]
          LEVEL=1
      
    • run sectool --auto, whole level 1 should run
  1. Running some tests on a specific level
    • set the [ACTION] section of /etc/sectool/sectool.conf as follows:
          [ACTION]
          LEVEL=5
          RUN=path bootloader
      
    • run sectool --auto, tests path and bootloader should run on level 5
  1. Running some tests on a default level
    • set the [ACTION] section of /etc/sectool/sectool.conf as follows:
          [ACTION]
          RUN=path bootloader
      
    • run sectool --auto, tests path and bootloader should run default level
  1. Including and excluding tests from a level in the auto action
    • set the [ACTION] section of /etc/sectool/sectool.conf as follows:
          [ACTION]
          LEVEL=1
          RUN=+bootloader -integrity
      
    • run sectool --auto, level 1 should run excluding integrity and including bootloader

Diff and sending a mail

  1. check that sectool is able to produce a meaningful diff
    • sectool --run suid; chmod a+s /bin/cp; sectool --run suid --diff; chmod a-s /bin/cp
  2. sending email - vanilla config
    • that is SEND_BODY=diff SEND_ATTACHMENT=full
    • make sure your sendmail is running and properly configured
    • sectool --run home_files --mail jhrozek@redhat.com
  3. sending email - full output in body, diff in attachment
    • set SEND_ATTACHMENT=diff and SEND_BODY=full in /etc/sectool/sectool.conf
  4. sending email via SMTP server
    • the default is via sendmail
    • configure your server, username and password in /etc/sectool/sectool.conf and change TARGET=smtp
  5. check the --clean action
    • run a test, make sure that results.xml exists
    • sectool --clean should remove it

Overriding levels

  1. check that level can be added/removed via config file
    • echo "path 1 2" > ~/.sectoolrc
    • test a level that is not included in the dsc
  2. check that disabling all levels work
    • echo "path" > ~/.sectoolrc

Logging

  • see /var/log/sectool.log is created, filled and properly formated
  • check that logrotate handles sectool.log