So this about a script which is called, but since it's not applicable it wants the
generic test to run instead.

I think this could be handled in two ways:
- Your proposal, meaning that when a deprecating test returns 'na' all tests which
are deprecated by this one should run. One pb here is that the python code needs
to know whether the called shellscript has concluded that the result is indeed
'na', or if just wants the generic code to run. So, to implement this we need a
new return code reflecting the need to run generic code
- Or by making sure a test is not called unless it can make a final decision
(including 'na'). Basically, this means that there must be a python module
declaring a group which the shell test can bind to using @group. The python code
then implements is_applicable() in the right way.

I think these solutions are comparable from a logic point of view. Practically,
in the second one a shell script developer needs help on the python side if there
isn't any group defined. OTOH, the first one makes the execution conditions somewhat
more complex and harder to overview, and adds a return code to the interface

There is a value in the simple model that if a test is deprecated, it won't run.
If possible, I would also stick to the original model for both python and shell
tests where a generic test is the last resort, and thus never deprecates. This
also means that a generic test always comes to some kind of conclusion.

I propose that we put this bug "on ice" until there's a usecase for the first
alternative. I'll solve your actual problem discussed on IRC by creating a perl
test module so that the test can run in group 'Perl'.

Edit: @name -> @group (!)

[http://pastebin.com/3B3L6AtC] : Rudimentary perl python test module (i. e., checks/perl.py), enough solve your shell script usecase by using @group:Perl.

At least I think so :)

At this point, I cannot see that this is a defect but rather a discussion on how to define things. Updating metadata.

I really want to avoid skipping tests accidentally. The more tests we have the easier it's going to be to overlook some tests hasn't been run at all for a long time. Generic tests also need to be deprecable (is that a word?) because for example "is latest version packaged" can be answered for some type of packages (R comes to mind).

We also use deprecations in Java tests to get rid of unneeded checks (debuginfo for example), but it is perhaps up to discussion if it's proper use-case.

I still feel that some reworking needs to be done, but not sure what exactly...and it's certainly not a problem for current plugins we ship so...deferring to future

It's perfectly OK to deprecate a generic test - the R usecase you mention is more or less the standard one. So is the java stuff; if a java test deprecates a generic one and returns 'NA', that's it.

If I understand you correctly, the problem is that a non-generic test effectively hides/disables a generic test, and that we might miss that this generic test is actually not run. This info is in the -d output, but that output is big and hard to use.

I actually don't see any simple solution. The deprecating logic looks OK to me, and works for the usecases you mention. Adding some list of deprecated tests to the report is possible, perhaps in an attachment at the end which normally is cut and not pasted. So is adding a new switch creating a clean output of deprecated tests. But I doubt that changing the core logic is the Right Thing To Do.

Problem is, I'm often wrong :)