Ticket #154 (closed enhancement: fixed)

Opened 3 years ago

Last modified 2 years ago

Tracker: critical path test case creation

Reported by: adamwill Owned by: adamwill
Priority: major Milestone: Fedora 15
Component: Wiki Version:
Keywords: Cc: lmacken, till, jlaska, johannbg, bruno, fenris02, masami, fcami, rhe, ykopkova, athmane
Blocked By: Blocking:

Description

problem

We need more specific per-package guidance for proven tester testing, to make sure the testing covers all the relevant critical path functionality of each package.

enhancement recommendation

We can create a group of test cases in the Wiki for each critical path package. Bodhi could integrate with this system and, in the future non-numeric karma system, provide checkboxes for each test case for packages which have test cases available. fedora-easy-karma could provide an interface to display and check off test cases from the Wiki.

This ticket can serve as a tracker for this process, which will be ongoing. To start with I will file tickets for each package which has so far been identified (on test and devel mailing lists) as an obvious candidate for such test cases, with a note of what should be covered for each package. Those tickets will block this one.

Attachments

critpath-src.txt (4.7 KB) - added by jlaska 3 years ago.
Critical Path SRC package names (0610)

Change History

comment:1 Changed 3 years ago by adamwill

  • Cc lmacken, till, jlaska added; lmacken till removed

comment:2 Changed 3 years ago by johannbg

  • Cc johannbg added

comment:3 Changed 3 years ago by fenris02

  • Cc fenris02 added

comment:4 Changed 3 years ago by masami

  • Cc masami added

comment:5 Changed 3 years ago by fcami

  • Cc fcami added

comment:6 Changed 3 years ago by rhe

  • Cc rhe added

comment:7 Changed 3 years ago by jlaska

  • Milestone set to Fedora 15

comment:8 Changed 3 years ago by ykopkova

  • Cc ykopkova added

comment:9 follow-up: ↓ 10 Changed 3 years ago by adamwill

here's a draft of a sample test case, for mdadm:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_testcase_mdadm_package_software

the production name would be QA:Testcase_mdadm_package_software , with QA:Testcase being our standard prefix for test cases (it's not up for debate within the context of this ticket, if we change it we must change it for all test cases), then 'mdadm' being the source package name, 'package' indicating this (that it's named based on a src.rpm name - perhaps it should come before the name rather than after?), and 'software' being the name of this particular test (referring to software RAID).

what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great. thanks.

comment:10 in reply to: ↑ 9 Changed 3 years ago by jlaska

Replying to adamwill:

here's a draft of a sample test case, for mdadm:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_testcase_mdadm_package_software

the production name would be QA:Testcase_mdadm_package_software , with QA:Testcase being our standard prefix for test cases (it's not up for debate within the context of this ticket, if we change it we must change it for all test cases), then 'mdadm' being the source package name, 'package' indicating this (that it's named based on a src.rpm name - perhaps it should come before the name rather than after?), and 'software' being the name of this particular test (referring to software RAID).

Seems sensible from a general best practices point of view (QA:Testcase_%{sourcerpm}_%{userdefined}). The "_package" seems extraneous, but I might be missing something. With regards to finding applicable tests for a certain update, the most important thing is the category structure, right? That's how other tools will locate applicable tests.

what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great.

In the page you linked, that test is in three categories ...

  1. Category:critpath_test_cases
  2. Category:mdadm_test_cases
  3. Category:mdadm_critpath_test_cases

I don't yet see the big picture for how the Categories will be grouped/organized, can you provide a hierarchical example?

comment:11 Changed 3 years ago by adamwill

the idea of the 'package' word is to clarify that the next word is a .src.rpm package name and not, well, anything else. I can see confusion arising with some .src.rpm names that are ambiguous and could be used in test case names in entirely different ways. but it's not strictly needed, no, with the categories.

"I don't yet see the big picture for how the Categories will be grouped/organized, can you provide a hierarchical example? "

that's one of the areas I'm not feeling totally sure of yet, but the idea of the organization in the example ticket is that *any* test case intended to aid critpath testing of any critpath package will be in 'critpath_test_cases', any test case relating to mdadm - even if it's not testing critpath functionality - will be in mdadm_test_cases, and just tests that are for mdadm and are for testing critpath functionality will be in mdadm_critpath_test_cases . The third set is the combination of the first two, basically. I suppose you can search for pages that are in both of two categories and then you wouldn't need the third set.

again, this is one i don't think is right yet, so better suggestions for doing the categorization are welcome.

comment:12 Changed 3 years ago by jlaska

Awesome, I like how you've focused on writing the test case *once*, and distinguishing the content from how it's grouped/organized. That distinction will help us with a possible future TCMS migration.

We discussed on IRC, but following up here for a larger audience.... I wasn't sure if the 3 groups you listed (critpath_test_cases, mdadm_test_cases, mdadm_critpath_test_cases) were all needed. Certainly they help for convenience, but it sounds like the consumers of this hierarchy would only need to find

  1. all mdadm test cases (Category:mdadm_test_cases)

... OR ...

  1. All critical path mdadm test cases (intersection between Category:mdadm_test_cases and Category:Critpath_test_cases).

I didn't initially think the intersection would be challenging to do. But thinking ahead to some remote API that consumers would use to find applicable tests, we may want to use the three categories as you suggest to avoid having each consumer defining logic for how it finds applicable tests. So in your example, we would treat Category:Critpath_test_cases as a way to denote test priority (where high == critpath, media, low). Category:mdadm_test_cases would be used to indicate membership to a test plan (the mdadm test plan). And finally, Category:Critpath_mdadm_test_cases would be our way to provide all *high* priority tests in the mdadm test plan. Does that make sense?

I know I'm getting too far ahead, but I'm trying to articulate our wiki organization in TCMS terms, and to reduce the requests consumers would have down to a simple API.

Using the 3 category suggestion, my ASCII-art attempt at a top-down view from Category:Test_cases would be ...

Category:Test_Cases
  |
  +-> Category:mdadm_test_cases
  |   |
  |   +-> Category:Critpath_mdadm_test_cases (e.g. high priority tests)
  |   |   |
  |   |   +-> QA:Testcase_mdadm_important_use_case_1
  |   |   +-> QA:Testcase_mdadm_important_use_case_2
  |   |   +-> QA:Testcase_mdadm_important_use_case_3
  |   |
  |   +-> QA:Testcase_mdadm_obscure_use_case_4
  |   +-> QA:Testcase_mdadm_obscure_use_case_5
  |   +-> QA:Testcase_mdadm_obscure_use_case_6
  |
  +-> Category:kernel_test_cases
  |   |
  |   +-> Category:Critpath_kernel_test_cases (e.g. high priority tests)
  |       |
  |       +-> ...
  .
  .
  .

And then the same view, but for just Category:Critpath_test_cases. This is probably the easiest thing for consumers of this data to manage.

Category:Critpath_Test_Cases
  |
  +-> Category:Critpath_mdadm_test_cases (e.g. high priority tests)
  |   |
  |   +-> QA:Testcase_mdadm_important_use_case_1
  |   +-> QA:Testcase_mdadm_important_use_case_2
  |   +-> QA:Testcase_mdadm_important_use_case_3
  +-> Category:Critpath_kernel_test_cases (e.g. high priority tests)
  |   |
  |   +-> QA:Testcase_kernel_important_use_case_1
  |   +-> QA:Testcase_kernel_important_use_case_2
  |   +-> ...
  .
  .
  .

While it's an extra category for each src.rpm, this seems like a really sane approach.

comment:13 follow-up: ↓ 14 Changed 3 years ago by adamwill

"I didn't initially think the intersection would be challenging to do. But thinking ahead to some remote API that consumers would use to find applicable tests, we may want to use the three categories as you suggest to avoid having each consumer defining logic for how it finds applicable tests. So in your example, we would treat Category:Critpath_test_cases as a way to denote test priority (where high == critpath, media, low). Category:mdadm_test_cases would be used to indicate membership to a test plan (the mdadm test plan). And finally, Category:Critpath_mdadm_test_cases would be our way to provide all *high* priority tests in the mdadm test plan. Does that make sense? "

Honestly, not really, no. =) The thinking behind creating the technically-not-needed combined category was 90% 'half-assing it as I was going along' and 10% for *non*-tool based cases: people just browsing the tests. It's easy via the API for tools to determine which test cases are members of both categories, but it's not easy for a user just browsing the Wiki to do it, AFAIK - there's no easy way to browse pages that are members of both one category and another category in the mediawiki user interface, that I know of.

But really, I'm not married to the idea. (That's why I like it, ba-dum tish. Tip your server.) I don't really see any problem with having tools derive the combination themselves - it's not like they'll all be implementing it in wildly different ways, I can't see how that would happen. I'm not sure manual tending of a combined category could produce results that would be different from doing it programmatically in any way other than the manual category occasionally getting screwed up, which *will* happen if we put squishy pink things in the loop.

comment:14 in reply to: ↑ 13 Changed 3 years ago by jlaska

Replying to adamwill:

Honestly, not really, no. =) The thinking behind creating the technically-not-needed combined category was 90% 'half-assing it as I was going along' and 10% for *non*-tool based cases: people just browsing the tests. It's easy via the API for tools to determine which test cases are members of both categories, but it's not easy for a user just browsing the Wiki to do it, AFAIK - there's no easy way to browse pages that are members of both one category and another category in the mediawiki user interface, that I know of.

Correct, not without http://www.mediawiki.org/wiki/Extension:Multi-Category_Search

But really, I'm not married to the idea. (That's why I like it, ba-dum tish. Tip your server.) I don't really see any problem with having tools derive the combination themselves - it's not like they'll all be implementing it in wildly different ways, I can't see how that would happen.

I can and have, I'd like to stay away from that. We're talking about a *small* set of tools at this point so the fallout would be minimal and manageable, agreed. My main point is, let's do our best to keep test cases and related test case metadata within the realm of TCMS (whether it's wiki or nitrate).

comment:15 Changed 3 years ago by adamwill

Fair enough. I guess the best approach would be to have it automatically generated 'server'-side - i.e. wherever we're storing the test cases, i.e., right now, the Wiki. I don't know if mediawiki can do that, but if we're looking at the Glorious Future where we have a TCMS, presumably the TCMS could do it?

comment:16 follow-up: ↓ 18 Changed 3 years ago by adamwill

sigh, I keep getting ready to send this out to lists and then hitting one more snag. =)

so, in discussion with jlaska, we both think it makes most sense to only create two new types of category: one for associating test cases with specific packages, one for marking them as critical path test cases.

Critical path test cases should be in the category Critical_path_Test_Cases . Note that this means *any test case which is primarily concerned with verifying critical path test cases* should be in this category. It doesn't mean any test case associated with a critical path package should be in the category, because critical path packages can have non-critical path uses and hence non-critical path test cases: a test case to test, say, tape drive functionality in the kernel is associated with a critical path package (the kernel) but is not a critical path test.

The category name for specific source packages should be:

Package_(srcrpmname)_Test_Cases

same reasoning as my initial proposal for the test case name: we need to specify that (srcrpmname), in this context, is specifically a source RPM name. I can imagine, for instance, we might want to create a generic category for Python test packages, called Category:Python_Test_Cases, and put test cases into that category which are not actually test cases for the Python .src.rpm. So we'd also have a Category:Package_python_Test_Cases . That could probably be a member of Category:Python_Test_Cases , but not vice versa.

I think the page name of each test case can be a bit more flexible and we don't really need to worry about defining it, the categories are the crucial thing.

Do we also want to suggest a category for each .src.rpm along the lines of:

Category:Package_python_core_Test_Cases

? Rationale: I'm thinking about how we want the bodhi or fedora-easy-karma listing for each package to look. I'm thinking of something like:


Update: python-2.7.9-blahblah

This update does blah blah foo foo whee whee poop.

The following test cases can be used to check the functionality of this package.

Critical path tests

These tests verify critical path functionality (link to critical path process page) and should take highest priority:

  • links to all tests in Category:Package_python_Test_Cases *and* Category:Critical_path_Test_Cases

Core functionality tests

These tests verify core functionality of the package and should take highest priority after critical path tests:

  • links to all tests in Category:Package_python_core_Test_Cases (but not Category:Critical_path_Test_Cases)

Remaining tests

These tests verify non-critical path, non-core functionality:

  • links to all other tests (in Category:Package_python_Test_Cases but not Category:Package_python_core_Test_Cases or Category:Critical_path_Test_Cases)


The thinking being that some packages will grow fairly large sets of tests (abrt already has, for instance) and it would help to isolate core functionality tests for rapid sanity checking.

random side observation: our naming of test case pages and categories currently stinks.

next big concern: where do we effectively document this? All we currently have on test cases is:

https://fedoraproject.org/wiki/Category:Test_Cases

I can add some overview and stuff to that page but it's not a terribly visible page, really. I guess the best approach is to edit that page and add some links to it from strategic other places, especially developer instruction pages. Thoughts?

comment:17 Changed 3 years ago by adamwill

looking through existing test cases I found that abrt already has a good set of test cases we can use as seeds and examples:

https://fedoraproject.org/wiki/Category:ABRT_Test_Cases

we just need to adjust them to the planned system.

I also found an interesting illustration of the point about using the package_ descriptor - we have a set of 'bootchart test cases':

https://fedoraproject.org/wiki/Category:Bootchart_Test_Cases

but these aren't test cases for checking the functionality of the bootchart srpm; they're various boot scenarios that you time *using bootchart*. the idea isn't to test bootchart, but to use bootchart to test boot speed...

comment:18 in reply to: ↑ 16 Changed 3 years ago by jlaska

Replying to adamwill:

The category name for specific source packages should be:

Package_(srcrpmname)_Test_Cases

That works fine. I think the "Package_" prefix is not needed, but it certainly doesn't hurt anything to have it (just longer wiki page names "Category:Package_mobile-broadband-provider-info_Test_Cases" vs "Category:Mobile-broadband-provider-info_Test_Cases", but that's certainly not horrible).

same reasoning as my initial proposal for the test case name: we need to specify that (srcrpmname), in this context, is specifically a source RPM name. I can imagine, for instance, we might want to create a generic category for Python test packages, called Category:Python_Test_Cases, and put test cases into that category which are not actually test cases for the Python .src.rpm. So we'd also have a Category:Package_python_Test_Cases . That could probably be a member of Category:Python_Test_Cases , but not vice versa.

I think the page name of each test case can be a bit more flexible and we don't really need to worry about defining it, the categories are the crucial thing.

Definitely!

Do we also want to suggest a category for each .src.rpm along the lines of:

Category:Package_python_core_Test_Cases

? Rationale: I'm thinking about how we want the bodhi or fedora-easy-karma listing for each package to look. I'm thinking of something like:


Update: python-2.7.9-blahblah

This update does blah blah foo foo whee whee poop.

The following test cases can be used to check the functionality of this package.

Critical path tests

These tests verify critical path functionality (link to critical path process page) and should take highest priority:

  • links to all tests in Category:Package_python_Test_Cases *and* Category:Critical_path_Test_Cases

Core functionality tests

These tests verify core functionality of the package and should take highest priority after critical path tests:

  • links to all tests in Category:Package_python_core_Test_Cases (but not Category:Critical_path_Test_Cases)

Remaining tests

These tests verify non-critical path, non-core functionality:

  • links to all other tests (in Category:Package_python_Test_Cases but not Category:Package_python_core_Test_Cases or Category:Critical_path_Test_Cases)


The thinking being that some packages will grow fairly large sets of tests (abrt already has, for instance) and it would help to isolate core functionality tests for rapid sanity checking.

The way I read your organization above is that the front-ends will present the tests by priority (critical path, core, *). I really like this visualization, but wonder if it is outside the scope for the first pass. Meaning, I think I'd want to see more focus on just getting critpath tests defined for as many critpath packages, rather than spending time on non-critpath or non-core stuff. With the setup you've designed, it doesn't exclude people from creating non-critpath tests.

An aside, I think the previously discussed Category: organization would lend better to allowing maintainers to prioritize tests themselves. For example ...

bodhi and f-e-k will look for wiki pages under Category:Package_%{sourcerpm}_Test_cases. Any tests in that category would be listed, any tests in sub-categories of that page will be presented as you demonstrate above. So to achieve your desired output, you might organize tests like ...

Category:Package_python_Test_Cases
|
+--> Category:Critpath_python_Test_Cases
|    |
|    +-> QA:Testcase_python_do_something_1 [1]
|    +-> QA:Testcase_python_do_something_2 [1]
|    +-> QA:Testcase_python_do_something_3 [1]
|
+--> Category:Core_python_Test_Cases
|    |
|    +-> QA:Testcase_python_do_something_4
|    +-> QA:Testcase_python_do_something_5
|
+--> Category:Low_python_Test_Cases
|    |
|    +-> QA:Testcase_python_do_something_6
|    +-> QA:Testcase_python_do_something_7
|
|--> QA:Testcase_python_do_something_8
|--> QA:Testcase_python_do_something_9

[1] Could also exist in Category:Critpath_test_cases, but not required

This would allow maintainers to group the tests as they see fit, but holds that everything must be anchored out of Category:Package_python_Test_Cases. We would still focus on prioritizing development of critpath tests, but the support exists for maintainers/packagers to organize as they feel is appropriate. Also, the mediawiki API provides the needed methods to find and present the information listed above.

Another something to keep in mind, any heavy/complex wiki organization we do may require extra work when migrating data to another system (which is a possible mid-term future scenario).

random side observation: our naming of test case pages and categories currently stinks.

Sure, page naming wasn't as important when we weren't attempting to integrate it with other tools so I gather some wiki renaming may be required. Or did it stink in another regard?

next big concern: where do we effectively document this? All we currently have on test cases is:

https://fedoraproject.org/wiki/Category:Test_Cases

I can add some overview and stuff to that page but it's not a terribly visible page, really. I guess the best approach is to edit that page and add some links to it from strategic other places, especially developer instruction pages. Thoughts?

Agreed. Do we need an SOP/HOWTO for each of the use cases where proventesters, or maintainers, will interact with this setup? Such as ...

  1. How to create a test case
  2. How to organize test cases
  3. How to post results for test cases
  4. ... ?

comment:19 Changed 3 years ago by adamwill

Thanks. I want to document the non-critpath usage of this system because I think it'll happen quite fast (in fact, we already have a set of abrt tests that should be converted to use the new system, and abrt isn't critpath...) and if we don't have the usage planned out ahead of time, people will do it however feels right to them and we'll wind up with a mess.

I think I've mostly got everything straight in my head now; I'm going to start drafting up changes to Wiki pages and linking to them here.

comment:20 Changed 3 years ago by adamwill

here's a draft of a test case creation SOP:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_test_case_creation

it's not really anything specific to this ticket, but it is a prerequisite. I plan to write a (short) SOP on package-specific test cases, linking to this more generic SOP, and then edit the test case category page to link to both (and possibly the test case template page), and then edit strategic pages in the packaging instructions section to point out to these pages. sound good?

comment:22 Changed 3 years ago by adamwill

I tweaked https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation somewhat and https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_test_case_creation a small amount. I think they're ready to go out for review now. I'll mail the lists for feedback.

comment:23 Changed 3 years ago by adamwill

I converted over one package's test cases to serve as an example of the new system: the test cases for the Radeon driver, xorg-x11-drv-ati . This was a good example as there's a lot of them, and they split neatly into critpath, core and extended to illustrate the optional advanced categorization system.

So, see:

https://fedoraproject.org/wiki/Category:Package_xorg-x11-drv-ati_test_cases

and note that one is also in:

https://fedoraproject.org/wiki/Category:Critical_path_test_cases

comment:24 Changed 3 years ago by fcami

Thank you Adam and James! This is excellent work. Two comments: The extended test cases (at least, the 3D ones) should be run on Intel also, and maybe on Nouveau (do we support 3D there?), so we might need a way to share these across drivers. We also probably need to define a list of hardware we want the tests executed on, as a lot of the bugs we encounter are hardware-dependent, hitting different paths in the radeon/mesa stack. Any idea how to handle this?

comment:25 Changed 3 years ago by adamwill

fcami: we have similar test cases for Intel and Nouveau already. I have not converted those to the new scheme because it's just a draft at present; I converted the Radeon test cases simply to serve as an example.

any elaboration of the Radeon test cases is really a separate topic from this process. The Radeon test cases themselves are not particularly significant to the package-specific test case process, they're merely examples. Please discuss things specific to the Radeon test cases elsewhere (file a new ticket, or mail the list). thanks!

comment:26 Changed 3 years ago by fcami

Sorry, I should have used more generic terms in my explanation. Let me try again :)

  1. we should share test cases between all supported drivers and not duplicate them - because otherwise, editing a test case relevant to multiple hardware drivers means editing multiple pages.
  1. how do we define success/failure (for a go/nogo meeting for instance) when some people manage to run the test cases and others don't, depending on hardware?

Note that 2. is probably more a "see what it's bugzilla" than a "see the test case results" question, so it's moot.

comment:27 Changed 3 years ago by adamwill

again, neither of those is really relevant to this effort. I really want to keep this ticket focused on the project it's for: a way to organize test cases for specific packages. Please can we discuss your thoughts in another ticket or on the mailing list? Thanks.

1 does suggest one point I've thought of in relation to this process but not spelled out, I suppose - there's no reason we can't write test cases to apply to more than one package (in this case, multiple video drivers) and add them to the categories for multiple packages. That's certainly acceptable under the scheme, yes.

comment:28 follow-up: ↓ 29 Changed 3 years ago by bruno

  • Cc bruno added

I have tried to create a test for squashfs-tools at: https://fedoraproject.org/wiki/QA:Testcase_squashfs-tools_compression

Does it look reasonable? Most of it is running a test script and making sure it doesn't print any "failed" messages.

comment:29 in reply to: ↑ 28 Changed 3 years ago by jlaska

Replying to bruno:

I have tried to create a test for squashfs-tools at: https://fedoraproject.org/wiki/QA:Testcase_squashfs-tools_compression

Does it look reasonable? Most of it is running a test script and making sure it doesn't print any "failed" messages.

Ooh, nothing speaks better than examples! :) Some comments...

  1. Bruno, where is upstream for this test script? Does it live anywhere else? If not, I'd recommend uploading that test script to the wiki as a file. Either way, you might want to link to it for download, rather than including it inline. I've made a few changes along these lines, you just need to upload and link to the script now.
  2. I suspect you've done so already, but a determination whether this is critpath for squashfs-tools, or core. This would impact which Category you use.
  3. I'm not super familiar with squashfs-tools, so perhaps a bit more wording in the description on how this test fits into the big picture for squashfs-tools? I've re-organized some of your content along these lines, feel free to adjust as needed.
  4. Perhaps include some examples of good test results, and/or an example of bad test results in the results section?

As noted above, I made a few adjustments to the test case. Of course, feel free to adjust/revert as you see fit.

comment:30 Changed 3 years ago by bruno

There is no upstream. I needed to make one so that I would have a reasonable test before putting a development version of squashfs-tools in rawhide. (Kyle asked me not to break things :-) I'll take a look at doing the upload and if I think it is as simple as it looks at first glance I'll do it shortly.

This is mostly crit path. squashfs-tools is used in live images. If uncompressing doesn't work then the images won't work. Arguably only gzip needs to work now, but hopefully we'll also need xz for F15. Also the unsquashfs path isn't used by live images, so in theory that could be skipped as well. But it seemed convenient to test these together. If it runs too long, the test data size could be cut back and the test would probably be as good. In my tests it runs reasonably quick on an older machine.

In theory there could be other core tests to make sure extended attributes and special files work correctly. But those features aren't really used by live images currently.

A more complete test, would be to test live images, but there could be other things blocking a live image test and they would take longer to do.

I can add a good results example. The only bad results I saw were when I had bugs in my test script.

comment:31 follow-up: ↓ 32 Changed 3 years ago by bruno

I fixed up the test case page for squashfs-tools.

One other note about this is that in the past I think testing of squashfs-tools by proventesters was slow because there wasn't a test case and people didn't know what to test. Eventually nirik ended up doing when I asked him about it. Hopefully this won't be as big of an issue in the future.

comment:32 in reply to: ↑ 31 Changed 3 years ago by jlaska

Replying to bruno:

I fixed up the test case page for squashfs-tools.

Nice, looks good to me!

One other note about this is that in the past I think testing of squashfs-tools by proventesters was slow because there wasn't a test case and people didn't know what to test. Eventually nirik ended up doing when I asked him about it. Hopefully this won't be as big of an issue in the future.

That's one of the big reasons Adam is proposing the wiki structure discussed in this ticket, so that we now have a place for people to document repeatable test procedures, and a consistent way to find them. Yay, thank you Adam!

comment:33 Changed 3 years ago by adamwill

I love it when a plan comes together!

Thanks for the test case Bruno. It looks like you should add it to the category Category:Package_squashfs-tools_test_cases and add that category to Category:Test_Cases . Did you not find that the instructions made this clear enough? If not, I'll have to revise them...

comment:34 Changed 3 years ago by bruno

I think when I first read them I wasn't sure it was required. And since I just had the one test case, I wasn't sure if it was worthwhile. I might have also seen an early version of the instructions as they were being worked on at least somewhat after I did the test case. I'll take care of the category change.

comment:35 Changed 3 years ago by bruno

I reviewed the SOP again and I think it should stress making a package category for each package more strongly. Also it should mention the trick used to get the package in the right alphabetical position instead of having everything end up under P.

comment:36 Changed 3 years ago by adamwill

thanks for the feedback, I'll update the page soon. yes, you should create a category even for just one test case, because it is still useful to help both people and programs locate it. If there isn't a category, tools like f-e-k and bodhi will not have a way to figure out that the test case should be associated with the package.

comment:38 Changed 3 years ago by bruno

That looks good. I found the real problem in my case, is that I did not realize that there were two documents and I was going mostly off the test case document, rather than the test plan document. I think I ran across both at various times, but when actually writing up the test case I was looking at the test case document. There is a link back to the test plan document, but I didn't check at that time.

I still like the changes to the test plan document. The sort trick I only knew about from the conversation about the radeon test cases. I'd be less likely to have made the mistake of not having a package category if I had been referring to both pages at the time I was actually making up the test case, but with the current wording I would have been sure to have made the package category.

comment:39 Changed 3 years ago by jlaska

I posted a python script (multi-cat-search.py) that can be used to demonstrate using the mediawiki API to find tests that apply for a particular source rpm name. See the --help for the command, it offers several options, including -ccritpath to limit results to only critical path tests.

I'm happy to adjust the script as needed by f-e-k or bodhi. This was just a proof-of-concept to demonstrate sifting through wiki using the API and user-input.

comment:40 Changed 3 years ago by jlaska

I think we can put a fork in the initial proof-of-concept of this ticket. Is there anything else you wanted to track here before closing?

comment:41 follow-up: ↓ 43 Changed 3 years ago by adamwill

well, the ticket actually got sidetracked: if you read the initial post, the intention was to use it to track the creation of the test cases themselves, not the process. So we could still use it for that.

comment:42 Changed 3 years ago by athmane

  • Cc athmane added

comment:43 in reply to: ↑ 41 Changed 3 years ago by jlaska

Replying to adamwill:

well, the ticket actually got sidetracked: if you read the initial post, the intention was to use it to track the creation of the test cases themselves, not the process.

How goes the progress? What's remaining?

So we could still use it for that.

It's too much for a single ticket, imo. But you choose ... I'm just cleaning up *stale* tickets.

comment:44 Changed 3 years ago by jlaska

Assuming I'm using my own script correctly (get-mediawiki-data), we have the following packages with wiki test cases.

Category:Package abrt test cases
Category:Package anaconda test cases
Category:Package biosdevname test cases
Category:Package clamav test cases
Category:Package Django test cases
Category:Package evince test cases
Category:Package firefox test cases
Category:Package gnome-color-manager test cases
Category:Package gnome-disk-utility test cases
Category:Package gammu test cases
Category:Package gnome-shell test cases
Category:Package httping test cases
Category:Package imsettings test cases
Category:Package ibus test cases
Category:Package mysql test cases
Category:Package mysql-libs test cases
Category:Package mysql-server test cases
Category:Package NetworkManager test cases
Category:Package nmap test cases
Category:Package nikto test cases
Category:Package openvas-scanner test cases
Category:Package openvas-libraries test cases
Category:Package openvas-client test cases
Category:Package preupgrade test cases
Category:Package report test cases
Category:Package rkhunter test cases
Category:Package squashfs-tools test cases
Category:Package smolt test cases
Category:Package wireshark test cases
Category:Package xorg-x11-drv-ati test cases
Category:Package xorg-x11-drv-nouveau test cases
Category:Package xorg-x11-drv-intel test cases

Combining the list above, with a recent critpath log, the following critpath packages have wiki test cases:

anaconda
NetworkManager
report
squashfs-tools
xorg-x11-drv-ati
xorg-x11-drv-nouveau
xorg-x11-drv-intel

comment:45 Changed 3 years ago by athmane

Just created new test case for openssh (=> critpath):

https://fedoraproject.org/wiki/QA:Testcase_OpenSSH

comment:46 Changed 3 years ago by athmane

I've categorized and tested two test-cases for cronie and perl (they are on critpath category) so they'll show up in Bodhi updates.

https://fedoraproject.org/wiki/QA:Testcase_Perl_sanity

http://fedoraproject.org/wiki/QA:Testcase_crontabs

comment:47 Changed 3 years ago by athmane

comment:49 Changed 3 years ago by athmane

I've just created a test-case for rsyslog (=> critpath):

https://fedoraproject.org/wiki/QA:Testcase_rsyslog

comment:50 Changed 3 years ago by jlaska

Athmane pointed out that the linked critpath.log shows binary packages, not src.rpms. I've attached a src.rpm list to this ticket for review. The list is taken form the 0610 rawhide critpath.log.

Changed 3 years ago by jlaska

Critical Path SRC package names (0610)

comment:51 Changed 3 years ago by jlaska

As requested during QA meeting, the following 13 critical path packages have wiki tests. There are a total of 432 src critical path packages.

anaconda
cronie
dracut
gnome-disk-utility
NetworkManager
perl
report
rsyslog
squashfs-tools
xorg-x11-drv-ati
xorg-x11-drv-nouveau
xorg-x11-drv-intel
firstboot

How do others feel about narrowing the scope of this ticket to a more attainable short-term goal? Perhaps all user-facing Desktop packages?

comment:52 Changed 3 years ago by athmane

I've just created a test-case for yum (=> critpath):

https://fedoraproject.org/wiki/QA:Testcase_Yum_basics

comment:53 Changed 3 years ago by athmane

I've just created a test-case for sendmail (=> critpath):

https://fedoraproject.org/wiki/QA:Testcase_Sendmail

comment:54 Changed 2 years ago by adamwill

  • Resolution set to fixed
  • Status changed from new to closed

Looking at this again, I've had a change of heart and I agree with Past James and not Past Adam: a single ticket for 'create test cases for 400+ critpath components' isn't much use, especially given trac doesn't let you have tickets depend on each other. So let's close it. That doesn't mean we should stop working on critpath test cases, though, this is just bureaucracy. :) Thanks athmane for your work so far on writing and categorizing test cases!

Note: See TracTickets for help on using tickets.