Ticket #72 (closed enhancement: fixed)

Opened 4 years ago

Last modified 4 years ago

Proposal for reducing test permutations

Reported by: rhe Owned by: rhe
Priority: major Milestone: Fedora 14
Component: Test Review Version:
Keywords: Cc: jlaska,kparal,robatino
Blocked By: Blocking:

Description

problem

So far we have similar test cases for a same test area, and it's not quite useful to test all of them.

analysis

Take RAID cases for example, as James said:"We aren't hitting any problems anymore that are specific to RAID0 but work on RAID5. The main focus for this tests is to make sure that we identify a real-world RAID install partitioning setup, anaconda can execute that partition scheme, and dracut can boot it."

enhancement recommendation

I would like to propose for removing the following cases and keeping a representative one instead:

RAID:

(Replace them with this case: QA:Testcase_Install_to_Hardware_RAID)

Preupgrade:


FTP Install:

(keep the NonAnonymous case)

If you have any thought about it, please feel free to tell, thanks.

Change History

comment:1 follow-up: ↓ 2 Changed 4 years ago by jlaska

I love this idea, thanks for proposing!

Regarding the RAID tests, does it make sense to keep a representative copy for each of the 3 main RAID types?

  1. Hardware RAID - QA:Testcase_Install_to_Hardware_RAID (as you've noted above)
  2. Software RAID - some autopart-like partition scheme using Software RAID devices?
  3. BIOS RAID - QA:Testcase_Install_to_BIOS_RAID

comment:2 in reply to: ↑ 1 ; follow-ups: ↓ 3 ↓ 4 Changed 4 years ago by rhe

Replying to jlaska:

I love this idea, thanks for proposing!

Regarding the RAID tests, does it make sense to keep a representative copy for each of the 3 main RAID types?

Of course, thanks for clarifying them! My plan was to keep Hardware RAID and BIOS RAID case, but yes, SW RAID one should be added in the matrix.

  1. Hardware RAID - QA:Testcase_Install_to_Hardware_RAID (as you've noted above)
  2. Software RAID - some autopart-like partition scheme using Software RAID devices?

I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?

  1. BIOS RAID - QA:Testcase_Install_to_BIOS_RAID

comment:3 in reply to: ↑ 2 Changed 4 years ago by rhe

Replying to rhe:

Replying to jlaska: <skip>

  1. Software RAID - some autopart-like partition scheme using Software RAID devices?

I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?

I've created a software RAID case based on the steps in QA:Testcase_Partitioning_Rootfs_On_RAID6:

In this case, root(/) is partitioned on a RAID device, please review it. Thanks!

comment:4 in reply to: ↑ 2 Changed 4 years ago by jlaska

Replying to rhe:

  1. Software RAID - some autopart-like partition scheme using Software RAID devices?

I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?

Oh yes, sorry for not clarifying. I can't think of a critical reason why we would want LVM involved for this scenario. It's certainly a valid test. It's possible there would be problems executing a RAID+LVM partition scheme, or with finding the '/' volume while booting the installed system. If desired, we have a test for LVM on RAID (see QA:Testcase_Partitioning_LVM_LV_on_RAID_PV). However, I don't have strong feelings here. I trust your judgment.

I've created a software RAID case based on the steps in QA:Testcase_Partitioning_Rootfs_On_RAID6:

In this case, root(/) is partitioned on a RAID device, please review it. Thanks!

Looks good, thanks Hurry! Yay, no more running 6 different software RAID installs :)

comment:5 in reply to: ↑ description Changed 4 years ago by rhe

Except the above raid tests, I plan to remove the following tests:

  1. Preupgrade:

Reason: In real world, users upgrade directly to the release after the next should be very few, and no release criteria is defined on this.

  1. Repository:

Reason: After booting with a media, the CD-ROM would be locked, therefore, it's normally impossible to exchange the CD/DVD to add another repo. InstallSourceDvd and InstallSourceCdrom? test installation using DVD/CD package repo.

  1. Package Install - Minimal:

Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it.

  1. Partitioning - Ext2 & UninitializedDisks?

Reason: How many people will install using ext2 filesystem?!

Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test.

What do you think? Feel free to comment if you have any ideas/suggestions/disagreements about these. Thanks.

comment:6 follow-ups: ↓ 7 ↓ 8 Changed 4 years ago by adamwill

"Reason: In real world, users upgrade directly to the release after the next should be very few"

This is not actually the case. Quite a lot of people run alternate releases, because that's the longest possible refresh cycle allowed by our maintenance policies. People who don't want to go through the hassle of doing a distro upgrade every six months run alternate releases, so they only have to do it every twelve months.

"and no release criteria is defined on this"

Indeed this is true, but I believe this test exists because we recognize that, in practice, people do run alternate releases so we should at least know about and be able to document any issues with this.

"Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it."

I don't quite understand this. We test the minimal install functionality to make sure it works. Sure, all the same packages are in the default install, but that's not the same as saying the minimal installation function works as intended. Am I misreading you?

"Reason: How many people will install using ext2 filesystem?! "

Agreed, this one is outdated now. (Some people use ext2 on SSDs because they think journalling will kill the hardware, but such people are wrong. :>)

"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. "

This seems reasonable.

comment:7 in reply to: ↑ 6 ; follow-up: ↓ 9 Changed 4 years ago by jlaska

Replying to adamwill:

"Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it."

I don't quite understand this. We test the minimal install functionality to make sure it works. Sure, all the same packages are in the default install, but that's not the same as saying the minimal installation function works as intended. Am I misreading you?

Hurry: I think we'll need to keep the minimal install test around for Fedora 14. However, I don't think we need to run this test for *every* usage scenario. Can both the default package set, and the minimal package set tests be moved into the general variations list? I don't think these need to be tested against each boot+install method.

Additionally, I've updated the minimal install package set test to ensure the system is able to access the network and install yum updates.

"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. "

I'd really like to also remove 'QA/TestCases/BootMethodsKVM' and it's XEN counterpart, or at least rename them to something more appropriate. Perhaps a set of storage device tests that confirm the detection and use of virt storage and network devices?

comment:8 in reply to: ↑ 6 Changed 4 years ago by rhe

Replying to adamwill:

"Reason: In real world, users upgrade directly to the release after the next should be very few"

This is not actually the case. Quite a lot of people run alternate releases, because that's the longest possible refresh cycle allowed by our maintenance policies. People who don't want to go through the hassle of doing a distro upgrade every six months run alternate releases, so they only have to do it every twelve months.

"and no release criteria is defined on this"

Indeed this is true, but I believe this test exists because we recognize that, in practice, people do run alternate releases so we should at least know about and be able to document any issues with this.

I see. If quite many people run alternative releases, I agree to keep this test, but then I hope the release criteria can reflect the priority of it rather than no priority like that in F13.

"Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it."

I don't quite understand this. We test the minimal install functionality to make sure it works. Sure, all the same packages are in the default install, but that's not the same as saying the minimal installation function works as intended. Am I misreading you?

As you said, my thought was that all the packages are the same in the default install, minimal one can be removed. But you're right, that doesn't mean the minimal install works, especially when James updated this test.

"Reason: How many people will install using ext2 filesystem?! "

Agreed, this one is outdated now. (Some people use ext2 on SSDs because they think journalling will kill the hardware, but such people are wrong. :>)

"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. "

This seems reasonable.

Many thanks for your comments Which make things more clear. :)

comment:9 in reply to: ↑ 7 Changed 4 years ago by rhe

Replying to jlaska:

<skip>

Hurry: I think we'll need to keep the minimal install test around for Fedora 14. However, I don't think we need to run this test for *every* usage scenario. Can both the default package set, and the minimal package set tests be moved into the general variations list? I don't think these need to be tested against each boot+install method.

No problem. Having default package set test in each scenario is easier for testers to add results. I don't think it's specific to any usage scenario as well, so I'll put it in general list.

Additionally, I've updated the minimal install package set test to ensure the system is able to access the network and install yum updates.

Thanks to both you and Adam for clarifying this issue, let's keep minimal install test.

"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. "

I'd really like to also remove 'QA/TestCases/BootMethodsKVM' and it's XEN counterpart, or at least rename them to something more appropriate. Perhaps a set of storage device tests that confirm the detection and use of virt storage and network devices?

I'm going to remove them and add a case QA:Testcase_Install_to_KVM instead.Thanks.

comment:10 follow-up: ↓ 11 Changed 4 years ago by rhe

I further updated the matrix, and I'm not sure if the following tests can be removed or not:

Reason: Not commonly used and require special device to support it.

Reason: I don't think No Swap should be separated as a test.

Reason: Is it a usual way to load kickstart file? I seldom use this way.

Reason: QA:Testcase_Preupgrade_from_older_release is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading.

What do you think? Please feel free to add comments and tell me if there are other tests in the removing list.

comment:11 in reply to: ↑ 10 ; follow-up: ↓ 12 Changed 4 years ago by jlaska

Apologies, I missed your latest updates in this ticket.

Replying to rhe:

I further updated the matrix, and I'm not sure if the following tests can be removed or not:

Reason: Not commonly used and require special device to support it.

I'd like to remove this test, but I'm afraid we need to ensure this is functional since it is included in the Beta release criteria (https://fedoraproject.org/wiki/Fedora_14_Beta_Release_Criteria). I've relied on feedback from pjones in previous releases to run this test. Mainly because running this test is still a mystery to me. I agree the conditions around this test are not ideal, but I don't have any great ideas for improving this right now :(

Reason: I don't think No Swap should be separated as a test.

This test still exists to target the dialog warning that swap is recommended. What do you recommend, should this test be absorbed into another test?

Reason: Is it a usual way to load kickstart file? I seldom use this way.

This is how preupgrade and snake delivery kickstarts. They open up the initrd.img and insert the ks.cfg there. We'll need to continue supporting this.

Reason: QA:Testcase_Preupgrade_from_older_release is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading.

Good point, this should be less of an issue preupgrading from F13 -> F14. The remaining preupgrade test is written to mimic a real-world upgrade scenario, so we should capture any /boot size issues as they surface.

What do you think? Please feel free to add comments and tell me if there are other tests in the removing list.

I can't think of any others at this time. Thanks!

comment:12 in reply to: ↑ 11 Changed 4 years ago by rhe

  • Resolution set to fixed
  • Status changed from new to closed

Replying to jlaska:

Apologies, I missed your latest updates in this ticket.

No worries.

Replying to rhe:

I further updated the matrix, and I'm not sure if the following tests can be removed or not:

Reason: Not commonly used and require special device to support it.

I'd like to remove this test, but I'm afraid we need to ensure this is functional since it is included in the Beta release criteria (https://fedoraproject.org/wiki/Fedora_14_Beta_Release_Criteria). I've relied on feedback from pjones in previous releases to run this test. Mainly because running this test is still a mystery to me. I agree the conditions around this test are not ideal, but I don't have any great ideas for improving this right now :(

I see, since it's included in beta criteria, let's keep this test.

Reason: I don't think No Swap should be separated as a test.

This test still exists to target the dialog warning that swap is recommended. What do you recommend, should this test be absorbed into another test?

Hmm, I think it depends on the importance of no swap partitioning. A possible way is to absorb this into the expected results of ext4 and ext3 partitioning tests, saying "If no swap partition is given, a warning dialogue will pop up...", but then no swap is not guaranteed to be tested. So, I think just keeping this test is fine.

Reason: Is it a usual way to load kickstart file? I seldom use this way.

This is how preupgrade and snake delivery kickstarts. They open up the initrd.img and insert the ks.cfg there. We'll need to continue supporting this.

I understand now.

Reason: QA:Testcase_Preupgrade_from_older_release is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading.

Good point, this should be less of an issue preupgrading from F13 -> F14. The remaining preupgrade test is written to mimic a real-world upgrade scenario, so we should capture any /boot size issues as they surface.

Okay, I'll remove QA:Testcase_Preupgrade_low_/boot_disk_space_to_install

Thank you all. I'm closing this ticket. Feel free to reopen in need.

Note: See TracTickets for help on using tickets.