Ticket #30 (closed enhancement: fixed)

Opened 4 years ago

Last modified 4 years ago

Improve preupgrade test case

Reported by: jlaska Owned by: rhe
Priority: major Milestone: Fedora 13
Component: Test Review Version:
Keywords: Cc: rhe
Blocked By: Blocking:

Description

https://fedoraproject.org/wiki/QA:Testcase_Preupgrade

This test should include having 3 kernels installed in /boot to more accurately reflect what real-world users will be starting from. The test now will not sufficiently model how much free space is available on the average users /boot partition. The test should be updated to reflect this.

Change History

comment:1 Changed 4 years ago by jlaska

Hurry ... do you think you might be able to assist with proposing some updates to the existing preupgrade tests?

comment:2 follow-up: ↓ 4 Changed 4 years ago by rhe

Hi James, I tested the latest package, if the space of /boot is not enough, a window would pop up saying 'we recommend you free up 1.1MB by deleting old kernel..'

So, I wanted to include these in test case:

  1. The total free space of /boot needed for preupgrade(but not essential). (Is it possible to estimate?)
  2. Tell that the window would pop up if the space is not enough, and methods to find enough space: a.the removal of all kernels that are older than the currently running

kernel.

b.tune2fs -r 0

I also don't wanna make a test case tooo long.

Thanks. What do you think?

comment:3 follow-up: ↓ 5 Changed 4 years ago by kparal

For Fedora 12, I think the default /boot size is 250MB (needs to be verified). So that can in updated in the test case.

Also the test case can mention that two kernels are easy to install from repositories (stable and updates repo). The third (and others) can be downloaded and installed from Koji:
http://koji.fedoraproject.org/koji/packageinfo?packageID=8

comment:4 in reply to: ↑ 2 ; follow-up: ↓ 6 Changed 4 years ago by jlaska

Replying to rhe:

Hi James, I tested the latest package, if the space of /boot is not enough, a window would pop up saying 'we recommend you free up 1.1MB by deleting old kernel..'

Is this window from preupgrade or from the anaconda installer? I think what we want is for preupgrade to correctly detect anaconda's /boot needs. The situation to avoid is when preupgrade thinks everything is okay ... but anaconda fails as there isn't sufficient disk space available.

So, I wanted to include these in test case:

  1. The total free space of /boot needed for preupgrade(but not essential). (Is it possible to estimate?)
  2. Tell that the window would pop up if the space is not enough, and methods to find enough space: a.the removal of all kernels that are older than the currently running

kernel.

b.tune2fs -r 0

I wouldn't worry about documenting the workarounds for getting more space in /boot in the test case. I think it's safe to leave that documented at http://fedoraproject.org/wiki/How_to_use_PreUpgrade#Troubleshooting.

I also don't wanna make a test case tooo long.

Thanks. What do you think?

The big thing I wanted to update was to make sure that the test case was written to properly capture the real-world upgrade case. Specifically, for users of F-11, they're likely to have multiple kernels installed on the system since they installed. The default maximum, as specified by yum.conf is 3 (installonly_limit=3). As Kamil points out, I think we just need to make sure the test has 3 total kernels installed, before testing.

comment:5 in reply to: ↑ 3 Changed 4 years ago by rhe

Replying to kparal:

For Fedora 12, I think the default /boot size is 250MB (needs to be verified). So that can in updated in the test case. <skip>

RC4 is still 200MB for default /boot.

comment:6 in reply to: ↑ 4 ; follow-up: ↓ 8 Changed 4 years ago by rhe

Thank James and Kparal, I've updated the test case, feel free for modification. https://fedoraproject.org/wiki/QA:Testcase_Preupgrade

comment:7 Changed 4 years ago by jlaska

  • Owner set to rhe

Hurry is doing all the work on this one, so assigning to rhe

comment:8 in reply to: ↑ 6 Changed 4 years ago by jlaska

Replying to rhe:

Thank James and Kparal, I've updated the test case, feel free for modification. https://fedoraproject.org/wiki/QA:Testcase_Preupgrade

Looks good Hurry. I've added some instructions on how to use the koji command to locate and download applicable kernels. If you think that's too much, feel free to remove.

I've made a few other minor wording changes, again feel free to correct/revert.

I like step#8. The only think I don't know is whether step#8 represents a failure case or not. But I think as you've written it, it should be good to push out to fedora-test-list for review.

Can you carry forward your changes to the other preupgrade test also (https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release)?

Thanks, James

comment:9 Changed 4 years ago by rhe

Wow,the case looks much better now. I've forward the changes to https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release.Thanks.

comment:10 follow-up: ↓ 11 Changed 4 years ago by kparal

I have just tried default F12 install, it still creates only 200MB for /boot :( So it should be probably corrected in the test case. Chris' changes (https://bugzilla.redhat.com/show_bug.cgi?id=534055#c7) will probably be effective only at F13.

comment:11 in reply to: ↑ 10 Changed 4 years ago by rhe

Replying to kparal:

I have just tried default F12 install, it still creates only 200MB for /boot :( So it should be probably corrected in the test case. Chris' changes (https://bugzilla.redhat.com/show_bug.cgi?id=534055#c7) will probably be effective only at F13.

Right, I have to change the case back to 200MB. Thanks.

comment:12 follow-up: ↓ 13 Changed 4 years ago by kparal

I have just used preupgrade on my bare metal (unfortunately the fixed preupgrade wasn't in updates yet). It is quite probably that we will hit preupgrade problems again, therefore we should pay more attention to it. I have a few remarks:

  1. My old kernel (from F11) didn't stay installed. Should it stay installed? In that case the test case should mention that.
  1. Do we have a bug number which relates to the need of using "tune2fs -m0 device"? Because that certainly should not be needed - the upgrade process must be running like root.
  1. I propose the test case should contain this requirement:

3a) If only one current kernel is installed, the update process must not complain about insufficient space. It must not be required to delete some files (efi/, grub/splash.xmp.gz), nor to tune the filesystem (tune2fs). If that happens, it's a fault of default partitioning (/boot created too small). A workaround should be present (at least a descriptions of steps needed to take to solve that issue).

3b) When more kernels are installed and the free space is not enough, user should be offered to remove all but most recent kernel. Removing kernels is a delicate matter and all users may probably not be able to do that on their own, so the best solution would be a dialog with just Next->Next->Finish interface. I don't know if current preupgrade offers this feature, but it would be certainly desired. We should create a ticket for that in that case.

These requirements should help us catch the problems next time and call for fixes in advance if needed.

comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed 4 years ago by jlaska

Replying to kparal:

  1. My old kernel (from F11) didn't stay installed. Should it stay installed? In that case the test case should mention that.

I've filed ticket#31 to track investigation of this effort. Anyone interested in owning and following up in that ticket?

  1. Do we have a bug number which relates to the need of using "tune2fs -m0 device"? Because that certainly should not be needed - the upgrade process must be running like root.

Good catch. We have bug#530541 which tracks the work arounds in place for F-12. Will Woods is close to this issue, I've asked for his thoughts on if there is anything to track the discussed phase#2 scripting of the current workarounds.

  1. I propose the test case should contain this requirement:

3a) If only one current kernel is installed, the update process must not complain about insufficient space. It must not be required to delete some files (efi/, grub/splash.xmp.gz), nor to tune the filesystem (tune2fs). If that happens, it's a fault of default partitioning (/boot created too small). A workaround should be present (at least a descriptions of steps needed to take to solve that issue).

I agree, but caution that our test cases should be written to how preupgrade is intended to work, not what we believe the user experience should be. It would be nice for it to be more seemless, but that support is not presently included in preupgrade. Until it is, I wouldn't feel comfortable with our test cases being based on that assumption.

Here's what I would recommend to meet your proposal ...

  1. Modify QA:Testcase_Preupgrade and QA:Testcase_Preupgrade_from_older_release so that the setup instructions have only 1 kernel installed prior to running preupgrade. Also, adjust the expected results so that no low disk-space dialogs present. If any low-disk space dialogs appear, the test can note that manual workarounds are available (see http://fedoraproject.org/wiki/How_to_use_PreUpgrade#Not_enough_space_in_.2Fboot), but the need for manual workarounds would be considered a test result of FAIL.
  2. Create a new test case (perhaps QA:Testcase_Preupgrade_low_/boot_disk_space). This test case intentionally will craft a /boot filesystem with insufficient free space, and ensure that preupgrade responds as expected to this scenario.

3b) When more kernels are installed and the free space is not enough, user should be offered to remove all but most recent kernel. Removing kernels is a delicate matter and all users may probably not be able to do that on their own, so the best solution would be a dialog with just Next->Next->Finish interface. I don't know if current preupgrade offers this feature, but it would be certainly desired. We should create a ticket for that in that case.

These requirements should help us catch the problems next time and call for fixes in advance if needed.

Let's create bugs against preupgrade for any requests for changes in behavior. For this ticket, I want to make sure that our preupgrade tests correctly reflect the current expectations for using preupgrade.

comment:14 in reply to: ↑ 13 Changed 4 years ago by rhe

Replying to jlaska:

Here's what I would recommend to meet your proposal ...

  1. Modify QA:Testcase_Preupgrade and QA:Testcase_Preupgrade_from_older_release so that the setup instructions have only 1 kernel installed prior to running preupgrade. Also, adjust the expected results so that no low disk-space dialogs present. If any low-disk space dialogs appear, the test can note that manual workarounds are available (see http://fedoraproject.org/wiki/How_to_use_PreUpgrade#Not_enough_space_in_.2Fboot), but the need for manual workarounds would be considered a test result of FAIL.
  1. Create a new test case (perhaps QA:Testcase_Preupgrade_low_/boot_disk_space). This test case intentionally will craft a /boot filesystem with insufficient free space, and ensure that preupgrade responds as expected to this scenario.

These make sense. Would change it soon.

comment:15 follow-up: ↓ 16 Changed 4 years ago by rhe

I've updated the cases. For the low /boot space case, the method I wrote is to install another kernel to make free space insufficient. An easier method is to create a file with certain size using dd and put the file in /boot. How do you feel the methods or do I get it wrong?

1.https://fedoraproject.org/wiki/QA:Testcase_Preupgrade 2.https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_low_/boot_disk_space 3.https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release 4.https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release_low_/boot_disk_space

comment:16 in reply to: ↑ 15 ; follow-up: ↓ 17 Changed 4 years ago by jlaska

Replying to rhe:

I've updated the cases. For the low /boot space case, the method I wrote is to install another kernel to make free space insufficient. An easier method is to create a file with certain size using dd and put the file in /boot. How do you feel the methods or do I get it wrong?

On closer inspection, it seems there are several low disk-space conditions we may wish to trigger.

  1. Not enough /boot space for preupgrade to download the installer kernel+initrd.img
  2. Not enough /boot space for preupgrade to download the install.img (see https://fedoraproject.org/wiki/How_to_use_PreUpgrade#Method_2:_Trick_preupgrade_into_downloading_the_installer)
  3. Not enough /boot space for anaconda to install a new kernel

As written, I think the low disk-space tests targets #3 above. I believe that should be sufficient for testing the real-world use case. I've made some minor adjustments to the wording of the following test cases. I think they look good.

I think we can focus our low disk-space tests on just upgrading from the previous release (not from the previous previous release). I'd be okay with not adding the following test to our matrix. Do you agree or have concerns?

comment:17 in reply to: ↑ 16 Changed 4 years ago by rhe

Replying to jlaska:

Replying to rhe:

I've updated the cases. For the low /boot space case, the method I wrote is to install another kernel to make free space insufficient. An easier method is to create a file with certain size using dd and put the file in /boot. How do you feel the methods or do I get it wrong?

On closer inspection, it seems there are several low disk-space conditions we may wish to trigger.

  1. Not enough /boot space for preupgrade to download the installer kernel+initrd.img
  2. Not enough /boot space for preupgrade to download the install.img (see https://fedoraproject.org/wiki/How_to_use_PreUpgrade#Method_2:_Trick_preupgrade_into_downloading_the_installer)
  3. Not enough /boot space for anaconda to install a new kernel

As written, I think the low disk-space tests targets #3 above. I believe that should be sufficient for testing the real-world use case. I've made some minor adjustments to the wording of the following test cases. I think they look good.

I also feeled that the third one already covers the real-world use.

I think we can focus our low disk-space tests on just upgrading from the previous release (not from the previous previous release). I'd be okay with not adding the following test to our matrix. Do you agree or have concerns?

I added this case to make the cases more integrated. But yes, Focusing on the previous release is enough and can also reflect that in the older release.

comment:18 Changed 4 years ago by jlaska

  • Component changed from Wiki to Test Review

comment:19 Changed 4 years ago by jlaska

Discussed during team sprint, the test cases look great! The additional failure cases noted in comment#16 will be something to target for a preupgrade focused test day.

Hurry will post the tests to the mailing list for final review. Leaving this open for now...

comment:20 Changed 4 years ago by rhe

  • Resolution set to fixed
  • Status changed from new to closed

Review finished so close it. Another ticket will be created for cases of different disk space checks in /boot for a test day.

Note: See TracTickets for help on using tickets.