Ticket #440 (closed task: fixed)

Opened 4 years ago

Last modified 4 years ago

Improve updates process to avoid "windows of doom"

Reported by: drago01 Owned by:
Priority: major Keywords: meeting
Cc: rdieter, jsmith, ianweller, rhughes Blocked By:
Blocking:

Description

Proposal topic

A recent PackageKit? updated moved the packagekitd binary to a different location and thus breaking the selinux label.

The result is that packagekitd silently fails to start and thus users who updated to this version will no longer receive update notifications and thus potentially render a lot of systems vulnerable as they stop getting updates.

The bug has been fixed in a recent selinux-policy package, but this does not really help the users that updated in this period aka the "window of doom" ... they will just don't get any update notification and presume that they just aren't any.

Overview

The package in question has gone through the new updates process but we still failed to detect this major flaw; a breaking the update infrastructure should be avoided at all cost as it dealing with it afterwards is always troublesome and in some cases even "unfixable".

The problem in this specific case it does only happen when selinux is in enforcing mode; which leads to the conclusion that the "proventesters" did test on a non standard configuration (we default to selinux on).

Problem space

See above.

Solution Overview

There are two problems here; the doomed users that where affected by that update and the fact that we let it happen.

I am afraid the former can't really be fixed the best we can do is to issue some statement and hope that it gets some attention from the press so that at least some of the affected users become aware of the issue.

As for avoiding it from happening again we proventesters should be educated that they should (must) test using some standard default configuration and probably go a step further and demand that they should not only ack an update with "works for me" but provide information on what exactly they tested and on which system configuration that allows the maintainer to judge whether his/her package got enough testing on a wide range of system configuration and seeing what tests where actually made is a way better deciding base than "works for me".

Active Ingredients

QA Proventesters, maybe the board / FPL for the "press statement"; package maintainers

Change History

comment:1 Changed 4 years ago by drago01

  • Keywords meeting added

comment:2 Changed 4 years ago by drago01

It seems that I was mixing two issues here the actual problem that causes update notifications to not appear was https://bugzilla.redhat.com/show_bug.cgi?id=615099 which is even worse as it affects everyone regardless of the status of selinux.

Which still means that we 1) have to deal with it somehow and 2) try to fix up the process to avoid it from happening again in the future.

Sorry for the confusion but unfortunately the problem still remains.

comment:3 Changed 4 years ago by rdieter

  • Cc rdieter added

comment:4 Changed 4 years ago by jsmith

  • Cc jsmith added

comment:5 Changed 4 years ago by rhughes

FWIW, I use my system with SELinux enabled, but not enforcing -- which allows me to see any AVCs that are generated with any new packages. In my testing, no AVCs were generated. I have no idea why.

comment:6 follow-up: ↓ 7 Changed 4 years ago by kevin

ok, so we wanted at the meeting to do two announcements here:

a) one to the main announce list:

This announcement will let users know that in some cases if they have not seen any updates in a while, they should open a terminal and 'su -c 'yum update to get the new PackageKit? and gnome-packagekit and selinux policy. Or is there some better way to get them just those updates? 'setenforce 0 / set to permissive mode' and update? Thoughts?

b) one to the devel-announce / test-announce lists:

This reminds people testing that they should make sure and have selinux enabled and note any avc's that come up in testing.

I'm not sure why there were no AVC's on permissive mode here. ;(

comment:7 in reply to: ↑ 6 Changed 4 years ago by drago01

Replying to kevin:

ok, so we wanted at the meeting to do two announcements here:

a) one to the main announce list:

This announcement will let users know that in some cases if they have not seen any updates in a while, they should open a terminal and 'su -c 'yum update to get the new PackageKit? and gnome-packagekit and selinux policy. Or is there some better way to get them just those updates? 'setenforce 0 / set to permissive mode' and update? Thoughts?

There is no new PackageKit? needed just gnome-packagekit and selinux-policy. Set to permissive and update will not work due to the second bug.

So the only options are

1) su -c "yum -y update" just update everything and reboot or 2) su -c "yum -y update gnome-packagekit selinux-policy" only update the affected packages

b) one to the devel-announce / test-announce lists:

This reminds people testing that they should make sure and have selinux enabled and note any avc's that come up in testing.

Yeah that's fine probably add a note to encourage verbose test result i.e not just "works for me" but "I tested x, y and z and found no regressions".

I'm not sure why there were no AVC's on permissive mode here. ;(

Yeah I talked with Richard about that before filling the ticket we had no clue what was going on there ...

comment:8 Changed 4 years ago by drago01

OK, to move this forward we should probably have something like this:


A recent update introduced two nasty bugs in our package management tool (PackageKit?) which results into users no longer getting update notifications.

We issued updates with fixes for the above issue but unfortunately, users won't get them unless they update by hand due to missing notifications.

Instructions on how to fix:

1) Open a Terminal (Applications -> Systemtools -> Terminal) 2) Type one of those commands:

su -c "yum -y update" OR su -c "yum -y update gnome-packagekit selinux-policy"

3) reboot

We will improve our QA process to avoid such issues from happening again.


Announcement for the provenpackers:


A recent update passed testing despite having a very nasty flaw [1], we should try to avoid this in the future as dealing with such issues are troublesome.

So when testing an update please make sure that you run a close to default configuration (like SELinux in enforcing mode).

As the recent bug has shown permissive isn't enough to catch all issue so please slip enforcing on at least during your testing.

When providing the test result to the maintainer via bodhi please not what you have actually tested and found working / non working.

Verbose information will provide the maintainers a better basis to decide on (push / not push) than simple reports like "works / does not work".


comment:9 follow-up: ↓ 10 Changed 4 years ago by kevin

For the first one, the selinux issue means not only are their no notifications, but also no updates possible via PK right? so perhaps:

Several recent updates introduced several bugs in our package update tool (PackageKit?) which results into users no longer seeing notifications of new updates or ability to apply them.

We have issued updates with fixes for the above issues but unfortunately, users won't get them unless they apply the pending fixed updates with another tool.

To work around the issue:

1) Open a Terminal (Applications -> Systemtools -> Terminal) 2) Type one of those commands:

su -c "yum -y update" (apply all pending updates now) OR su -c "yum -y update gnome-packagekit selinux-policy" (apply updates fixing only this specific bug).

3) reboot your machine, or logout/login (this will fix this issue, but if you applied all updates in the previous step you may want to reboot for other updates that require them).

We are working hard to improve our QA process to avoid such issues from happening again.

The second one looks generally ok to me. I might do some re-wording on it, but the gist is fine. Would you guys like me to send that one out?

comment:10 in reply to: ↑ 9 Changed 4 years ago by drago01

Replying to kevin:

For the first one, the selinux issue means not only are their no notifications, but also no updates possible via PK right? so perhaps:

Several recent updates introduced several bugs in our package update tool (PackageKit?) which results into users no longer seeing notifications of new updates or ability to apply them.

We have issued updates with fixes for the above issues but unfortunately, users won't get them unless they apply the pending fixed updates with another tool.

To work around the issue:

1) Open a Terminal (Applications -> Systemtools -> Terminal) 2) Type one of those commands:

su -c "yum -y update" (apply all pending updates now) OR su -c "yum -y update gnome-packagekit selinux-policy" (apply updates fixing only this specific bug).

3) reboot your machine, or logout/login (this will fix this issue, but if you applied all updates in the previous step you may want to reboot for other updates that require them).

We are working hard to improve our QA process to avoid such issues from happening again.

The second one looks generally ok to me. I might do some re-wording on it, but the gist is fine. Would you guys like me to send that one out?

Fine by me.

comment:11 Changed 4 years ago by kevin

ok, shall I send something out? Or Jared, would you like to do so?

comment:12 Changed 4 years ago by jsmith

Yes, I'd be happy to send out the update. It will probably be Monday before I get it sent out though, as I'll be travelling most of the day today.

comment:13 Changed 4 years ago by pfrields

Does this affect Fedora 12 as well as 13?

comment:14 Changed 4 years ago by pfrields

I talked to Jared just now -- we'll get something out sooner than that. I had some minor wording changes to the text for the announcement.

Recent updates to Fedora introduced two unexpected bugs that, when combined, result
in users no longer seeing notifications of or being able to apply new updates.
Updates have been shipped to fix these issues, but users will not receive
notifications for them unless they first apply the pending fixed updates using the
"yum" tool.

To work around the issue:

1. Open a Terminal (Applications > System Tools > Terminal)

2. Type one of those commands:

  a. To choose to apply all pending updates now:
     su -c "yum -y update"

  b. To choose to apply updates fixing only this specific issue:
     su -c "yum -y update gnome-packagekit selinux-policy"

3. Reboot your machine, or logout and login.  Note that logout/login will complete
the fix for the notification issue.  However, if you applied all updates in the
previous step, you should reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to
our testing and quality assurance processes to avoid this and similar problems in
the future.

comment:15 Changed 4 years ago by kevin

Looks great. One nitpick: "when combined" is not really accurate. Either bug (the selinux one or the gnome-packagekit one) would cause the issue, they don't need to be combined. perhaps "Either of which result in" ?

We still need to make sure this only affects f13 before sending and note that...

comment:16 Changed 4 years ago by jsmith

Latest draft. Speak now if you have any other changes you'd like to see made before it gets sent out.

Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
 "yum" tool.  (Please note that these issues should only affect Fedora 13.  Fedora 12 does not appear to be affected.)

To work around the issue:

 1. Open a Terminal (Applications > System Tools > Terminal)

 2. Type either of the following commands and enter the password for the root user when prompted.
  a. To choose to apply all pending updates now:
     su -c "yum -y update" 
  b. To choose to apply updates fixing only the specific issues mentioned above:
     su -c "yum -y update gnome-packagekit selinux-policy"

 3. Log out and and then log back in, or reboot your computer.  Note that logout/login will complete the fix for the notification issue.  However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.

comment:17 Changed 4 years ago by jsmith

More slight changes, after continued dialog with stickster:

Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
 "yum" tool.  (Please note that these issues should only affect Fedora 13.  Fedora 12 does not appear to be affected.)

To work around the issue:

 1. Open a Terminal (Applications > System Tools > Terminal)

 2. Type either of the following commands and enter the password for the root user when prompted.
  a. To choose to apply all pending updates now:
     su -c "yum -y update" 
  b. To choose to apply updates fixing only the specific issues mentioned above:
     su -c "yum -y update gnome-packagekit selinux-policy"

 3. Log out and and then log back in, or reboot your computer.  Note that logout/login will complete the fix for the notification issue.  However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.

comment:18 Changed 4 years ago by jsmith

Need to "--skip-broken" to yum commands

Recent updates to Fedora introduced two unexpected bugs, either of which results in users no longer seeing notifications of or being able to apply new
updates.

Updates to the relevant packages have been shipped to fix these issues, but users will not receive notifications for them unless they first apply the pending fixed updates using the
 "yum" tool.  (Please note that these issues should only affect Fedora 13.  Fedora 12 does not appear to be affected.)

To work around the issue:

 1. Open a Terminal (Applications > System Tools > Terminal)

 2. Type either of the following commands and enter the password for the root user when prompted.
  a. To choose to apply all pending updates now:
     su -c "yum -y --skip-broken update" 
  b. To choose to apply updates fixing only the specific issues mentioned above:
     su -c "yum -y --skip-broken update gnome-packagekit selinux-policy"

 3. Log out and and then log back in, or reboot your computer.  Note that logout/login will complete the fix for the notification issue.  However, if you applied all updates in the previous step, you may need to reboot for other updates that require it.

We apologize for the inconvenience, and we are working on further improvements to our testing and quality assurance processes to avoid this and similar problems in the future.

If you have any further questions or require additional assistance, please see https://fedoraproject.org/wiki/Communicating_and_getting_help for ways to get additional help from the Fedora community.

comment:19 Changed 4 years ago by pfrields

I talked with Kevin Fenzi a bit, and we agreed that after the parentheses in paragraph 2, we should add something like this:

{{{Packages fixing these issues were pushed to the Fedora software repositories around 2010-07-22. Users who have done a manual 'yum update' since that push may have already received these fixes.}}}

We don't want to say "ignore this email," but we should also acknowledge that lots of people do 'yum update' out of habit a couple times a week.

comment:20 Changed 4 years ago by ianweller

  • Cc ianweller added

When this notice is out, we'll add a link to the notice to start.fp.o, fp.o, and fp.o/wiki/* to make sure people know there is an issue.

I'll also make sure some of the #fedora admins know so they can set the /topic and spread the word to other admins.

comment:22 Changed 4 years ago by kevin

Is there any further actions to take here? Or shall we close this out now?

comment:23 follow-up: ↓ 24 Changed 4 years ago by kkofler

Doesn't recent PackageKit try to update itself first? So I think you can still end up with a broken PackageKit due to an outdated selinux-policy even if you update now, unless the latest PackageKit has a versioned Conflicts on the old selinux-policy!

comment:24 in reply to: ↑ 23 Changed 4 years ago by drago01

Replying to kkofler:

Doesn't recent PackageKit try to update itself first? So I think you can still end up with a broken PackageKit due to an outdated selinux-policy even if you update now, unless the latest PackageKit has a versioned Conflicts on the old selinux-policy!

UHG!

Yeah you are right lets image this case: 1) Fresh install 2) PackageKit? only updates itself, rpm and yum 3) You end up hitting the brokenness.

I am not sure how a conflict would solve that though. A "Requires: selinux-policy >= fixedVersion" would.

comment:25 follow-up: ↓ 27 Changed 4 years ago by pfrields

  • Cc lennart added

If there are package changes recommended or needed, it would make sense to cc the maintainer here.

comment:26 Changed 4 years ago by pfrields

  • Cc rhughes added; lennart removed

Argh, wrong maintainer, sorry for the noise.

comment:27 in reply to: ↑ 25 Changed 4 years ago by drago01

Replying to pfrields:

If there are package changes recommended or needed, it would make sense to cc the maintainer here.

The right place for this is bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=620943

FESCo shouldn't be needed / involved anymore.

comment:28 Changed 4 years ago by kevin

  • Resolution set to fixed
  • Status changed from new to closed

I think we can close this now. The process should work for anyone not getting updates, and hopefully we will have PackageKit? requiring the needed selinux-policy so new updates will work.

comment:29 Changed 4 years ago by ianweller

I'm nuking the global message on start.fp.o, fp.o and fp.o/wiki/* today if there are no objections. (It's been 30 days.)

comment:30 follow-up: ↓ 31 Changed 4 years ago by jsmith

Sounds good to me. Please go ahead and nuke the global message.

comment:31 in reply to: ↑ 30 Changed 4 years ago by drago01

Replying to jsmith:

Sounds good to me. Please go ahead and nuke the global message.

No please don't.

As long as this bug https://bugzilla.redhat.com/show_bug.cgi?id=620943 isn't fixed people doing new installs are STILL affected by the issue.

comment:32 Changed 4 years ago by kevin

That bug is now closed and the fixed package pushed out to f13 stable. ;)

Do we need a common bugs entry on this? Or shall we just close it out?

There could be some people still affected, so it would be good to at least have a place to point them for more info when we encounter them.

comment:33 Changed 4 years ago by pfrields

FWIW, I added text to update the common F13 bugs page:

https://fedoraproject.org/w/index.php?title=Common_F13_bugs&diff=198870&oldid=196791

I think the notice can now be nuked.

comment:34 Changed 4 years ago by kevin

I agree. Ian? Can you remove the note? and then we can close this...

Note: See TracTickets for help on using tickets.