The QA process was clearly something that went terribly wrong (again) and needs a large rethink and to be better aligned with the Fedora QA general process.

A few big ones:
updates in testing not getting appropriate karma (docker at least)
apparent critical release criteria having proper test cases and actually in the matrix to ensure verification eg ansible [1] and root bind mount[2].

We should never get to a case we're in the 11th hour and 59th minute of a release and discover an apparent core feature [3] is terribly broken.

To be clear this isn't directed at any one person, but the whole cloud WG. It really wasn't good enough. If it can't be tested by automation then it should be tested manually, either way the results from any automation should be closely checked and verified.

[1] https://fedorahosted.org/cloud/ticket/126
[2] https://lists.fedoraproject.org/pipermail/cloud/2015-October/006033.html
[3] https://lists.fedoraproject.org/pipermail/cloud/2015-October/006015.html

Replying to [comment:1 pbrobinson]:

The QA process was clearly something that went terribly wrong (again) and needs a large rethink and to be better aligned with the Fedora QA general process.

A few big ones:
updates in testing not getting appropriate karma (docker at least)
apparent critical release criteria having proper test cases and actually in the matrix to ensure verification eg ansible [1] and root bind mount[2].

We should never get to a case we're in the 11th hour and 59th minute of a release and discover an apparent core feature [3] is terribly broken.

To be clear this isn't directed at any one person, but the whole cloud WG. It really wasn't good enough. If it can't be tested by automation then it should be tested manually, either way the results from any automation should be closely checked and verified.

I agree that we did not do proper job in testing Cloud images for Fedora 23 on time. The autocloud project was put into production almost in the last moment. It still needs to be monitored closely and 100s of more tests are required to test the images properly.

We are also missing any dedicated cloud qa volunteer group from this SIG.

To start fixing the issues, I have started with few new volunteers (and I also made a public call for help [1]), who will help us to do manual testing of the cloud images. They will also debug any failures in the autocloud side. I have already got ack from the Infra about getting access to the Fedora Infra Cloud where these new volunteers will test the images. They are also helping us to write more tests for autocloud project. A few of those are already live in production. We are also making sure to create new regression tests for anything we find over mailing lists/bugzilla/and other places.

[1] https://kushaldas.in/posts/need-help-to-test-fedora-cloud-images.html

Reassigning to myself to put in a final update and close ticket.

bump

Final Update (on this ticket):

Had a good conversation with Peter at FOSDEM, and a bit more at DevConf.cz. We all seem to understand that more testing is needed and how things went wrong. I also sat down with Dennis to start documenting the chain of production from source to final images for Atomic. Will document that on the wiki and make sure we know where all the parts live.

We probably need to do some level-setting/onboarding for the workgroup. I might talk to Paul about this and see if we can also prep materials for that.

Finally, going to ask Kushal to give a "state of testing" to the mailing list and we need to start trying to fill gaps where needed there.

Think that covers it - please shoot me an email if you have any questions.