Ticket #204 (closed task: fixed)

Opened 4 years ago

Last modified 3 years ago

Adjust watchers to pass over all information at once (required for depcheck)

Reported by: wwoods Owned by: jskladan
Priority: major Milestone: 0.4.4
Component: tests Keywords:
Cc: Blocked By:
Blocking:

Description

Actually get the depcheck test running on post-bodhi-update.

Change History

comment:1 Changed 4 years ago by wwoods

The key bit here is putting some code in the test wrapper that provides depcheck with a full list of all the packages that have requested (but not been granted) the given kojitag. This is necessary for correct testing - we can't test updates individually, since updates can be interdependent.

comment:2 Changed 3 years ago by kparal

The test wrapper can't help in this case. If you run "watch-bodhi-requests.py --dryrun" you will see that for every new update in Bodhi a new autoqa instance is called with different arguments:

autoqa post-bodhi-update --title libvmime07-0.7.1-4.fc14 --updateid FEDORA-2010-17068 --target-tag dist-f14-updates --arch i686 --arch x86_64 --arch noarch libvmime07-0.7.1-4.fc14
autoqa post-bodhi-update --title libisofs-0.6.38-1.fc14 --updateid FEDORA-2010-17030 --target-tag dist-f14-updates --arch i686 --arch x86_64 --arch noarch libisofs-0.6.38-1.fc14
autoqa post-bodhi-update --title openmpi-1.4.3-7.fc14 --updateid FEDORA-2010-16765 --target-tag dist-f14-updates --arch i686 --arch x86_64 --arch noarch openmpi-1.4.3-7.fc14
...

The test wrapper will receive only arguments from each of this call, separately. It can't group it together.

We will need to do some ugly hack (like creating new watcher tailored to depcheck) or change our workflow logic quite a bit. Probably the best way would be to adjust watchers to pass over all information at once. And then the hook or the particular test's control.autoqa file can adjust the handling as needed.

comment:3 Changed 3 years ago by kparal

  • Milestone changed from Package Update Acceptance Test Plan - depcheck to 0.4.4

comment:4 Changed 3 years ago by jskladan

  • Owner set to jskladan
  • Status changed from new to assigned
  • Summary changed from add depcheck to post-bodhi-update hook to Adjust watchers to pass over all information at once (required for depcheck)

comment:5 Changed 3 years ago by jlaska

I spoke to wwoods about some of the remaining depcheck tasks, and he discussed a depcheck task dependency on this ticket. He clarified some of my confusion on this ticket ... I'll try to summarize below.

As Kamil notes in comment#2, post-bodhi-update currently will launch 'autoqa' for every new update discovered. Wwoods indicated that this is the desired outcome for the watcher. I didn't understand why until he mentioned that we want our watchers to match the behavior expected in the 'glorious future'. That is, when the message bus is in action, AutoQA would theoretically receive notification about *each* bodhi update. It wouldn't receive a single notification for when the pool of bodhi updates changes (that could certainly be another event to monitor, but it sounds like it was different from the original intent for post-bodhi-update). Additionally, there may be tests that we want to run against a single bodhi-update (rpmlint/rpmguard/initscripts by the looks of http://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan)? Will felt we should keep the post-bodhi-update watcher behavior as it is now, but provide some shared code that can grab a list of bodhi-updates needed for testing. This code would be used in the wrapper for tests that need to operate over the entire set, such as depcheck, future-conflicts-test and upgradepath.

Of course, all that is a bit of guess work since we don't have a message bus in action. But it at least describes the original idea for the bodhi event notification. In general though, for tests, like depcheck, that operate over the entire update set ... it's an odd workflow that we haven't yet fully documented. For example, some questions remain for me ...

  1. Should the test provide feedback for all packages in the set that PASS, or only the package it was asked to test? This seems like we would have a lot of duplicate testing going on if we only provide feedback for the requested update.
  2. If providing feedback against all -pending updates, shouldn't the wrapper get a list of *untested* -pending updates, and only test that set? If no untested -pending updates ... no testing needed? Otherwise, again we'd have repeated testing.
  3. Then there is the whole timing issue kparal and wwoods were discussing regarding multiple depcheck tests running at the same time, each trying to move items out of the -pending set, and into the ACCEPTED -pending set.

Will expects to have another blog post published shortly to attempt to answer some questions about the flow. Does this make sense? Are there additional flows we need to consider?

comment:6 Changed 3 years ago by wwoods

[It] seems like we would have a lot of duplicate testing going on if we only provide feedback for the requested update.

Agreed. It seems to make sense for the test to provide feedback for all packages that pass, not just the package that triggered the test.

  1. If providing feedback against all -pending updates, shouldn't the wrapper get a list of *untested* -pending updates, and only test that set? If no untested -pending updates ... no testing needed? Otherwise, again we'd have repeated testing.

No. Repeated testing is required for packages that haven't been accepted. We have to keep testing them until they get accepted, or they get pulled out of the 'pending' list (by being obsoleted or forcibly removed).

To illustrate the point: say you have an update for package A, which requires package B. The build for B isn't done yet, though, so A gets tested and fails. When the update for B arrives, we need to test A again - and then it will pass, and be accepted.

So the wrapper needs:

  1. A way to get the pending package list (let's call it get_pending())
  2. A way to get the accepted package list (which is a subset of the pending list). Let's call it get_accepted().

We can do get_pending() just by listing the contents of the -pending tag. get_accepted() is not as straightforward, but we can achieve that by checking our previous test results (as reported in Bodhi or wherever) to see which pending packages have previously passed. Something like:

accepted = []
for p in get_pending():
    if check_accepted(p):
        accepted.append(p)
return accepted
  1. Then there is the whole timing issue kparal and wwoods were discussing regarding multiple depcheck tests running at the same time, each trying to move items out of the -pending set, and into the ACCEPTED -pending set.

Yes - this is ticket #248. And Kamil is correct: if the pending/accepted sets change during the test, our results become invalid. So when this happens, the test wrapper needs to retry the test with the new data.

We can optimize this though. If any new packages have been added to the 'pending' set, that means that a new test run has already been scheduled for those new packages - so we can just cancel our run and let the other one handle it.

See ticket #248 for details.

comment:7 Changed 3 years ago by jskladan

  • Status changed from assigned to closed
  • Resolution set to fixed

The ability to batch-schedule is present in the new_koji_watcher branch. Closing.

Note: See TracTickets for help on using tickets.