For 2011-11-07 meeting.
I wont be able to make it to the meeting tonight, so I'll just give my 2¢ here.
I think there still is a lot to be discussed, considered, done etc, but generally I am positive about this feature. Even if I still have some concerns about backwards compatibility and the contingency plan (which I'll add to the talk page in the wiki), I am generally +1.
FESCo decision:
defer to next week.
Given the scope of this change, I think it might be prudent to defer this decision until the new FESCo has been elected. It seems unfair to decide this and then dump it on the newcomers to execute.
As in the original discussion, I'm tentatively OK with this, although I have concerns about the implementation on upgrade - I don't want to have a /bin directory with a forest of symlinks inside it, for example.
In the 2011-11-14 meeting fesco had 4 -1 votes, and 4 +1 votes... and we were not clear if notting was +1 or not, given that the current feature calls for a forrest of symlinks in /bin.
Waiting on nottings clarified vote here. ;)
Just to add here: Can you please hold off any mass filing of bugs or working on a bunch of packages until FPC has had a chance to visit this and the scope is all clear? I think it would be bad to start doing one thing, then have to back up and rework that if the scope changes. Thanks.
Essentially, I want the changes constrained to the directory tree, even on upgrade, such that an upgraded system resembles a freshly installed system. Such a mechanism also allows us to avoid packaging changes, in the general case.
Harald has mentioned that there's a way to do this (which isn't detailed on the feature page) - what is this?
From Harald:
{{{ Basically the idea is:
So, nobody can install the changed packages without installing the newer filesystem.
Updating the filesystem outputs this:
$ sudo rpm -Uvh x86_64/filesystem-3-1.fc17.x86_64.rpm Preparing... ########################################### [100%]
Please read the Release Notes on http://fedoraproject.org/..... and run /usr/sbin/migrate-rootdirs-to-usr
error: lua script failed: /lib is not a symlink
error: filesystem-3-1.fc17.x86_64: install failed error: filesystem-2.4.44-1.fc16.x86_64: erase skipped
would first make a dry run, reporting any errors it cannot resolve automatically
would check if a dry run works and executes the renaming and/or copying of the files and directories to /usr and finally move away the directories and create the toplevel symlinks ( bin -> usr/bin, lib -> usr/lib, sbin -> usr/sbin, lib64 -> usr/lib64 )
The system is now ready for the new packages.
}}}
Given those safeguards to ensure upgrades == fresh installs, I'm OK with it. +1
Can migrate-rootdirs-to-usr be implemented in the filesystem %pre script as lua instead of separately? One of the driving rules about Fedora packaging is that it needs to assume that no one is watching the output from rpm.
I can see that if anaconda and preupgrade are both invoking the script (they are invoking it, correct? Not just including it?) what we're trying to do is catch all the upgrade cases and give messages where the user will see them. FPC might allow that but it would be better if it could be self contained and simply do the right thing. If it can't be implemented in lua in the %pre, do the unsupported means of upgrading show these messages? For instance, yumex, packagekit-gnome, and kpackagekit?
If the user does a yum upgrade, are they assured that the transaction will abort in a place where they haven't changed their installation in disasterous ways? glibc would be one of the packages which conflicts which would order filesystem ahead of it... I suppose it's to early to tell if glibc is going to change such that packages will order it (and thus filesystem) into the front of the transaction.
Side note -- I'm liking this general strategy better than modifying each package with %ghost'd symlinks in /bin etc... good job so far.
Do I understand it right that the new filesystem will not work with unmigrated packages? Or will it somehow automagically work with unmigrated packages even if the unmigrated packages are updated (without the migration) on a system with the new filesystem package?
It should work with unmigrated packages that don't have things with the same binary name in /bin|/usr/bin.
Replying to [comment:8 notting]:
... which would make it approved, if I'm reading right.
OK, but that excludes the sbin bin merge then because all the consolehelper packages would be broken if unmigrated at one go.
Replying to [comment:13 tmraz]:
I think we found a solution, where we can nicely split out the sbin/bin merge from this feature without duplicated work.
+5 votes, feature is approved.
Best leave this open for the communication that has to be done with FPC before it can progress to actually doing work -- We could change the ticket Type if that interferes with reports/workflow, though. /me also takes meeting keyword off until there's an update from the FPC side.
Also just noticed that there hasn't been an FPC ticket opened yet. And you probably want to remove the sections of the Feature that talk about every package needing to have the %ghost symlinks so that the members of the FPC who haven't been following this ticket are not confused by those sections existing but no longer being necessary.
Speaking of packaging - if we have the work constrained to the filesystem packages, and the migrate-directories script (or whatever it's called), would any other packages need to be modified?
With emphasis on need, I think the answer is no. We probably want the other packages to be updated to install into the real directories rather than targetting the symlinks but that process could be a gradual cleanup process instead of sonmething that must be done.
We currently convert all RPMs that install things in /lib, /lib64, /bin, /sbin. Some install compat symlinks which need to be cleaned up. The bin+sbin merge is excluded from this step.
A third of the work is already done in local git copies on Harald's and my box. The plan is to finish it this week, and then need to find a way to mass-commit or mass-bugzilla them. :)
Does the feature page currently reflect current scope and changes? If not, could you please update it?
Also, can you please wait for the FPC to take a look at the above changes and comment before commiting anything?
Also, could you make your git repo available? That way people could look at proposed changes and we can get a good idea before they land?
Thanks!
My point is - if the spec doesn't need to be converted, why rush? It allows for easier portability between releases, for example.
The feature owners came to us this week to have rpm on the builders replaced. after the pain of managing a different rpm in the past ive told them no, they need to get the new features they need into RHEL6 as thats what we use on the builders. they have gotten buy in and it will be in the next RHEL point release, The feature owners have put presure on releng to have rpm replaced on the builders before rhel6.3 which after consultation with other Red Hat release engineers and infrastructure we are unwilling to do. and we reccomend the feature be pushed back to f18 when it can be built in a supportable environment.
The issues we have are a few fold, we need to co-ordinate rpm updates across primary and secondary arches. as well as provide support to people outside of fedora who build fedora on top of rhel. while the feature ownser said to send everyone to them they still will come to us for support. in the past we have also had issues whith other software that links to rpm. we have had to maintain our own net-snmp in the past. Using unreleased software on the build hosts makes things difficult for fedora and those who consume our work. The policy from releng after learning the hard way is to only run supported released software.
Given that the request for a RHEL 6 version of RPM was opened in the middle of December, I'm not thrilled that this is suddenly coming to a head now - if this was an absolute requirement that it be a released version of RHEL 6 rpm, instead of a to-be-released version, this should have been raised at a much earlier date.
Had releng been approached before the last week it would have been raised and communicated.
I understand - this is also a FESCo issue - we should have brought in rel-eng during the feature discussion when it was noted that rpm changes might be required.
We didn't even know before, that koji is running with RHEL6 and learned in the process, that Fedora seems to depend heavily on RHEL and RHEL decisions.
ausil: I see pros and cons for both decisions. I agree that adding different rpm on our builders will be hard to maintain. If FESCo decided to deploy UsrMove in F-17, what rel-eng would need to do? What would be the plan of deployment and maintainance?
First we need extensive testing of the rpm, 2 things need to be right. 1) is that things work as expected when you have a filesystem rpm that needs the new rpmlib. and 2) we need to ensure that we can still build fedora 15 and 16 as well as epel for rhel 4 5 and 6. to my knowledge the rpmlib is not yet testable since a filesystem rpm with the rpmlib requires has never been built. we haver in the past had minor rpm features break things in unintended ways.
We would then need to have an appropriate rpm built for i686, x86_64, ppc64, s390x as well as f15 rpm builds for arm arches, and sparc. so that all the secondary arches can also accomodate the change. we would need to make a repo in fedora infrastructure, add the repo to the builders via puppet and update the builders. we would then need to point everyone outside of fedora to that repo when they come asking and answer the question of why it has to be different.
things work as expected when you have a filesystem rpm that needs the new rpmlib to my knowledge the rpmlib is not yet testable since a filesystem rpm with the rpmlib requires has never been built
We have tested all that locally, but we can obviously not commit that package to koji to test it any further, because koji would break with the old RPM on the builder, and not allow us to build any further package for that distro after that commit, because the buildroot setup would fail.
we need to ensure that we can still build fedora 15 and 16 as well as epel for rhel 4 5 and 6
The added flags should be a dead export of (rpmlib) provides. It does not cause any problems in rawhide, which has the patched RPM version, but does not use the flag, like the older releases will not use it.
We did all reasonable testing on our side. There is surely never a guarantee that there are no bugs. But if there are any, they should all be trivial to fix. I'm convinced, that the only real test setup to get confidence in the correct behaviour is to install the new RPM manually on one of the builders, or a copy of a builder machine, and watch it working. Please let us know if we can do such a test from our side if that's what you are looking for.
we would then need to point everyone outside of fedora to that repo when they come asking and answer the question of why it has to be different
It should only affect Fedora 17 builds on RHEL6 infrastructure. The RHEL6 RPM version with the feature is already approved and will be available when it passed all the usual paperwork. The window between the Fedora 17 release and the RHEL release with the needed RPM patch is not that large.
We can put all needed information and instructions in a wiki page, and point anybody who needs to builds F17 on RHEL6 in the time window until the RHEL6 RPM is released, to that page.
You can expect any needed help from our side that we can provide. Please let us know. Thanks!
FESCo: deferred. No decision was reached.
Useful links from meeting:
https://bugzilla.redhat.com/attachment.cgi?id=541981&action=diff (haraldh, 19:12:42) https://bugzilla.redhat.com/show_bug.cgi?id=761000 (haraldh, 19:14:13) http://koji.fedoraproject.org/koji/buildinfo?buildID=295445 (nirik, 19:26:06) https://fedoraproject.org/wiki/Features/UsrMove#Buildsystem_Transition (haraldh, 19:49:45)
Here's the testing I think rel-eng is looking for.
(Dennis: Please add/correct):
1 Build new RHEL6 rpm with patch.
2 Build new filesystem with conflict.
3 Given the following list:
kernel glibc rpm bash coreutils binutils gcc xulrunner
On a rhel6 instance use mock with no plugins enabled to rebuild packages in section 3 for F15, F16, rawhide, EL4, EL5, EL6
Repeat step 4 with patched rpm and confirm no differences in produced rpms (That cannot be explained by timestamps, etc).
A wiki page with the changed rpm info and pointer to patch that we can point people to.
A fedorapeople or the like repo with patched rpm for people to download.
Replying to [comment:33 kevin]:
Here's the testing I think rel-eng is looking for. (Dennis: Please add/correct): 1 Build new RHEL6 rpm with patch. 2 Build new filesystem with conflict.
? This only needs to be build for Fedora >= 17.
3 Given the following list: kernel glibc rpm bash coreutils binutils gcc xulrunner On a rhel6 instance use mock with no plugins enabled to rebuild packages in section 3 for F15, F16, rawhide, EL4, EL5, EL6 Repeat step 4 with patched rpm and confirm no differences in produced rpms (That cannot be explained by timestamps, etc). A wiki page with the changed rpm info and pointer to patch that we can point people to. A fedorapeople or the like repo with patched rpm for people to download.
Replying to [comment:34 harald]:
Replying to [comment:33 kevin]: Here's the testing I think rel-eng is looking for. (Dennis: Please add/correct): 1 Build new RHEL6 rpm with patch. 2 Build new filesystem with conflict. ? This only needs to be build for Fedora >= 17.
Here is a filesystem srpm, which if rebuild, will only install on a converted filesystem with the patched rpm:
http://harald.fedorapeople.org/downloads/usrmove/filesystem-3-2.fc17.src.rpm
Replying to [comment:35 harald]:
Replying to [comment:34 harald]: Replying to [comment:33 kevin]: Here's the testing I think rel-eng is looking for. (Dennis: Please add/correct): 1 Build new RHEL6 rpm with patch. 2 Build new filesystem with conflict. ? This only needs to be build for Fedora >= 17. Here is a filesystem srpm, which if rebuild, will only install on a converted filesystem with the patched rpm: http://harald.fedorapeople.org/downloads/usrmove/filesystem-3-2.fc17.src.rpm 3 Given the following list: kernel glibc rpm bash coreutils binutils gcc xulrunner On a rhel6 instance use mock with no plugins enabled to rebuild packages in section 3 for F15, F16, rawhide, EL4, EL5, EL6 Repeat step 4 with patched rpm and confirm no differences in produced rpms (That cannot be explained by timestamps, etc). A wiki page with the changed rpm info and pointer to patch that we can point people to. A fedorapeople or the like repo with patched rpm for people to download.
http://people.redhat.com/harald/downloads/rpm/4.8.0-19.el6.0.usrmove.1/
OK, who's doing the testing of builds with this?
Yeah, the filesystem rpm only needs f17, but we need to test all the other things we build with our builders to make sure there are no regressions in the rpm change.
Everything built, installed and looked fine on our side.
All tests passed successfully, and are available inside the local network on this temporarily reserved host: hp-bl685cg6-01.rhts.eng.bos.redhat.com
The buildroot bootstrap of the individually runs used the new rpm (and if applicable filesystem) and looked like this: {{{ readline x86_64 6.0-3.el6 base 178 k rpm x86_64 4.8.0-19.el6.0.usrmove.1 local 897 k rpm-libs x86_64 4.8.0-19.el6.0.usrmove.1 local 307 k setup noarch 2.8.14-13.el6 base 149 k }}}
We built filesystem, rpm, binutils, bash, glibc, ... on different distros with various combinations for updated packages in the buildroot and on the installed host without running into any problems.
The saved test results for the individual distros are available for inspection here: {{{ /usr-move/ /home/test/cvs//RHEL-/ /home/test/git/f/*/ }}}
(In case further testing wants/needs to be done on that box by anybody, note, that the patched version of RPM is still installed on this system)
Please take us to the next step now. Thanks!
Resolved at Fesco meeting on 06-02-2012.
Login to comment on this ticket.