I will note that https://bugzilla.redhat.com/show_bug.cgi?id=1195811 has been accepted as a Fedora 22 Alpha Blocker, making this issue imminently important. Votes in the ticket would be helpful, as if we are going to make this call, we probably want to do so quickly to have any hope of hitting the Go/No-Go meeting.

http://pki.fedoraproject.org/wiki/Tomcat_8 also suggests that at least in some cases it is impossible to support both Tomcat 7 and 8 with the same codebase; so what about the ''other'' packages that depend on Tomcat? Will we fix 1 and and break 5 others by reverting?

It seems to me that with the few days' notice that those packages had, few are likely to be in any better shape than dogtag. But yes, there's a chance this might also result in some other packages needed to downgrade. According to repoquery:

{{{
$ repoquery --whatrequires tomcat
guacamole-0:0.9.3-2.fc22.noarch
hadoop-httpfs-0:2.4.1-6.fc22.noarch
jenkins-0:1.598-1.fc22.noarch
oat-appraiser-0:1.6.0-16.fc22.x86_64
olfs-0:1.11.0-3.fc22.noarch
oozie-0:4.0.1-2.fc22.noarch
pki-server-0:10.2.1-1.fc22.noarch
thermostat-webapp-0:1.2.0-12.fc22.noarch
tomcat-admin-webapps-0:8.0.18-2.fc22.noarch
tomcat-docs-webapp-0:8.0.18-2.fc22.noarch
tomcat-jsvc-0:8.0.18-2.fc22.noarch
tomcat-webapps-0:8.0.18-2.fc22.noarch
tomcatjss-0:7.1.1-1.fc22.noarch
}}}

So that's the potential risk.

I'm going to refrain from voting until the Tomcat developers have a chance to weigh in here. Since mitr just CC'd them (I think?), I'm guessing they need a bit of time.

I emailed them a week ago, just for the record.

{{{
Hello, maintainers of Tomcat. We have a bit of a problem.

Tomcat 8 landed in Rawhide and the Fedora 22 branch ten days ago and
brought with it some significant ABI changes, at least one of which[1]
has broken the Dogtag Certificate System (and by extension, FreeIPA
and the Fedora Server Domain Controller).

There are also numerous FTBFS of dependent packages in Rawhide[2].

Given the lateness of this major feature change, I would like to
request that the Tomcat maintainers revert this change and downgrade
to Tomcat 7 for Fedora 22. This way, we have a cycle in Rawhide to
deal with the fallout of the ABI changes.

I'm CCing FESCo so they are aware of this request.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1195811
[2]
http://koschei.cloud.fedoraproject.org/groups/maven?order_by=-state%2Cname
}}}

The FESCo list was CCed on that email.

Sorry, hit enter too soon.

We engaged with Alexander Kurtakov, but the Dogtag people have determined that the porting is far too significant to do during Alpha Freeze, it will take at least several weeks of effort. Thus I recommend that this upgrade be moved to Rawhide rather than F22.

Replying to [comment:7 sgallagh]:

Sorry, hit enter too soon.

We engaged with Alexander Kurtakov, but the Dogtag people have determined that the porting is far too significant to do during Alpha Freeze, it will take at least several weeks of effort. Thus I recommend that this upgrade be moved to Rawhide rather than F22.

Do you have a pointer to the email threads on that engagement? I'd like to read them.

The thread you CC'd FESCo on last week didn't really provide a lot of insight beyond "stuff is broken" and the Tomcat people saying "we'll look into it."

What little involvement we had was via IRC and the BZ linked above. Someone in #freeipa may have logs, I can check if you want.

I'm one of the Dogtag developers. We finally managed to bring the Dogtag up with Tomcat 8, but so far we have only completed very basic tests. There are many other things that need to be verified (including interoperability with IPA) before we can be confident that there is no more major problems. In addition to that, we still need to modify Dogtag and Tomcat JSS to support both Tomcat versions because not all platforms support Tomcat 8. It probably takes two more weeks to complete this. If this time frame this is not acceptable, we'd recommend reverting Tomcat 8 in F22 back to Tomcat 7.

Please see the following page for the latest status: http://pki.fedoraproject.org/wiki/Tomcat_8

Just in case it's not very clear from the above: '''this is blocking Fedora 22 Alpha''', and we are two days away from Go / No-Go. We don't necessarily have the luxury of time to wait for the maintainers to respond. We're already at very high risk (nearly certain, really) of slipping a week, and I doubt anyone wants another multi-week slip Alpha.

Looking at the other dependencies:

• guacamole has not been built since January, and has not seen a significant change since last October.
• hadoop has not been built since last October.
• jenkins was updated to a new upstream release in February, but I see no indication anywhere that anything much changed in its relationship to Tomcat.
• oat has not been built since last August.
• olfs has not been built since last October.
• oozie has not been built since last September.
• thermostat is regularly updated and the latest version is quite new, but again I see no indication of a relationship to Tomcat 8, and indeed the docs explicitly list 6 and 7 as supported but not 8: http://icedtea.classpath.org/wiki/Thermostat/UserGuideV1.2, "At this point, we know the thermostat webapp works with Tomcat 6, Tomcat 7, Jetty 8, Jetty 9, JBoss AS 7 and Wildfly 8."
• tomcatjss has not been built since last September.

It seems really unlikely that anything in the above is so much better with Tomcat 8 than Tomcat 7 that it would justify delaying the release for weeks in order to integrate FreeIPA with Tomcat 8.

FreeIPA is a core component of Fedora. If everyone involved in FreeIPA development is saying it's not possible to rebase FreeIPA on Tomcat 8 at this point in the cycle without substantial delays, and we don't want substantial delays, the '''only''' reasonable decision is to revert Tomcat 8. I can't honestly see what the Tomcat maintainers could have to say that could change that; do we really think they're going to come up with a magic wand that makes FreeIPA work with Tomcat 8 tomorrow, or somehow magically makes us not worry about FreeIPA as a core Fedora component any more? If not, what exactly is it that we are waiting for?

Yeah. +1 revert here.

Just in case it wasn't clear, I'm also +1 for reverting.

Replying to [comment:11 adamwill]:

Looking at the other dependencies:
Thanks for doing the research. +1 to revert, then.

FreeIPA is a core component of Fedora. If everyone involved in FreeIPA development is saying it's not possible to rebase FreeIPA on Tomcat 8 at this point in the cycle without substantial delays, and we don't want substantial delays, the '''only''' reasonable decision is to revert Tomcat 8. I can't honestly see what the Tomcat maintainers could have to say that could change that;

I was worried about something like “Reverting would break jenkins, making further development on Fedora very difficult, and fixing jenkins again would also involve significant delays.” It is not quite clear that FreeIPA is such a “core” component that it having FreeIPA working would outweigh breaking several other components. (And given Fedora’s “First” value, the pain should by default, though not automatically, be on the project that is late to update.)

Replying to [comment:14 mitr]:

I was worried about something like “Reverting would break jenkins, making further development on Fedora very difficult, and fixing jenkins again would also involve significant delays.” It is not quite clear that FreeIPA is such a “core” component that it having FreeIPA working would outweigh breaking several other components. (And given Fedora’s “First” value, the pain should by default, though not automatically, be on the project that is late to update.)

I'd generally agree, if FreeIPA had any warning that an update was going to be needed. As previously noted, no communication was made prior to the Tomcat 8 rebase.

Replying to [comment:15 sgallagh]:

Replying to [comment:14 mitr]:

(And given Fedora’s “First” value, the pain should by default, though not automatically, be on the project that is late to update.)

I'd generally agree, if FreeIPA had any warning that an update was going to be needed.

Upstream, Tomcat 8 was released in June 2014. (That does not translate to an updated package available in Fedora, but should count at least as a general warning.)

+1 revert sounds like the best solution. That makes +4...

+1 to revert.

Now who's actually going to go do it?

Hi,

Just an update from the Dogtag team.

Endi has done a great job identifying all of the changes required to get Dogtag working on tomcat 8.
We now have it working (in the sense that it starts up and issues a certificate) under tomcat 8.

Endi has figured out changes that will allow us to support both tomcat 7 and tomcat 8. This will require changes in pki-core (dogtag) and tomcatjss.

Based on Endi's work, we can provide the following estimates:

1. Changes to dogtag and tomcatjss to support tomcat 8 for new installs, and to be working for IPA will take one week development time. We would test using the IPA CI tests.

2. Adding migration scripts to support upgrades from existing instances will take another week.

I've got a scratch-build running right now for the revert:
http://koji.fedoraproject.org/koji/taskinfo?taskID=9133969

I just asked the dogtag folks to run their test suite against the above build. Assuming that works, I'm going to bump epoch in F22 and Rawhide and rebuild. (Rawhide will keep Tomcat 8, but I still need to bump epoch to ensure an upgrade path).

Thanks very much to the Dogtag team, but FWIW I still strongly believe reversion is the only correct choice. We have a whole process - the Change process - which is intended to be used to ensure that changes like this are properly managed.

It makes a mockery of that process if someone can land a package which entirely breaks something as fundamental to one of our Flavors as FreeIPA with no announcement and no prior notification, and our response is not "this is not acceptable" but "OK, fine, we'll ask the FreeIPA developers to move heaven and earth to become compatible with it, on a timescale which is completely at odds with the development schedule we carefully planned out".

If we were going to do that, we might as well just ditch the Change process and let people land stuff whenever they like.

So thanks to FESCo for voting to revert, and I believe we should continue on that path.

(also, no disrespect at all to the Dogtag devs, but an old QA rule of thumb states that all developer estimates should be, at a minimum, tripled. :>)

I confirmed that the domain controller role deploys on Fedora 22 with the scratch build I created and provides at least a basic smoketest of functionality (I tested user lookups and 'getcert' to confirm that the expected certificates were generated).

Since this test passed, I have issued an official build of tomcat in [http://koji.fedoraproject.org/koji/buildinfo?buildID=617710 Fedora 22] and [http://koji.fedoraproject.org/koji/buildinfo?buildID=617707 Rawhide].

I've also issued a Bodhi [https://admin.fedoraproject.org/updates/tomcat-7.0.59-2.fc22 update] for the tomcat 7 packages.

Replying to [comment:11 adamwill]:

Just in case it's not very clear from the above: '''this is blocking Fedora 22 Alpha''', and we are two days away from Go / No-Go. We don't necessarily have the luxury of time to wait for the maintainers to respond. We're already at very high risk (nearly certain, really) of slipping a week, and I doubt anyone wants another multi-week slip Alpha.

Looking at the other dependencies:

• guacamole has not been built since January, and has not seen a significant change since last October.
• hadoop has not been built since last October.
• jenkins was updated to a new upstream release in February, but I see no indication anywhere that anything much changed in its relationship to Tomcat.
• oat has not been built since last August.
• olfs has not been built since last October.
• oozie has not been built since last September.
• thermostat is regularly updated and the latest version is quite new, but again I see no indication of a relationship to Tomcat 8, and indeed the docs explicitly list 6 and 7 as supported but not 8: http://icedtea.classpath.org/wiki/Thermostat/UserGuideV1.2, "At this point, we know the thermostat webapp works with Tomcat 6, Tomcat 7, Jetty 8, Jetty 9, JBoss AS 7 and Wildfly 8."
• tomcatjss has not been built since last September.

I'm one of the Thermostat people and we haven't tested Tomcat 8. It might or might not work. Reverting to 7 for F22 seems safer for us, but we'll bend to the decisions being made here :)

Ok, +1 to revert

Discussed in today's fesco meeting, sgallagh says this is effectively done.