In a comment to my review request for castxml
https://bugzilla.redhat.com/show_bug.cgi?id=1212835
it was pointed out that there might be a need for a formal bundling exception for a copylib called kwsys.
The src/kwsys/CMakeLists.txt file in the sources says the following, indicating that the authors of the code consider this to be copylib:
Each user of the copylib code configures it to use a specific namespace. E.g. castxml configures it to be compiled with the cxsys:: namespace. Other users of kwsys uses other namespaces.
The castxml code is at github:
https://github.com/CastXML/CastXML
The kwsys copylib code is in the subdirectory:
https://github.com/CastXML/CastXML/tree/master/src/kwsys
A bit of information that probably is relevant.
At least one package already in Fedora already uses the kwsys copylib:
https://admin.fedoraproject.org/pkgdb/package/InsightToolkit/
It might also be useful for FPC to know where to actually find the upstream kwsys source: http://public.kitware.com/gitweb?p=KWSys.git
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-08-06/fpc.2015-08-06-16.02.txt):
How is this different from gnulib and libiberty, which both serve a similar purpose, are similarly huge (e.g., libiberty contains even a C++ demangler!), yet both got a copylib exception?
gnulib is somewhat special in that it's not meant to be a library that everyone includes all of, you just bundle the functions you use. Almost all of them are also small fixes for std. library functions, which aren't even used in Fedora.
libiberty is somewhat protected in that it's mainly used by GCC/binutils, it's a crappy implementation and I'm not sure I'd vote to let random apps. use it but for GCC shrug ...
kwsys seemed even more glib like than libiberty, to me, and it's already got namespacing (unlikely libiberty). But in a similar vein nobody is going to remove cmake if the bundling doesn't get fixed, but random xml apps. that FPC know nothing about have a higher bar to get over.
kwsys review ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1251500
FWIW, "castxml" is not a "random xml app", but the successor to gccxml (which isn't being updated for new GCC versions).
gnulib is somewhat special in that it's not meant to be a library that everyone includes all of, you just bundle the functions you use.
Same here:
The Kitware System Library is intended to be included in other projects. It is completely configurable in that the library's namespace can be configured and the components that are included can be selected invididually.
Reopening due to vehement opposition from upstream.
Please follow: http://public.kitware.com/pipermail/cmake-developers/2015-August/026010.html https://bugzilla.redhat.com/show_bug.cgi?id=1251500#c13
To fill out the action item here:
ACTION: The opinion seems to be that this is a mess, -1 for bundling unless you can come up with an important usecase for castxml getting into Fedora soon. (geppetto, 16:56:15)
castxml is required for ITK's (InsightToolkit in Fedora) language wrappings as of 4.8:
ENH: use castxml in wrapping
http://www.itk.org/Wiki/ITK/Release_4.8_Changed_From_Previous
I do not know of ITK-in-Fedora's update schedule. Other packages which use gccxml now may migrate to castxml in new versions. Querying rawhide, CableSwig (seems to be ITK-related), InsightToolkit, and root-reflex (doesn't seem to have moved yet) use it currently.
Here's what I can say in answer to the bundling questions. If upstream has any specific comments, that would be useful.
Different consumers of KWSys use somewhat different versions of the library and only use certain parts of it, though there does appear to be some attempt to synchronize the code over time.
Some changes are, some aren't. Upstream makes no guarantees of API stability.
There is no current Fedora package.
Upstream uses versions of the library that are appropriate for that project.
Upstream (Kitware) is definitely opposed to unbundling KWSys. They have been very receptive to unbundling other code in their other products though.
I have no sense of this. I have not seen any specific concerns raised.
As a maintainer of many of the packages that bundle KWSys I can attest that this would be huge undertaking to unbundle from each of the projects. Each project incorporates KWSys into it under a different namespace so it would require a great deal of patching to the user code.
Upstream appears fairly uninterested. Although perhaps willing to take a look at specific items that could be unbundled or isolated to the consuming code.
Other users of KWSys - cmake, vtk, paraview, itk
Replying to [comment:15 orion]:
Sure. I'll answer it for CMake, and VTK (ParaView uses it via VTK) as those are the ones I'm familiar with. ITK is likely the same answers. I doubt non-Kitware projects use kwsys (directly).
Has the library behaviour been modified? If so, how has it been modified? If the library has been modified in ways that change the API or behaviour then there may be a case for copying. Note that fixing bugs is not grounds to copy. If the library has not been modified (ie: it can be used verbatim in the distro) there's little chance of an exception. Different consumers of KWSys use somewhat different versions of the library and only use certain parts of it, though there does appear to be some attempt to synchronize the code over time.
Updates to kwsys are merged using a git subtree merge. kwsys' master branch is merged as-is into the expected location in the consuming project's source tree (the script Brad uses to do this puts the instructions for the merge in the commit message which mentions the date and git hash used).
Why haven't the changes been pushed to the upstream library? If no attempt has been made to push the changes upstream, we shouldn't be supporting people forking out of laziness. Some changes are, some aren't. Upstream makes no guarantees of API stability.
Our commit hooks (should) deny changes to the subtree-merged directory without going through the subtree merge process, so no non-upstream changes should appear in consuming projects.
Is upstream keeping the base library updated or are they continuously one or more versions behind the latest upstream release? Upstream uses versions of the library that are appropriate for that project.
"Upstream" is upstream for consuming projects and the library. The process for updating it ensure that code always flows down, not up.
What is the attitude of upstream towards bundling? (Are they eager to remove the bundled version? are they engaged with the upstream for the library? Do they have a history of bundling? Are they argumentative?) Upstream (Kitware) is definitely opposed to unbundling KWSys. They have been very receptive to unbundling other code in their other products though.
Yep. There's even a dashboard for testing external copies of projects (modulo those with known issues such as not having a system version (e.g., xdmf libraries) or Fedora has too-new of a dependency (e.g., protobuf (it's on a list, hopefully for 5.0)).
Does the maintainer of the Fedora package of the library being bundled have any comments about this? As a maintainer of many of the packages that bundle KWSys I can attest that this would be huge undertaking to unbundle from each of the projects. Each project incorporates KWSys into it under a different namespace so it would require a great deal of patching to the user code.
Agreed.
Is there a plan for unbundling the library at a later time? Include things like what features would need to be added to the upstream library, a timeline for when those features would be merged, how we're helping to meet those goals, etc. Upstream appears fairly uninterested. Although perhaps willing to take a look at specific items that could be unbundled or isolated to the consuming code.
Though without a definite timeline on our side.
And ParaView is due to transitive use through VTK (though it does use bits VTK doesn't use, VTK is the "bundler" here.
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-08-20/fpc.2015-08-20-16.00.txt):
The package in review has been updated according to my understanding of the revised decision:
Let me know if I have misinterpreted anything.
That looks good. Couple items I'll mention here for future reference for other packages using KWSys:
If you look in the kwsys CMakeLists.txt file you'll find that other components are activated as dependencies of others. For example, SystemTools enables Directory, FStream, and Encoding, so all of those should be noted as well.
For projects that do not remove files that they don't need, you can use something like this to remove them: {{{
find PATHTOKWSYS -name *.[ch]* | grep -vE '^PATHTOKWSYS/([a-z].*|Configure|SharedForward|String.hxx|LIST_OF_COMPONENTS)(C|CXX|UNIX)?.' | xargs rm }}}
e.g. for vtk: {{{ find Utilities/KWSys/vtksys/ -name *.[ch]* | grep -vE '^Utilities/KWSys/vtksys/([a-z].*|Configure|SharedForward|String.hxx|Base64|CommandLineArguments|Directory|DynamicLoader|Encoding|FStream|Process|RegularExpression|System|SystemTools|SystemInformation|FundamentalType|MD5|Glob)(C|CXX|UNIX)?.' | xargs rm }}}
We had talked about a version string for the bundled provides, but I don't know if we had agreed on a format.
Not sure why I didn't close this one earlier, given the removal of the bundled library policy, there's no reason for this ticket to be open.
Metadata Update from @ellert: - Issue assigned to orion
Login to comment on this ticket.