I talked to dgilmore in person and he agrees this can be changed. In Rawhide for the time being, just to test it, and once we branch F19, it will be in F19.

Dennis, could you please have a look at it and change the default for Rawhide, if possible? Thanks.

Dennis, Kevin, I'd really like to have this implemented before F19 branches, because then it's too late and we will have to wait for F20.

It cuts the metadata size by 30%, that's huge! It saves 1/3 of your waiting time for yum refresh!

Can one of you please have a look at necessary changes? It probably requires just a minor tweaks in rel-eng scripts. Thanks!

I agree this would be nice to do and now would be a good time. ;)

I guess we need:

• change mash to use xz for it's createrepo

• Add pyliblzma to base? Or ask yum folks to add it as a requires?

Adding notting here for comment.

Replying to [comment:3 kevin]:

• Add pyliblzma to base? Or ask yum folks to add it as a requires?

It's already pulled in as a requirement of yum in F18. So check that off the list. :)

mash just uses the createrepo default - is there a reason that couldn't change?

Replying to [comment:5 notting]:

mash just uses the createrepo default - is there a reason that couldn't change?

I'll gladly create an upstream ticket for it if desired. But please note:

1. it most likely won't be done before F19 branches
2. I assume the --xz option was added for the very reason that distributions might easily change the behavior to suit their needs. It also hints that there was a reason to stay with gz by default.
3. I think the default compression method won't be changed in createrepo unless all important distributions that use it have their package managers adjusted to work with xz. Remember, this is not just about Fedora and yum. I have no idea what is the state of xz support in other package managers.

I think it makes sense to add --xz to Fedora rel-eng scripts now and also create an upstream ticket, which will probably take years to be resolved.

Well, if mash used the command line (which it doesn't).

In any case, changing the default makes it consistent everywhere, and allows us to have per-release settings outside of modifying all the tools & scripts.

The upstream trac is full of spam and nobody seems to care:
http://createrepo.baseurl.org/report/1

There is no point creating a ticket there. But when looking into git, I see mostly Red Hatters anyway:
http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=summary

In this light, we can consider https://bugzilla.redhat.com/show_bug.cgi?id=700020 being an upstream report. As you can see, not much response there either. I'll try to make createrepo developers pay more attention to it.

Still, it would be great if we could change Fedora's default independently of createrepo default. Because my feeling is that that won't be adjusted in the near future.

Also, this commit is interesting:

http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=blobdiff;f=docs/createrepo.8;h=ff359de383c218f717fe0a15cdff68c1c84b6a49;hp=47343924f2abd91e69da9a0ac035681d7e8daaf9;hb=ccbae23bf7752f7ec7f5de49cbf4c49415f773a2;hpb=f3a4ebe8b9290b3df21ad3c689889e7f1ced2316

It's pretty recent (2012-11-19), so it doesn't seem like they're going to change the default "compat" setting soon.

James Antill responded, please read https://bugzilla.redhat.com/show_bug.cgi?id=700020#c13

Is it doable to tweak mash, or do you prefer to wait a year or so before xz becomes the default in createrepo?

Thinking...

The builders run RHEL 6, and use koji to install from the buildroots there. Since they use the koji repos, they likely would have the old metadata compression, so they'd still work. But there may be oppportunity for breakage there that I'm not aware of.

mash would work, in as much as rawhide is composed in a rawhide mock chroot.
The releases are composed on a release-ish install, so that would work there, but we'd need to make sure the createrepo calls in pungi, etc. are updated to match. But this would mean the release repos are different than any random createrepo that is run by the a Fedora user. If we're OK with that, we could do that.

I'd be ok with the difference there. (ie, I'd be ok with us just changing this in mash/pungi for now at least)

Is there any chance this can get implemented in F20?

Any chance this can be done for F21 and Rawhide? yum supports xz in all currently-supported Fedora releases, so the transition would be smooth and probably un-noticable.

From rel-eng side, it just means adding --compress-type=xz to the createrepo call in the script.

This seems like a pretty good win. The bz2 files in Rawhide right now are 56M, and that goes down to 47M with xz — almost a 20% improvement. And if we go to xz -9, it's 43M — a 25% "discount".

Action items:
* Add XZ compression option to [https://git.fedorahosted.org/cgit/pungi.git pungi] and [https://git.fedorahosted.org/cgit/mash/ mash]
* pungi uses the createrepo python api, mash executes createrepo, maybe uses the python api as well
* Changes should be somilar to directory hashing
* Patches should be sent to the buildsys list
* The patches need to be included into the Rawhide packages
* Then something (FIXME: figure out what) needs to be adjusted to call pungi/mash with the new option

Dennis to look at implementing.

Mailed in patches:

• https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004275.html
• https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004276.html

Any status updates? Since the patches have been merged, are we likely to see xz-compressed metadata in Rawhide anytime soon?

This was enabled in rawhide and epel7beta yesterday. ;)

and now local fedpkg mockbuild for rawhide is broken on rhel6. :-(

Error: xz compression not available  

I can still build for dist f19, f20 and el6.

'yum install pyliblzma' should fix it.