Ticket #5721 (new task)

Opened 19 months ago

Last modified 2 months ago

Need application icons and application descriptions

Reported by: rhughes Owned by: rel-eng@…
Milestone: Fedora 21 Alpha Component: other
Keywords: Cc: bochecha
Blocked By: Blocking:

Description

Hi,

For the application installer preview I need the application icons and localized description data. I had the fedora-app-data package NAKd, but something needs to provide this data. Spot seemed to think it would be okay to include as metadata.

The idea is to:

  • Generate the appstream.xml and appstream.tar files on koji
  • xmlmerge the .xml files and cat the .tar files when mash'ing the compose together.

I've got C code to do the former, although someone familiar with python probably wants to re-write the code before putting it into koji. For F20 we can probably just ship the 15 most popular application icons and descriptions in gnome-software, although this won't scale with all the apps in Fedora.

Any questions, I'm hughsie on IRC. Thanks.

Change History

comment:1 Changed 19 months ago by kevin

Perhaps we could get this data from tagger/packages?

You are already getting tag data from tagger, or is that not the case yet?

Tagger/packages already has the icons and descriptions, however, I don't know about the localized descriptions.

Perhaps we can discuss this next week and see if this approach would work?

comment:2 Changed 19 months ago by till

  • Component changed from koji to other

comment:3 follow-up: ↓ 4 Changed 18 months ago by ralph

I don't think tagger can export any of that, but packages can:

http://pkgwat.rtfd.org

Everything but localized descriptions... :/

comment:4 in reply to: ↑ 3 Changed 18 months ago by mclasen

Replying to ralph:

I don't think tagger can export any of that, but packages can:

http://pkgwat.rtfd.org

Everything but localized descriptions... :/

To be clear, we're no looking for a way to extract anything from .spec files - we're adding metadata that is specifically written for this purpose to upstream tarballs, and want to extract this metadata at build time.

The approach has gotten a legal ok here: https://lists.fedoraproject.org/pipermail/legal/2013-September/002232.html

comment:5 Changed 17 months ago by mclasen

Has there been any progress on this ? It would be great to get this in place early for f21.

comment:6 Changed 17 months ago by bochecha

  • Cc bochecha added

comment:7 follow-up: ↓ 9 Changed 16 months ago by ausil

hi,

sorry for the delay here, so im really trying to understand things so i know where we can fit the generation into our processes. it really seems like its something that does not fit into any of the existing compose processes. We will need tooling and a way to generate the metadata in a clear repeatable way.

the big issue i see is we need to install the rpms to be able to read the file, its not something we can really extract from the rpm metadata. so it really doesnt fit into createrepo which is the tool we use to generate repo metadata today.

we would need something like createrepo, that will only pull the data from new updates and merge in with existing data to ensure we don't have to read the file extracted from every package every single run.

I also don't really get how you want to distribute the data.

I kinda feel like the best option here is a web service that runs on fedora infra. that has a two fold purpose, one be a web based app store, two provide the metadata for client app. I could be really wrong though. maybe we should have a meeting to discuss it.

comment:8 Changed 16 months ago by ausil

  • Milestone changed from Fedora 20 Beta to Fedora 21 Alpha

comment:9 in reply to: ↑ 7 Changed 15 months ago by rhughes

Replying to ausil:

We will need tooling and a way to generate the metadata in a clear repeatable way.

Have you seen https://github.com/hughsie/fedora-appstream/blob/master/README.md ?

the big issue i see is we need to install the rpms to be able to read the file

No, you just need the built binary rpm -- no installing involved.

I also don't really get how you want to distribute the data.

As an extra appdata.xml.gz file stored on each mirror as metadata. The idea is that PackageKit? would download this using yum/dnf whatever and then copy the files into the right locations.

one be a web based app store

Not going to work; we've designed a thick client and need the metadata offline.

maybe we should have a meeting to discuss it.

Can we set up a conf call to discuss this? I really need to get this moving, and I'm very willing to actually write the code where required. Perhaps early next week, aim for 4pm GMT time as a few people may well be in the US.

Thanks,

Richard

comment:10 follow-up: ↓ 11 Changed 15 months ago by ausil

I'm currently on Australian Eastern Standard Time, 4PM GMT is 2AM for me. Sorry can't do.

Some issues, what you proposed as tooling is a really tiny piece of the picture. if you want the data on the mirrors along with the rest of the metadata the generation needs to be integrated into createrepo. you would need some way to get access to the icons and images as I really don't want to add that much extra data into the mirrors. which puts us back to a webapp for it, which then opens up a webstore, and being able to add a app to all your systems etc. I really think it needs to be on the roadmap.

Dennis

comment:11 in reply to: ↑ 10 ; follow-up: ↓ 12 Changed 15 months ago by rhughes

Replying to ausil:

I'm currently on Australian Eastern Standard Time, 4PM GMT is 2AM for me. Sorry can't do.

Okay, that's fair enough :) What about 2130 GMT?

Some issues, what you proposed as tooling is a really tiny piece of the picture.

Sure, but a tiny part that allows the appdata to stay up to date.

if you want the data on the mirrors along with the rest of the metadata the generation needs to be integrated into createrepo.

The only change I see needed to createrepo would be to cat all the -icon.tar files together and also to xmlmerge the new .xml files then gzip two extra files.

you would need some way to get access to the icons and images as I really don't want to add that much extra data into the mirrors.

The -icons.tar and .xml file would be just two extra files from the koji job, just like the rpms, or the log files.

which puts us back to a webapp for it, which then opens up a webstore, and being able to add a app to all your systems etc.

A webstore is not in scope, and we don't support this multi-system model. If you want to do that kind of thing you should be using satellite or RHN. There are a whole host of sticky problems with doing a web store like having to upload your package lists to a remote server and having to depsolve on the remote host which made this model impossible. The privacy, security and hosting requirements are prohibitive and not what we're going to do.

I really think it needs to be on the roadmap.

There is not going to be a webapp from the GNOME team, we've designed and built a local fat-client for a local system. Either we work together and host this on the Fedora infrastructure, or we'll have to do something different that doesn't involve Fedora.

Richard.

comment:12 in reply to: ↑ 11 ; follow-up: ↓ 13 Changed 15 months ago by ausil

Replying to rhughes:

Replying to ausil:

I'm currently on Australian Eastern Standard Time, 4PM GMT is 2AM for me. Sorry can't do.

Okay, that's fair enough :) What about 2130 GMT?

2130 GMT is 730 am I can make that.

Some issues, what you proposed as tooling is a really tiny piece of the picture.

Sure, but a tiny part that allows the appdata to stay up to date.

if you want the data on the mirrors along with the rest of the metadata the generation needs to be integrated into createrepo.

The only change I see needed to createrepo would be to cat all the -icon.tar files together and also to xmlmerge the new .xml files then gzip two extra files.

We would need to extract the files from all the rpms and put them together

you would need some way to get access to the icons and images as I really don't want to add that much extra data into the mirrors.

The -icons.tar and .xml file would be just two extra files from the koji job, just like the rpms, or the log files.

what koji job? if we wrote a koji plugin to extract them at rpm build time we would then need to have mash fetch them all together. realistically though it should be extracted at createrepo time because that then enables you to make a repo from anywhere and have the data in it.

which puts us back to a webapp for it, which then opens up a webstore, and being able to add a app to all your systems etc.

A webstore is not in scope, and we don't support this multi-system model. If you want to do that kind of thing you should be using satellite or RHN. There are a whole host of sticky problems with doing a web store like having to upload your package lists to a remote server and having to depsolve on the remote host which made this model impossible. The privacy, security and hosting requirements are prohibitive and not what we're going to do.

not at all. you only need to know which apps are chosen, but anyway its not on the table right now even if i think its how it should be done.

I really think it needs to be on the roadmap.

There is not going to be a webapp from the GNOME team, we've designed and built a local fat-client for a local system. Either we work together and host this on the Fedora infrastructure, or we'll have to do something different that doesn't involve Fedora.

That is your choice, i am trying to work with you here. how big is the data that we are looking at?

Dennis

comment:13 in reply to: ↑ 12 Changed 15 months ago by rhughes

Replying to ausil:

what koji job? if we wrote a koji plugin to extract them at rpm build time we would then need to have mash fetch them all together.

Yes, that works too -- did you see the design notes in https://github.com/hughsie/fedora-appstream/blob/master/README.md?

realistically though it should be extracted at createrepo time because that then enables you to make a repo from anywhere and have the data in it.

Right, but I was told not to do this a while ago as it would mean we need to read all the rpms at compose time, and apparently we don't have the i/o capacity to do that. This is why I split the build + compose stages in fedora-appstream.

How big is the data that we are looking at?

5.8M fedora-20-icons.tar.gz 804K fedora-20.xml.gz

It takes about 40 minutes to generate that data if you try to build everything at compose time, or takes about 10 seconds if you generate the data for each package and then just stitch it all together. If you mean how big is the data we have to decompress and parse, it's about 7.3Gb, or basically every package in fedora that ships a desktop file or font, and a few extra.

Richard

comment:14 Changed 10 months ago by ausil

just to follow up here.

Richard and I sat down at devconf and spoke in person about hwat is needed and why.

Richard agreed to get the functinality to create the metadata added into createrepo and createrepo_c we are still anticipating using createrepo in F21. with a move in the future likely.

the plan is to have createrepo extract and add into the metadata info for only changed packages.

comment:15 Changed 2 months ago by rhughes

I blogged a bit about what I think should happen here: http://blogs.gnome.org/hughsie/2014/12/17/actually-shipping-appstream-metadata-in-the-repodata/ -- the unsolvable sticky issue is that the build servers (quite rightly...) don't have access to tools like nm, image loading libraries and can't download random files from various untrusted places. The delta-generation run of appstream-builder could be hooked up into createrepo_c still, but it doesn't solve the other bigger issues.

Could the compose process download the two files from alt.fedoraproject.org and then do modifyrepo (or modifyrepo_c) on the final produced metadata? I'd be happy with that as an interim step, and it means we can stop shipping metadata in a silly package for F22.

comment:16 follow-up: ↓ 17 Changed 2 months ago by ausil

The contents need to come from inside the packages and it all needs to happen as an integrated part of making the repo. downloading it from alt is not an okay interim step. as soon as you have done the work that we have told you needs to be done on multiple occasions we will ensure that we update createrepo and ensure that the appdata is part of the metadata.

comment:17 in reply to: ↑ 16 ; follow-up: ↓ 18 Changed 2 months ago by rhughes

Replying to ausil:

downloading it from alt is not an okay interim step

Can I ask why not? Security issues?

as soon as you have done the work that we have told you needs to be done

I've done the work; all the builder functionality is now available as a GIR-introspectable library which can be consumed from createrepo. The biggest issue is now that I need the builder to download screenshots from the untrusted internet. If the builders are now able to do this then I'll press ahead with the createrepo_c changes. I also need gdk to be installed to be able to convert different icon formats to PNG.

Richard.

comment:18 in reply to: ↑ 17 Changed 2 months ago by ausil

Replying to rhughes:

Replying to ausil:

downloading it from alt is not an okay interim step

Can I ask why not? Security issues?

We do not inject random data into the processes.

as soon as you have done the work that we have told you needs to be done

I've done the work; all the builder functionality is now available as a GIR-introspectable library which can be consumed from createrepo. The biggest issue is now that I need the builder to download screenshots from the untrusted internet. If the builders are now able to do this then I'll press ahead with the createrepo_c changes. I also need gdk to be installed to be able to convert different icon formats to PNG.

the compose boxes will never be able to download from random hosts on the internet. the contents all needs to exist inside the packages.

Richard.

Note: See TracTickets for help on using tickets.