#3893 Statistics for cloud image
Closed: Fixed None Opened 10 years ago by mattdm.

I know we have stats on direct downloads of the desktop image from the getfedora web page. (And, I assume also from anyone using the mirrormanager redirects.) I'm interested in seeing the equivalent for the cloud image.

If possible, I'd also like to see stats for people clicking the "launch" button on the new web page if possible (although I expect those are low at present).

Thanks!


So, you would like to see:

  • Number of cloud image downloads clicked on?

  • Number of clicks on launch buttons for cloud images in ec2?

Adding stickster here as he has been working on the iso images stats gathering, perhaps he could add cloud images to that?

Replying to [comment:1 kevin]:

So, you would like to see:

  • Number of cloud image downloads clicked on?

  • Number of clicks on launch buttons for cloud images in ec2?

Yes. And possibly it's a subset of the first, but if I can see number of hits to the cloud image redirects you put in place (which are now used in the RDO and upstream OpenStack documentation), that'd be great too.

I suspect that stats for the destination of the request covers both? Having them broken out (page clicks and hits to the URL from other sources) would be useful if possible.

I'm adding easyfix here in case there's an apprentice that would like to work on this.

It should be just whipping up a script to gather the needed info from logs on log02.

ok, here's a stab at this:

Disclaimer: The numbers below are simply unique IP's that followed the 'direct download' link from the spins page and mirrormanager directed them to a mirror to download. It doesn't include torrents, or people who downloaded directly from dl.fedoraproject.org, or people who downloaded directly from their local mirror. See additional disclaimers at ​https://fedoraproject.org/wiki/Statistics.

Direct downloads of the Images:

{{{
2013-07-02 0
2013-07-03 17
2013-07-04 13
2013-07-05 27
2013-07-06 12
2013-07-07 7
2013-07-08 15
2013-07-09 13
2013-07-10 23
2013-07-11 11
2013-07-12 17
2013-07-13 14
2013-07-14 13
2013-07-15 15
2013-07-16 18
2013-07-17 15
2013-07-18 14
2013-07-19 16
2013-07-20 20
2013-07-21 20
2013-07-22 10
2013-07-23 24
2013-07-24 9
2013-07-25 15
2013-07-26 22
2013-07-27 19
2013-07-28 13
2013-07-29 14
2013-07-30 14
2013-07-31 17
2013-08-01 12
2013-08-02 10
2013-08-03 21
2013-08-04 10
2013-08-05 12
2013-08-06 20
}}}

The ami launching thing looks like to me like it just posts direct to amazon, so not sure we have anything there, but I could be wrong. We should consult with websites team on that.

Thanks so much. I'll remember to talk with the web team about making the ami link more trackable when we do the f20 refresh.

Oops. I missed x86_64, that was just 32bit.

Here's updated numbers with both:

{{{
2013-07-02 0
2013-07-03 68
2013-07-04 93
2013-07-05 85
2013-07-06 56
2013-07-07 23
2013-07-08 31
2013-07-09 82
2013-07-10 93
2013-07-11 64
2013-07-12 98
2013-07-13 66
2013-07-14 41
2013-07-15 37
2013-07-16 82
2013-07-17 67
2013-07-18 79
2013-07-19 90
2013-07-20 76
2013-07-21 49
2013-07-22 31
2013-07-23 88
2013-07-24 85
2013-07-25 82
2013-07-26 91
2013-07-27 69
2013-07-28 41
2013-07-29 41
2013-07-30 80
2013-07-31 69
2013-08-01 80
2013-08-02 64
2013-08-03 72
2013-08-04 31
2013-08-05 39
2013-08-06 76
}}}

Would it be possible to have this as a monthly script which e-mails the cloud mailing list with totals for that month? (with i386 and x86_64 totaled separately, please.)

What is the status here? I'm interested working on this ticket if it is still active.

Status is "desparately needed; thanks!"

Still open and available. ;)

Basically needs a script created and a cron job on log02 to run it.

logs are under /var/log/hosts/ on log02.

Had a quick look at log02. I'm wondering which hosts are relevant in this case. From which hosts are cloud images downloaded from?

should be the proxy0 machines... they are the ones that get the requests. There should be a download.fedoraproject.orgaccess.log for each proxy per day that has the hits.

Here is first try. When run the script will gather stats for the previous month and email it to the cloud email list. Have a look at it and tell me what you think.

Some minor nits:

Might make sender 'admin@fedoraproject.org' (if we want people to be able to reply back with questions/issues), or 'nobody@fedoraproject.org' (a address that just drops all replies) if we don't expect folks to reply.

I guess nobody makes more sense... they can file tickets with issues.

Otherwise it looks pretty good. Could you provide a sample output here so we can sanity check it and see if Matt needs anything else?
(or possibly do two... december and jan?)

Script's output for december 2013:

Download statistics for cloud images in month 12-2013 :

  • 32-bit arch :

    total = 1045, qcow2 = 728, raw.xz = 317

  • 64-bit arch :

    total = 6443, qcow2 = 4691, raw.xz = 1752

  • qcow2 images :

    total = 5419, 32-bit = 728, 64-bit = 4691

  • raw.xz images :

    total = 2069, 32-bit = 317, 64-bit = 1752

Output for november 2013:

Download statistics for cloud images in month 11-2013 :

  • 32-bit arch :

    total = 1578, qcow2 = 1324, raw.xz = 254

  • 64-bit arch :

    total = 5792, qcow2 = 4531, raw.xz = 1261

  • qcow2 images :

    total = 5855, 32-bit = 1324, 64-bit = 4531

  • raw.xz images :

    total = 1515, 32-bit = 254, 64-bit = 1261

This is what I get. Check the results and let me know if anything isn't OK.

Moving all currently open easyfix tickets to the HANDYWAVY-FUTURE milestone.

I'm clearing the assigned status on all easyfix tickets.

If you are an apprentice actively working on this ticket, feel free to reassign to yourself. Otherwise let a new apprentice have a look.

What is pending here? Maybe I can finish it off.

Replying to [comment:21 adimania]:

What is pending here? Maybe I can finish it off.

I think "someone to set it up as a cron job?"

It will probably also need to be updated for F21

minor fix to get the stats from the right path
cloud-image-stat-cron.2.py

I have done a minor fix to get the stats from the right path. I'll get this in cron now.

I don't see any change that I need to make with respect to F21. The script looks generic to me and unless I am wrong, it should work as is.

So, log01 is now in ansible. Also, we are frozen, so added to it will need a freeze break. ;)

I'd say make a new roles/cloudstats/ role and add it to the playbooks/hosts/logserver.yml.

Get a patch setup and mail it to the list for a freeze break to apply. :)

includes cloud-image-stat-cron.2.py version of the file
roles-cloudstats.tgz

Replying to [comment:24 kevin]:

So, log01 is now in ansible. Also, we are frozen, so added to it will need a freeze break. ;)

I'd say make a new roles/cloudstats/ role and add it to the playbooks/hosts/logserver.yml.

Get a patch setup and mail it to the list for a freeze break to apply. :)

Attaching diff for logserver.yml playbook, and cloudstat role.
Kindly, check and provide feedback

Login to comment on this ticket.

Metadata