• Update python-fedora's pkgdb.py module to use the new URL structure. fchiulli is currently testing this and fixing bugs with me.
• Test db update: (Toshio tested update working 02-08-2010).
• Expect a few more changes to the db schema wrt repo and packagebuilds
• Test pkgdb-sync-yum: (Toshio running an import on app01.stg 02-08-2010)
• Test all URLs are working
• Update staging
• Test new features:
• critpath
• sqlitebuildtags
• App Browser
• Alert people on fedora-devel-list that new pkgdb is in staging with breaking changes to the URLs.

• python-argparse has been branched for EL-5 and I'm the maintainer there. Still waiting to hear back on python-cpio.
• Current db schema looks good. Only concern is the rpmfiles table. mbacovsk and I have talked about this and cut the size somewhat. We're waiting on getting a full yum repo import before deciding if we must do more work.
• mbacovsk has fixed and optimizaed pkgdb-sync-yum quite a bit. We were able to sync three repos yesterday (F-11-i386, F-11-i386-update, and F-11-i386-updates-testing). However, that wasn't a full day's work; much of it was spent debugging crashes due to unicode. We'll see how far we can get over the weekend.
• fchiulli has identified many places where python-fedora was broken and fixed them. he's also identified several bugs in the pkgdb server that have been fixed.
• mbacovsk has made some UI improvements.

We'll need rpm-4.6.0 in order to process all the xz compressed rpms. Installing that from the builder's repo onto app01.stg. We'll need to confirm with mmcgrath that it's fine to do that on bapp01.

Once the data is all synced from the repo (< 1 week; pkgdb website can be used concurrently) a new run that has no new packages takes this long:

real 8m2.537s
user 6m30.012s
sys 0m13.545s

It's a little noisy for running in cron jobs; we'll need to look at tuning it to only report on errors.

Otherwise, the import script is looking good.

Some Database statistics:
rpmfiles table: 898 MB
pkgdb size: 2787 MB
Number of repositories whose packages are synced: (38 rows)

Selecting from rpmfiles on app01 is a little bit slow but it is acceptable. (well under a minute for all of these queries:

select * from rpmfiles where name like '%/bash';
select * from rpmfiles where name = '/bin/bash';
select * from rpmfiles where packagebuildid = 5254;

However, I think the reason is that the table is still small enough to be cached in memory. Hopefully since we only keep files for active releases we'll continue to fit within that constraint. But we'll have to watch it when we deploy it to production (where tables from other applications are being used at the same time -- and therefore kicking this tbale out of the cache).

The import script verbosity can be altered in pkgdb.cfg

[[[pkgdb-sync-yum]]]
level='ERROR'
qualname='pkgdb-sync-yum'
handlers=['error_out']

will suppress anything but errors. The remaining output consists of deprecation warnings and yum plugins info.

The import script running on all synced data spend majority of time in yum and not much left to optimise in our code. I'm going to commit some minor speed ups of the application import though.

The misformated the config excerpt should have looked like this:
{{{
[[[pkgdb-sync-yum]]]
level='ERROR'
qualname='pkgdb-sync-yum'
handlers=['error_out']
}}}

Update has occurred -- some minor problems 9broken links and such) remain and can be cleaned up at leisure. The import script is currently broken. Need to fix that before closing this ticket. I will deploy changes that mbacovsk created tomorrow to see if it brings things to a working state.

Import script is fixed. Thanks mbacovsk!