• solve ownership of /mnt/koji/{mash,tree} - it used to be a local "releng" group, so will be added back (as localreleng so it doesn't conflict with the global releng group)

• install fedmsg certs for build{branched,rawhide} (FIXED)
• install intltool, packagedb-cli, mutt - needed by build{branched,rawhide} to process comps (FIXED locally)
• increase capacity of the new hub in koji db, it will allow more newRepos to run in parallel, we had 6 on the less powerful old hub (FIXED by koji edit-host)
• update kojira config with correct certs (FIXED in ansible, kojira user is common to all kojis)
• /etc/mock/fedora-{branched,rawhide}-compose-i386.cfg is missing - needed by build{branched,rawhide} (FIXED locally by cloning fedora-rawhide-i386.cfg)
• installed /etc/cron.daily/koji-prune-work to clean /mnt/koji/work (FIXED locally)

Replying to [ticket:4840 sharkcz]:

changes to incorporate to ansible for secondary koji hubs
- add /mnt/koji export to /etc/httpd/conf.d/kojihub.conf

append the following default snippet
{{{

Also serve /mnt/koji

Alias /kojifiles "/mnt/koji/"

<Directory "/mnt/koji">
Options Indexes SymLinksIfOwnerMatch
#If your top /mnt/koji directory is not owned by the httpd user, then
#you will need to follow all symlinks instead, e.g.
#Options Indexes FollowSymLinks
AllowOverride None
Require all granted
#If you have httpd <= 2.2, you'll want the following two lines instead
#of the one above:
#Order allow,deny
#Allow from all
</Directory>
}}}

• disk space management in /tmp, koji-shadow stores downloaded srpms under /tmp/koji-shadow and when free disk space is lower than the amount configured in /etc/kojid/kojid.conf, the builder daemon stops accepting (newRepo) jobs

• proxy user is not set for the web UI
  {{{
  <Rathann> sharkcz: any idea why I can't login to s390.koji?
  <Rathann> using the same certificate as for primary koji
  <Rathann> AuthError: emailAddress=buildsys@fedoraproject.org,CN=s390.koji.fedoraproject.org,OU=Fedora Builders,O=Fedora Project,ST=North Carolina,C=US is not authorized to login other users
  }}}

Replying to [comment:4 sharkcz]:

• disk space management in /tmp, koji-shadow stores downloaded srpms under /tmp/koji-shadow and when free disk space is lower than the amount configured in /etc/kojid/kojid.conf, the builder daemon stops accepting (newRepo) jobs

we should move that to /var/tmp as on RHEL7 up and Fedora /tmp is on tmpfs by default.

I think that we should have a seperate box for running koji-shadow on, it could be shared amongst arches on one per arch. but I do not think it needs to live on the hub box.

Replying to [comment:3 sharkcz]:

Replying to [ticket:4840 sharkcz]:

changes to incorporate to ansible for secondary koji hubs
- add /mnt/koji export to /etc/httpd/conf.d/kojihub.conf

append the following default snippet
{{{

Also serve /mnt/koji

Alias /kojifiles "/mnt/koji/"

<Directory "/mnt/koji">
Options Indexes SymLinksIfOwnerMatch
#If your top /mnt/koji directory is not owned by the httpd user, then
#you will need to follow all symlinks instead, e.g.
#Options Indexes FollowSymLinks
AllowOverride None
Require all granted
#If you have httpd <= 2.2, you'll want the following two lines instead
#of the one above:
#Order allow,deny
#Allow from all
</Directory>
}}}

I do not think we should use this snippet. we do not export primary that way. we have all the directories in /mnt/koji exported in / for example
{{{
http://s390pkgs.fedoraproject.org/mash/
http://s390pkgs.fedoraproject.org/packages/ there is a bug in that it redirects to https on kojipkgs
http://s390pkgs.fedoraproject.org/scratch/
http://s390pkgs.fedoraproject.org/work/
http://s390pkgs.fedoraproject.org/compose/
http://s390pkgs.fedoraproject.org/repos/
}}}

Replying to [comment:6 ausil]:

Replying to [comment:4 sharkcz]:

• disk space management in /tmp, koji-shadow stores downloaded srpms under /tmp/koji-shadow and when free disk space is lower than the amount configured in /etc/kojid/kojid.conf, the builder daemon stops accepting (newRepo) jobs

we should move that to /var/tmp as on RHEL7 up and Fedora /tmp is on tmpfs by default.

agreed, will send a patch, although the infra hosts have the tmpfs mount disabled (at least the s390 has /tmp directly on /)

Replying to [comment:8 ausil]:

Replying to [comment:3 sharkcz]:

Replying to [ticket:4840 sharkcz]:

changes to incorporate to ansible for secondary koji hubs
- add /mnt/koji export to /etc/httpd/conf.d/kojihub.conf

append the following default snippet
{{{

Also serve /mnt/koji

Alias /kojifiles "/mnt/koji/"

<Directory "/mnt/koji">
Options Indexes SymLinksIfOwnerMatch
#If your top /mnt/koji directory is not owned by the httpd user, then
#you will need to follow all symlinks instead, e.g.
#Options Indexes FollowSymLinks
AllowOverride None
Require all granted
#If you have httpd <= 2.2, you'll want the following two lines instead
#of the one above:
#Order allow,deny
#Allow from all
</Directory>
}}}

I do not think we should use this snippet. we do not export primary that way. we have all the directories in /mnt/koji exported in / for example
{{{
http://s390pkgs.fedoraproject.org/mash/
http://s390pkgs.fedoraproject.org/packages/ there is a bug in that it redirects to https on kojipkgs
http://s390pkgs.fedoraproject.org/scratch/
http://s390pkgs.fedoraproject.org/work/
http://s390pkgs.fedoraproject.org/compose/
http://s390pkgs.fedoraproject.org/repos/
}}}

IIRC the whole /mnt/koji was a workaround, the "packages redirect" has been again added back from ansible master copy, will attach a patch

Where do we stand here? what needs still to be done?

Closing this out, if there's anything we should do, please re-oepn