Ticket #9 (closed task: fixed)

Opened 5 years ago

Last modified 3 years ago

Package autotest

Reported by: wwoods Owned by: jkeating
Priority: major Milestone:
Component: packaging Keywords:
Cc: jkeating, lmr, Blocked By:
Blocking:

Description

Package up at least the server/web frontend parts of autotest: http://autotest.kernel.org/

Note that this will probably require license review.

Attachments

0003-Adjust-apache-config-paths-for-a-Red-Hat-Fedora-styl.patch (5.8 KB) - added by jlaska 5 years ago.
0003-Adjust-apache-config-paths-for-a-Red-Hat-Fedora-styl.patch

Change History

comment:1 Changed 5 years ago by wwoods

  • Priority changed from blocker to major

comment:2 Changed 5 years ago by jlaska

  • Cc jkeating, lrodrigu added

Adding Jesse (fedora release engineering) and Lucas (upstream autotest) to the cc-list as both are actively working to package autotest. This ticket is meant to track status of the effort as it relates to the fedora autoqa project, any patching or communication should probably take place upstream.

comment:3 Changed 5 years ago by jkeating

I spent a good deal of time on this over the past few days. I've got a good start on a spec that splits things into autotest and autotest-client. I've gone over the contents of the tarball with upstream and figured out what can be omitted and what has to be packaged, as well as many things that need to be patched to use system libraries. More work will continue.

comment:4 Changed 5 years ago by jlaska

  • Owner set to jkeating

Jesse is doing all the work on this so far, assigning ticket.

comment:5 follow-up: ↓ 7 Changed 5 years ago by jkeating

  • Status changed from new to assigned

I've uploaded some initial packages to http://jkeating.fedorapeople.org/review/ They need to be tested, and still require the last mile configuration done.

comment:6 Changed 5 years ago by jlaska

While testing autotest-client package yesterday, I noticed that the autotest server does a few host verification steps, and then rsync's the server's copy of client code over. This happens whether or not the autotest-client package is installed on the test host. I still need to understand a bit more about everything it's rsyncing over ... but making autotest more fedora-package aware might be a bit more involved (including upstream patches which you may have already talked about with Lucas)?

I think this ticket was intended to get something formally packaged on the server for use in a fedora-infrastructure hosted instance. With the packages you've provided for .el5, does that accomplish what we need for now?

comment:7 in reply to: ↑ 5 Changed 5 years ago by jlaska

Replying to jkeating:

I've uploaded some initial packages to http://jkeating.fedorapeople.org/review/ They need to be tested, and still require the last mile configuration done.

I've installed a RHEL5 system to test your autotest server package. I've had to install several dependencies from other projects (as well as rebuild a few). The only one I cannot satisfy is the %requires on autotest-client.

Download from brew ...

Rebuild for .el5 ....

So far the only missing dep is:

Error: Missing Dependency: autotest-client = 0.10.0-1.ep5.el5 is needed by package autotest-0.10.0-1.ep5.el5.noarch (autotest-0.10.0-1.ep5.el5.noarch.rpm)

comment:8 follow-up: ↓ 9 Changed 5 years ago by jkeating

ah, I guess I need to stuff the autotest-client build through brew as well for EL5. I've built and posted it in the same url.

comment:9 in reply to: ↑ 8 Changed 5 years ago by jlaska

Replying to jkeating:

ah, I guess I need to stuff the autotest-client build through brew as well for EL5. I've built and posted it in the same url.

I've installed the packages and am attempting to configure the server per the instructions at http://www.linux-kvm.org/page/KVM-Autotest/Server_Install#Server_Installation_Steps.

  • PASS - package install
  • WARN - need to ensure the deps and rebuilds I did in comment#5 are available for fedora infrastructure
  • WARN - autotest package %scripts should create an autotest user
    • All files should be owned by autotest user?
  • WARN - autotest package should also create an id_rsa for the autotest user?
  • WARN - should the autotest package also require ... java-1.6.0-openjdk-devel, mysql-server (which are needed for a complete autotest install).
  • WARN - need to include a httpd autotest.conf file
    # cat << EOF >/etc/httpd/conf.d/autotest.conf
    Include "/usr/share/autotest/apache/conf/all-directives"
    AddHandler cgi-script .cgi
    EOF
    
  • FAIL - all the apache .conf directives included with autotest are hard-coded to use /usr/local/autotest - this causes httpd to fail
    • I temporarily fixed this problem by doing ...
      # for FILE in /usr/share/autotest/apache/conf/* 
      do 
      sed -i "s|/usr/local|/usr/share|" $FILE ;
      done
      
  • FAIL - unable to run DB migrations since client/setup_modules.py is missing from the autotest and autotest-client packages.

comment:10 follow-up: ↓ 11 Changed 5 years ago by jkeating

I added an autotest user and set ownership of all files (except docs) to autotest.

I create an rsa key file set autotest-id_rsa(.pub) is created in /usr/share/autotest/

java requires is not necessary as its build time only

mysql-server requires is wrong because the db could live elsewhere

I patched the autotest apache config files (but I forgot to install the apache/apache-conf to apache)

I fixed the install scripts to pick up setup_modules.py (and a bunch of other files I was missing)

New packages uploaded, please try them.

comment:11 in reply to: ↑ 10 ; follow-up: ↓ 12 Changed 5 years ago by jlaska

New packages uploaded, please try them.

Reinstalled and retesting...

  • WARN - should we add a %requires for Django-0.96-1.el5
    • Running Django's syncdb script fails with anything but Django-0.96-1.el5 works correctly.
  • WARN - the documentation notes some SELinux changes. These seem to be required for proper operation. Not sure if these should be added to a README file, or if this can be handled in the packaging or %post* scripts?
semanage fcontext -a -t bin_t '${AUTOTESTDIR}/tko(/.*cgi)?'
restorecon ${AUTOTESTDIR}/tko/*
  • FAIL - What user do we want the autotest scheduler to run as? I've been running it as the user autotest, but with the current autotest user account info ... that doesn't work since logins are disabled for user autotest. Any thoughts here? I've marked this as FAIL since the scheduler can't run until we figure this part out.
    • Some of the python scripts intentionally fail if running as UID=0
  • WARN - Several additional locations hard-code the path /usr/local/autotest. Should we continue to carry a patch to move these to /usr/share/autotest or should we give up for now and leave it all in /usr/local/autotest? I'm starting to lean towards the latter option.
    • The file /usr/share/autotest/apache/conf/tko-directives still references /usr/local/autotest
    • The file /usr/share/autotest/server/autotest.py:644 - added /usr/share/autotest to list of paths
    • The file /usr/share/autotest/utils/test_importer.py - has /usr/local/autotest hard coded
    • The file /usr/share/autotest/client/tools/autotest:10 - added /usr/share/autotest to the list of paths
  • FAIL - I'm not able to get a successful job running with this autotest package yet. So far the issues I'm seeing are:
    • Need to resolve which user will be responsible for running the scheduler daemon (see above)
    • The output files under /usr/share/autotest/results need to be readable by apache.
    • Each job fails to run on the client system ... I'm still debugging why

Once we get something that works well, I'll attach the following files in the hopes we can include them in the packaging:

  • README.Fedora - this file will explain in detail the steps required to setup the database
  • /etc/init.d/autotest - this initscript will be responsible for starting and stopping the monitor_db.py scheduler daemon

comment:12 in reply to: ↑ 11 Changed 5 years ago by jlaska

Replying to jlaska:

  • FAIL - I'm not able to get a successful job running with this autotest package yet. So far the issues I'm seeing are:

Looks like we need to include the directory %{autotestdir}/client/tests in the autotest package (or autotest-client) for things to work as expected. With that change, I have managed to schedule a successful job.

comment:13 Changed 5 years ago by jkeating

-2 packages uploaded.

autotest:

  • Patch path issues
  • Set a shell for the autotest user to allow running init script
  • Fix ssh key generation to run as autotest user
  • SELinux fixes

client:

  • Package the client/tests dir
  • Patch /usr/local to /usr/share where it matters

comment:14 follow-up: ↓ 15 Changed 5 years ago by jlaska

Thanks for the fixes. More testing ...

  • INFO - should autotest %require gwt and /usr/bin/java? I think these are both required by the frontend.
  • FAIL - when attempting to compile the front-ends as recommended from the setup instructions, the command fails
       [ERROR] Unable to find 'autotest/AfeClient.gwt.xml' on your classpath; could be a typo, or maybe you forgot to include a classpath entry for source?
    

) may need updates

  1. Should that be called from %postin ?
  2. The APPDIR variable may need to be set according to file locations in the autotest package

comment:15 in reply to: ↑ 14 Changed 5 years ago by jlaska

Replying to jlaska:

  • FAIL - when attempting to compile the front-ends as recommended from the setup instructions, the command fails
       [ERROR] Unable to find 'autotest/AfeClient.gwt.xml' on your classpath; could be a typo, or maybe you forgot to include a classpath entry for source?
    

) may need updates

  1. Should that be called from %postin ?
  2. The APPDIR variable may need to be set according to file locations in the autotest package

Another thought, should the compilation of the above files be done during src.rpm build time, or left up to the sysadmin to perform on the running instance? It's not clear to me when/how frequent compiling the front-end is.

comment:16 Changed 5 years ago by jkeating

The compilation /is/ done at rpm build time. You will /not/ need to run the compile after installing the package. The only time the compile needs to be done is when the source .java files change, which is when upstream change them. I purposefully do not package the compile bits as they aren't needed for the operation of autotest. gwt and /usr/bin/java are not required for runtime of autotest server, only at build time and that's during the build of the srpm.

Are there any issues if you don't try to compile and use the pre-compiled bits in the rpm (basically skipping that step in the setup).

comment:17 Changed 5 years ago by jlaska

Looking good, not able to get the scheduler processing jobs just yet. Will need to revisit with a clearer head. Some minor issues observed so far:

  • WARN - the rpm %script sets the autotest shell to /sbin/bash (should be /bin/bash). I think this caused the rsa generation to fail since "su -c" fails to find shell /sbin/bash
  • WARN - The following files seem to still reference /usr/local/autotest
    • /usr/share/autotest/global_config.ini
    • /usr/share/autotest/apache/drone-conf
  • INFO - as for the RSA key, I'm not sure of a way to do this such that passwordless logins are enabled by default for configured autotest test clients. As it's done currently, when the scheduler (run as autotest) attempts to ssh to a client system, it will not offer the RSA key. Looking at base_utils.py at the method get_public_key() it's looking for a key in '~/.ssh'. Later in that function, if it wasn't able to find a key, it creates one. I'm not sure how to get this right just yet, but I'll continue understanding how autotest handles this.

comment:18 Changed 5 years ago by jlaska

  • WARN - /etc/httpd/conf.d/autotest.conf directs error output to /var/log/httpd/error.log, while the stock httpd configuration directs error output to /var/log/httpd/error_log.

comment:19 Changed 5 years ago by jlaska

  • FAIL - Appears I still need to manually adjust SELinux permissions for the files /usr/share/autotest/tko/*.cgi. The quick fix was:
    $ chcon -t bin_t /usr/share/autotest/tko/*.cgi
    
  • FAIL - The stock apache config that comes with autotest is missing a line to enable the .cgi handler. Adding the following to the top of /etc/httpd/conf.d/autotest.conf resolves the issue:
    AddHandler cgi-script .cgi
    

With all of the above changes (in this comment and the 2 prior) I am able to install autotest and schedule and execute jobs on a client system.

The outstanding issue I'm not clear on how to resolve is creating the RSA key for the autotest user. The following steps were needed to enable passwordless logins to test clients:

  1. Create a key in HOMEDIR /usr/share/autotest/.ssh/ by typing: ssh-keygen -q -t rsa -N ''
  2. Add ssh key to test clients: ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@client.example.com

I think the rpm %scripts need to to place the RSA key in /usr/share/autotest/.ssh/ so that the autotest server properly finds and uses the key when connecting to test clients.

My next task will be to reinstall the system and repeat the steps, provide the updated README.fedora, and attempt to get the initscript working again.

comment:20 Changed 5 years ago by jkeating

-3 server uploaded.

  • Move ssh key into autotest home .ssh/ and name it generically
  • Ghost the ssh dir
  • More selinux fixes

(- adjust apache config patch for proper paths and cgi declaration)

comment:21 Changed 5 years ago by jlaska

  • FAIL - autotest has a hard %requires on the autotest-client of the same NVR.
      --> Missing Dependency: autotest-client = 0.10.0-3.ep5.el5 is needed by package autotest-0.10.0-3.ep5.el5.noarch (autotest-0.10.0-3.ep5.el5.noarch.rpm)
    autotest-0.10.0-3.ep5.el5.noarch from autotest-0.10.0-3.ep5.el5.noarch.rpm has depsolving problems
    

I bumped the release of autotest-client and performed a scratch build to work around this issue for now (see https://brewweb.devel.redhat.com/taskinfo?taskID=1865487).

Continuing with testing ...

comment:22 Changed 5 years ago by jlaska

I've rebuild autotest and autotest-client to include:

  • Add patch: 0006-Background-scheduler.patch -- Not yet posted upstream.
  • Added README.fedora
  • Added /etc/rc.d/init.d/autotest
  • Created symlink /usr/sbin/monitor_db.py -> /usr/share/autotest/scheduler/monitor_db.py

The packages are available at:

With these packages ... everything is good on my end. I can install and configure the system, and start the autotest scheduler initscript.

comment:23 Changed 5 years ago by jlaska

src and noarch packages from previous comment available on my people page as well (http://jlaska.fedorapeople.org/autoqa/)

comment:24 Changed 5 years ago by jkeating

I've made some changes.

  • Make README.fedora a patch to the source code
  • Make initscript a patch to the source code
  • re-work background patch to be git compliant
  • Remove macros for install
  • Drop release level requirement on autotest-client. Version is good enough

I've also modified README.fedora a bit, referencing the fact that you can use an external mysql server and that the things you're installing with yum should be listed as Requires in the package itself.

This should continue to work for you, with your existing -client package (since that didn't change and I made the requirement less strict).

I'll start working on the rpmlint issues for Fedora packaging.

comment:25 Changed 5 years ago by jlaska

Spoke w/ jkeating about next steps around packaging and hosting in fedora-infrastructure.

Identified the following action items:

  • [jlaska] - create autoqa ticket to get packages into infrastructure repo (see ticket#32)
  • [jkeating] - will investigate getting packages into infrastructure repo
  • [jkeating] - will send an email upstream about making rsync configurable
  • [jlaska] - will confirm w/ mmcgrath whether ssh passwordless is a show stopper
  • [jlaska] - create fedora-infrastructure RFR for autoqa and autoqa-stage
    • local disk space - keep it local - 15G guest
    • local ram - 1024
    • Use QA1 as a test client
    • Also create a staging environment

comment:26 Changed 5 years ago by jlaska

Jkeating: I'm chasing down a few issues with regards to the test clients and reporting the test logs back to the server. It seems two additional empty directories are needed in the autotest-client package:

/usr/share/autotest/client/{profilers,deps}

I've manually created these directories, and it resolves several tracebacks while processing test results.

jkeating: Can you make this change to the autotest-client package?

I'm also chasing something down with regards to SELinux and the context of the test scripts initiated from a cronjob (as will be the case with the watchers). The context of the running process is getting EPERMDENIED when trying to rsync/scp test logs into /usr/share/autotest/results/....

comment:27 Changed 5 years ago by jkeating

Updated autotest:

  • Wed Jul 08 2009 Jesse Keating <jkeating@…> - 0.10.0-6
  • Move apache config files into /etc/
  • Drop some unneeded files
  • Set permissions accordingly
  • Remove unneeded #! and add a missing one

and autotest-client

  • Mon Jul 13 2009 Jesse Keating <jkeating@…> - 0.10.0-6
  • Patch for permissions and for spurious #!s
  • Don't package up setidle.c
  • Fri Jul 10 2009 Jesse Keating <jkeating@…> - 0.10.0-5
  • Package the deps and profilers dirs

have been uploaded. With these we have no rpmlint issues that cannot be ignored, so a good test from start to finish would be good.

comment:28 Changed 5 years ago by jlaska

  • Cc dpravec added

Are the package dependencies available in a fedora-infrastructure yum repo for testing as well?

comment:29 Changed 5 years ago by jlaska

jkeating, are the latest autotest package located externally? I don't see autotest-0.10.0-6 available on your fedora people page and it doesn't look like they are available yet in the infrastructure repos.

David Pravec is going to help out with inspecting the packages. I have found them internally through a brew scratch build and posted them to my people page. Let us know if those are not the correct builds.

comment:30 follow-up: ↓ 31 Changed 5 years ago by jlaska

jkeating provided an update repo link that contains autotest and the deps - http://jkeating.fedorapeople.org/infra. The remaining deps should be satisfied by the EL5 fedora-infrastructure repo.

comment:31 in reply to: ↑ 30 Changed 5 years ago by jlaska

Replying to jlaska:

jkeating provided an update repo link that contains autotest and the deps - http://jkeating.fedorapeople.org/infra. The remaining deps should be satisfied by the EL5 fedora-infrastructure repo.

Updated from Jkeating in ticket#32. To satisfy deps you'll need to have a system:

comment:32 Changed 5 years ago by jlaska

Looks like incorrect apache config paths. I haven't fully tested yet, however did see the following problem:

# /etc/init.d/httpd start
Starting httpd: httpd: Syntax error on line 210 of /etc/httpd/conf/httpd.conf: Syntax error on line 26 of /etc/httpd/conf.d/autotest.conf: Syntax error on line 5 of /etc/httpd/autotest.d/all-directives: Could not open configuration file /usr/share/autotest/apache/conf/django-directives: No such file or directory
                                                           [FAILED]

Since all directives are in /etc/httpd/autotest.d now, there is no need for /etc/httpd/autotest.d/all-directives.

comment:33 Changed 5 years ago by jkeating

Whoops, should have caught that. Fixed in -7 in the infra/ dir.

comment:34 Changed 5 years ago by dpravec

james laska found out that there are some files missing, especially /usr/share/autotest/frontend/afe/site_rpc_interface.py

$ rpmdiff autotest-0.10.0-6.el5.noarch.rpm autotest-0.10.0-7.el5.noarch.rpm  | grep removed
removed     REQUIRES config(autotest) = 0.10.0-6.el5
removed     REQUIRES config(autotest) = 0.10.0-6.el5
removed     /etc/httpd/autotest.d/all-directives
removed     /usr/share/autotest/frontend/afe/site_rpc_interface.py
removed     /usr/share/autotest/frontend/afe/site_rpc_interface.pyc
removed     /usr/share/autotest/frontend/afe/site_rpc_interface.pyo

I do not have older package versions, but i can confirm that 0.10.0-7.el5 is having problems. My frontpage gives me error:

ViewDoesNotExist at /afe/server/rpc/

and later in the (error) page is:

!Exception Value: Could not import frontend.afe.views. Error was: cannot import name site_rpc_interface

comment:35 Changed 5 years ago by jlaska

I have applied the following spec file change (see http://fpaste.org/paste/20506) and rebuilt the autotest package. An updated package is available from my people page (http://jlaska.fedorapeople.org/autoqa/autotest-0.10.0-8.el5.noarch.rpm). This may not be the final fix ... jkeating is communicated with the upstream #autotest folks for guidance.

Dpravec: this should be sufficient to continue with testing the autotest server.

comment:36 Changed 5 years ago by jlaska

Updated the autotest packages on my people page to use the latest build from Jesse (https://brewweb.devel.redhat.com/taskinfo?taskID=1913323).

http://jlaska.fedorapeople.org/autoqa/autotest-0.10.0-8.el5.noarch.rpm

comment:37 Changed 5 years ago by jlaska

Testing looks good so far. I am able to install the dependencies as noted in comment#30 and have the autotest service up and running, queueing and running jobs.

What's the next step?

  1. Create package review request for autotest?
  2. Push autotest package into fedora-infra repos?

comment:38 Changed 5 years ago by jlaska

Updated packages prepared by jkeating and posted to http://jkeating.fedorapeople.org/infra/

Instructions for setting up autotest available at https://fedoraproject.org/wiki/Install_and_configure_autotest

Testing inprogress ...

comment:39 Changed 5 years ago by jlaska

The error log used by autotest appears to have changed from error_log to error.log. This isn't a big isuse, just a change from autotest-0.10 and different from what I expected.

/etc/httpd/conf.d/autotest.conf: ErrorLog? /var/log/httpd/error.log

comment:40 follow-up: ↓ 41 Changed 5 years ago by jlaska

More testing of autotest-0.11 ...

Did the monitor_db.py --logfile patch no longer apply?

# /etc/init.d/autotest restart
Stopping autotest: [  OK  ]
Starting autotest: usage: monitor_db.py [options] results_dir

monitor_db.py: error: no such option: --logfile
[FAILED]

comment:41 in reply to: ↑ 40 Changed 5 years ago by jlaska

  • Cc lmr added; lrodrigu removed

Replying to jlaska:

More testing of autotest-0.11 ...

Did the monitor_db.py --logfile patch no longer apply?

nm ... the patch in question was for --background (that seems to be applied). It appears that the --logfile option is no longer available. The /etc/init.d/autotest initscript will need to be adjusted accordingly.

I'll attach that, but it appears that another 0.11 change now requires the scheduler daemon to run as root. I've spoken to Lucas about this. I believe Lucas will raise this with upstream shortly.

Once we hear back, I'll update if any initscript changes are needed.

comment:42 Changed 5 years ago by jlaska

Updated --background patch submitted upstream ... I'm 110% it's going to be cleaned up before being accepted.

Changed 5 years ago by jlaska

0003-Adjust-apache-config-paths-for-a-Red-Hat-Fedora-styl.patch

comment:43 Changed 5 years ago by jlaska

Also attached updated 0003-Adjust-apache-config-paths-for-a-Red-Hat-Fedora-styl.patch to address the issue noted in comment#39

comment:44 Changed 5 years ago by jlaska

Current todo list ...

  • Update patch 0003-Adjust-apache-config-paths-for-a-Red-Hat-Fedora-styl.patch - see attachment
  • Update patch 0004-Change-usr-local-to-usr-share.patch - Add global_config.ini drone_installation_directory: /usr/local/autotest
  • Update background script ... use monitor_db_babysitter instead - INPROGRESS

comment:45 Changed 4 years ago by jlaska

  • Cc dpravec removed

Okay, everything is working now using autotest-0.11. Updated just about all of the patches and rev'd new autotest-0.11.0-2 and autotest-client-0.11.0-2 packages. Packages available at ... http://jlaska.fedorapeople.org/autoqa

jkeating, can you review the changes (changelog updated in each package). If you're happy, I'll close this ticket out.

Is the next step to add these packages to the infrastructure repo?

comment:46 Changed 4 years ago by kparal

# rpm -hiv http://jlaska.fedorapeople.org/autoqa/autotest-client-0.11.0-2.el5.noarch.rpm
Retrieving http://jlaska.fedorapeople.org/autoqa/autotest-client-0.11.0-2.el5.noarch.rpm
Preparing...                ########################################### [100%]
   1:autotest-client        warning: user autotest does not exist - using root
warning: group autotest does not exist - using root
warning: user autotest does not exist - using root
warning: group autotest does not exist - using root
warning: user autotest does not exist - using root
[snip]
warning: user autotest does not exist - using root
warning: group autotest does not exist - using root
########################################### [100%]

comment:47 Changed 4 years ago by jlaska

so I guess that means autotest-client should be providing the autotest user, not the autotest server package. I'll update the packages.

Jesse: I know you're probably swamped with F-12 release stuff. What are the steps to follow to get the autotest* packages included in the fedora-infra repo?

comment:48 Changed 4 years ago by jlaska

  • Status changed from assigned to closed
  • Resolution set to fixed

Updated to autotest-0.11.0-4.el5 (see http://jlaska.fedorapeople.org/autoqa). Updated patch4 to include global_config.ini

* Thu Nov 19 2009 James Laska <jlaska@redhat.com> - 0.11.0-4
- Updated Patch4 (0004-Change-usr-local-to-usr-share.patch) so that
  global_config.ini also uses /usr/share/autotest

I'm closing this ticket out. This thing is packaged. I'll track adding the packages to fedora-infrastructure repos in ticket#32

comment:49 Changed 3 years ago by anonymous

  • Milestone israwhidebroken.com deleted

Milestone israwhidebroken.com deleted

Note: See TracTickets for help on using tickets.