Automatic Bug Reporting Tool (ABRT) x.x. Overview ABRT is the Automatic Bug Reporting Tool. ABRT consists of a daemon, abrtd, which runs silently in the background most of the time. It springs into action when an application crashes, or kernel oops is detected. It then collects the relevant crash data such as a core file if there is one, the crashing application's command line parameters, and other data of forensic utility. ABRT currently supports detection of crashes in applications written in the C/C++ and Python languages, and kernel oopses. ABRT is capable of reporting problems to a remote issue tracker. Reporting can be configured to happen automatically at every detected crash, or problem data can be stored locally, reviewed, reported, and deleted manually by a user. Implemented reporting tools can send problem data to a Bugzilla database, RHTSupport site, upload it using FTP/SCP, email it, or write to a file. The part of ABRT which handles already existing problem data (as opposed to, for example, creation of new problem data) has been factored out into a separate subproject, libreport. It provides a generic mechanism for analyzing and reporting problems, and it is used by projects other than ABRT. However, ABRT and libreport operation and configuration are closely integrated, therefore we discuss them as one here. x.x. Packages Libreport packages provide the following: libreport Configurational files (/etc/libreport/*) C library (/usr/lib/libreport.so*) libreport-cli report-cli tool libreport-gtk report-gtk tool libreport-newt report-newt tool libreport-python Python binding (/usr/lib/python*/site-packages/report/*) libreport-plugin-bugzilla reporter-bugzilla tool and corresponding configuration files libreport-plugin-rhtsupport reporter-rhtsupport tool and corresponding configuration files libreport-plugin-reportuploader reporter-upload tool and corresponding configuration files libreport-plugin-mailx reporter-mailx tool and corresponding configuration files libreport-plugin-kerneloops reporter-kerneloops tool and corresponding configuration files libreport-plugin-logger reporter-print tool and corresponding configuration files libreport-compat report tool (compatibility with old report package) ABRT packages provide the following: abrt-libs C library used by other abrt packages (/usr/lib/libabrt*.so*) abrt abrtd, abrt-server, abrt-action-save-package-data, abrt-handle-upload /etc/abrt/* abrt-addon-ccpp service files for abrt-ccpp service CCpp.conf, event configuration for analyzing of C/C++ crashes abrt-action-analyze-core abrt-action-list-dsos abrt-action-analyze-c abrt-action-analyze-backtrace abrt-action-generate-backtrace abrt-action-install-debuginfo abrt-action-install-debuginfo-to-abrt-cache abrt-hook-ccpp abrt-addon-kerneloops service files for abrt-oops service abrt-dump-oops abrt-action-analyze-oops abrt-addon-vmcore service files for abrt-oops service abrt-action-analyze-vmcore abrt-addon-python abrt-action-analyze-python tool Python hook integration (/usr/lib/python*/site-packages/*) abrt-retrace-client abrt-retrace-client tool and corresponding configuration files abrt-cli abrt-cli abrt-gui abrt-applet abrt-gui icons, .desktop etc (FIXME: why there are two sets of icons?) abrt-desktop Meta-package: by installing it, you will install all of the above. The package itself is empty. x.x. Installing ABRT In order to use ABRT, first ensure the abrt-desktop package is installed on your system by running, as root: # yum install abrt-desktop For more information on installing packages with Yum, refer to Section "Installing Packages". abrtd is typically configured to start up at boot time. You can check that it is is running using the following command, as root: # service abrtd status abrt (pid 1559) is running... If you receive "abrtd is stopped", you can start the abrtd service by running, as root: # service abrtd start Starting abrt daemon: [ OK ] You can ensure that the abrtd service initializes at startup time by running the following command, as root: # chkconfig abrtd on If you want ABRT to catch C/C++ crashes, follow similar steps to check or configure abrt-ccpp service. If you want ABRT to detect kernel oopses, follow similar steps to check or configure abrt-oops service. Note that this service can't catch oopses which hang or reboot the machine immediately. ABRT's applet can be started by hand by running the abrt-applet program as a normal user when logged into your desktop session, or by arranging for it to be started when the GUI session is initialized. For example, on the GNOME desktop, this can be configured in System → Preferences → Startup Applications. = Configure kernel panic detection = 1. edit /etc/grub.conf and append crashkernel=128M to the line starting with "kernel" 2. install package abrt-addon-vmcore 3. reboot the machine Detected kernel crashes will be stored in /var/spool/abrt as any other detected problem x.x. Running ABRT ABRT saves problem data in "problem report" format. Problem report is stored as a directory with files named analyzer, architecture, coredump, cmdline, executable, kernel, os_release, reason, time, uid etc. During analysis, more files can be created, such as backtrace. x.x.x. Using the Graphical User Interface When problem data is saved, a broadcast D-Bus message is sent about it. If abrt-applet is running, it receives this message and displays a red alarm icon in the Notification Area. You can open the GUI application by clicking on this icon. Alternatively, you can open the ABRT GUI application by clicking Applications -> System Tools -> Automatic Bug Reporting Tool. ...<more detailed GUI explanation>... x.x.x. Using the Command Line Interface Problem data saved by abrtd can be viewed, reported, and deleted using the command line interface. To get a list of not yet reported problems, simply execute abrt-cli list: # abrt-cli list 5. Directory: /var/spool/abrt/pyhook-2011-05-06-15:13:13-3782 count: 2 executable: /usr/bin/gnome-tweak-tool package: gnome-tweak-tool-3.0.3-1.fc15 time: Fri 06 May 2011 03:13:13 PM CEST uid: 0 9. Directory: /var/spool/abrt/ccpp-2011-05-05-10:04:02-1893 count: 1 executable: /usr/bin/gnome-control-center package: control-center-18.104.22.168-4.fc15 time: Thu 05 May 2011 10:04:02 AM CEST uid: 500 This output contains basic information about every problem. The UID field shows the ID of the user which ran the program that caused the crash. The package field shows the name and version of the package that contains the program, and the executable field shows the location of the binary or script that crashed. The count field indicates how many times the same crash happened. To report a certain problem, enter abrt-cli --report DIR or abrt-cli --r DIR, where DIR is a directory with problem data. # abrt-cli report /var/spool/abrt/ccpp-2011-05-05-10:04:02-1893 How you would like to analyze the problem? 1) Local GNU Debugger Select analyzer: _ ABRT analyzes the problem. This might take a while. After this, abrt-cli opens a text editor with the content of the report. You can see what is being reported, and you can fill in instructions on how to reproduce the problem and other comments. You should also check the backtrace, because the backtrace might be sent to a public server and viewed by anyone, depending on the reporter settings. When you are done with editing data, save your changes and close the editor. You will be asked which of the configured reporters you want to use to send the report. How would you like to report the problem? 1) Logger 2) Red Hat Customer Support (rhel only) 3) Bugzilla (It's not available in rhel by default) Select reporter(s): _ If you do not want to report a certain problem, you can delete it. To delete a certain problem data, use "abrt-cli rm DIR" command. Note that ABRT performs a detection of duplicate crashes by comparing new crashes with all locally saved crashes. For a repeating crash, ABRT requires you to act upon it only once. However, if you delete the problem data of that crash, the next time this specific crash occurs, ABRT will treat it as a new one: ABRT will alert you about it, prompt you to fill in a description, and report it. This can be redundant. To avoid having ABRT to annoy you with repeated notification about a recurring crash, don't delete its problem data and let duplicate detection to work its magic. x.x. Configuring ABRT x.x.x. /etc/libreport/*.conf Problem report goes through "events" in its life. For example: (1) it is created, (2) analyzed, (3) reported to Bugzilla. The set of possible events is not fixed - they can be added as needed by the admin. The main configuration file which specifies how problem reports should be processed is /etc/libreport/report_event.conf. report_event.conf file supplied by standard installation contains the following include directive: include events.d/*.conf which pulls in configuration added by other packages. Upon installation, ABRT and libreport packages add their respective configuration files into /etc/libreport/events.d/. Processing of a problem report is done by "running events" on it. /etc/libreport/report_event.conf and files included from it describe what should be done on each event. Example: EVENT=report_Bugzilla analyzer=Python reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf This method of configuration superseeds many aspects of abrt.conf in ABRT 1.x The following event names are used by ABRT and libreport: post-create: run by abrtd on newly created problem data directories. If after running this event abrtd will detect that new problem data has the same UUID (hash) element as one of the existing ones, it will delete the new problem. analyze_SFX: used to process collected data. For example, analyze_LocalGDB event runs gdb on core dump and produces backtrace. abrt-gui shows the list of analyze events for the user to choose. collect_SFX: used to collect additional information. abrt-gui shows the list of collect events and lets user select which ones user wants to run. report_SFX: used to report the problem. abrt-gui shows the list of report events for the user to choose. Programs which use libreport internally may define and use other event names. System administrator's task is to define actions which should be taken when these events are run, and/or to define additional analyze, collect and report events if this is needed. /etc/libreport/report_event.conf and files included from it consist of directives and rules. Directives start with a reserved word. Currently, there is only one directive, "include". Its format is "include FILE". It causes files which match FILE to be read and parsed as if they are inserted textually where this directive occurs. FILE can use shell pattern metacharacters (*,?,etc) to specify multiple files. Relative paths are interpreted relative to current file. Rule starts with a line with non-space leading character. All subsequent lines which start with space or tab form one rule. Rules specify which programs to run on the dump directory. Each rule may have conditions to be checked before the program is run. Conditions have form VAR=VAL, VAR!=VAL or VAL~=REGEX, where VAR is either word "EVENT" or a name of dump directory element to be checked (for example, "executable", "package", hostname" etc). If all conditions match, the remaining part of the rule (the "program" part) is run in the shell. All shell language constructs are valid. All stdout and stderr output is captured and passed to ABRT and possibly to abrt's frontends and shown to the user. If the program terminates with nonzero exit code, the event processing is considered unsuccessful and is stopped. Last captured output line, if any, is considered to be the error message indicating the reason of the failure, and may be used by ABRT as such. If the program terminates successfully, next rule is read and processed. This process is repeated until there are no more matching rules. x.x.x. /etc/abrt/*.conf The following configuration directives are supported in /etc/abrt/abrt.conf: WatchCrashdumpArchiveDir = /var/spool/abrt-upload/ This directive is commented out by default. Enable (uncomment) it if you want abrtd to auto- unpack crashdump tarballs which appear in the specified directory — in this case /var/spool/ abrt-upload/ — (for example, uploaded via ftp, scp, etc.). You must ensure that whatever directory you specify in this directive exists and is writable for abrtd. abrtd will not create it automatically. If you are using SElinux consult abrt_selinux manpage (man abrt_selinux). MaxCrashReportsSize = <size_in_megabytes> This option sets the amount of storage space, in megabytes, used by ABRT to store all crash information from all users. The default setting is 1000 MB. Once the quota specified here has been exceeded, ABRT will continue catching crashes, and in order to make room for the new crash dumps, it will delete the oldest and largest ones. The following configuration directives are supported in /etc/abrt/abrt-action-save-package-data.conf: OpenGPGCheck = YES/NO Setting the OpenGPGCheck directive to yes (the default setting) tells ABRT to only analyze and handle crashes in applications provided by packages which are signed by the GPG keys whose locations are listed in the /etc/abrt/gpg_keys file. Setting OpenGPGCheck to no tells ABRT to catch crashes in programs from all packages. BlackList = nspluginwrapper, valgrind, strace[, MORE_PACKAGES] Crashes in packages and binaries listed after the BlackList directive will not be handled by ABRT. If you want ABRT to ignore other packages and binaries, list them here separated by commas. ProcessUnpackaged = YES/NO This directive tells ABRT whether to process crashes in executables that do not belong to any package. BlackListedPaths = /usr/share/doc/*, */example* Crashes in executables in these paths will be ignored by ABRT. The following configuration directives are supported in /etc/abrt/plugins/CCpp.conf: MakeCompatCore = YES/NO Specifies whether ABRT's core catching hook should also create a core file in a way similar to the case when ABRT is not installed (typically in the current directory of the crashed program, and only if ulimit -c setting allows it). Default is "yes". SaveBinaryImage = YES/NO Specifies whether ABRT's core catching hook should save the binary's image in addition to core dump. It may be useful if you want to be better equipped for debugging of crashes which occurred in the binaries which have been deleted. Default is "no". x.x.x. Configuring Automatic Reporting ABRT can be configured to automatically report any detected issues or crashes, without user interaction. This can be achieved by changing analyze and report rules into post-create rules. For example, standard installation contains the following rule: EVENT=report_Bugzilla analyzer=Python test -f component || abrt-action-save-package-data reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf By replacing "EVENT=report_Bugzilla" by "EVENT=post-create", you can instruct ABRT to report Python crashes to Bugzilla immediately and without user interaction. (Note that post-create event is run by abrtd, which usually runs under root and the required credentials must be set in the conf file). x.x.x. Uploading/reporting using proxy servers reporter-bugzilla and reporter-upload tools respect http_proxy and ftp_proxy environment variables. When you run them as part of a reporting event, environment variables are inherited from the process which performs reporting: usually, ABRT's GUI or cli. Therefore you can use traditional method of specifying HTTP or FTP proxy servers by setting these variables in your working environment. If you arrange these tools to be as part of a post-create event, they are run as children of abrtd process. In this case, you should either modify abrtd's environment, or modify the rules to set these variables. Example: EVENT=post-create analyzer=Python test -f component || abrt-action-save-package-data export http_proxy=http://proxy.server:8888/ reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf x.x. Configuring Centralized Crash Collection You can set up ABRT so that crash reports are collected from multiple systems and sent to a dedicated system for further processing. This is useful when an administrator does not want to log into hundreds of systems and manually check for crashes found by ABRT. In order to use this method, you need to install the libreport-plugin-reportuploader plugin (yum install libreport-plugin-reportuploader). The steps to configure ABRT's centralized crash collection are: x.x.x Complete the following steps on a dedicated system ("server system"): Create a directory to which you want the crash reports to be uploaded to. Usually, /var/ spool/abrt-upload/ is used (the rest of the document assumes you are using this directory). Make sure this directory is writable by the abrt user. In the /etc/abrt/abrt.conf configuration file, set the WatchCrashdumpArchiveDir directive to the following: WatchCrashdumpArchiveDir = /var/spool/abrt-upload/ Choose your preferred upload mechanism; for example, FTP or SCP. For more information on how to configure FTP, refer to Section "FTP". For more information on how to configure SCP, refer to Section "Using the scp Utility". For security reasons, make sure that uploads can only be performed by a specific user and with a password. The rest of the document assumes that the username used for uploads is USERNAME and the password is PASSWORD. If you do not already have a suitable username which can be used to perform uploads under, you may use the abrt user which already exists on every system where ABRT is installed. It is advisable to check whether your upload method works. For example, if you use ftp, upload a file using an interactive FTP client: $ ftp ftp> open SERVERNAME Name: USERNAME Password: PASSWORD ftp> cd /var/spool/abrt-upload 250 Operation successful ftp> put TESTFILE ftp> quit Check whether TESTFILE appeared in the correct directory on the server system. It is advisable to check and modify the MaxCrashReportsSize directive in /etc/abrt/abrt.conf. It needs to be set to a larger value if the expected volume of crash data is larger than the default 1000 MB. Consider whether, and where, you want to generate backtrace of C/C++ crashes. If you don't want to generate backtrace at all, or if you decide to generate it locally on the machine where crash occurred, you need to disable backtrace generation on the server. In standard ABRT installation, backtrace is generated by this rule in ccpp_events.conf: EVENT=analyze_LocalGDB analyzer=CCpp abrt-action-analyze-core.py --core=coredump -o build_ids && abrt-action-install-debuginfo-to-abrt-cache --size_mb=4096 && abrt-action-generate-backtrace && abrt-action-analyze-backtrace By adding condition "remote!=1", you can make sure these rules don't apply for uploaded problem data. (Uploaded data has remote=1, all other data has no "remote" element). Consider whether, and where, you want to collect package information ("package" and "component" elements in problem data). If you plan to sent your problem reports to Bugzilla or RHTSupport, this information is mandatory. Out-of-the-box configuration will run abrt-action-save-package-data immediately after crash. In most cases this is acceptable. If it is not, modify or remove the rule which runs abrt-action-save-package-data. The ProcessUnpackaged directive in /etc/abrt/abrt-action-save-package-data.conf may need to be set to "yes": server system may lack packages installed on the systems which upload problem data. If uploaded C/C++ crash happened in a packages which is not installed on server machine and ProcessUnpackaged is set to "no", abrt-action-save-package-data will decide to delete this problem data. If you decided to collect package information on client machines, another alternative exists: modify the rule which runs abrt-action-save-package-data: EVENT=post-create component= abrt-action-save-package-data so that it doesn't run on data uploaded from remote machines: EVENT=post-create remote!=1 component= abrt-action-save-package-data x.x.x. Complete the following steps on every client system which will use the central management method. If you don't want to generate backtrace at all, or if you decide to generate it on a server, you need to disable backtrace generation on clients. This can be achieved by deleting or commenting out of the corresponding rules in ccpp_events.conf. If you decided to not collect package information on client machines, delete or comment out the rule which runs abrt-action-save-package-data. Add a rule to upload problem reports to the server. For example, if you want to automatically and immediately upload all problem reports as soon as they are detected, you can use the following rule: EVENT=post-create reporter-upload -u scp://user:email@example.com/dir/ If you want to make upload to be one of the reporters for the user who uses abrt-cli or abrt-gui on the client machine, you can use rule similar to this one: EVENT=report_UploadToMyServer reporter-upload -u scp://user:firstname.lastname@example.org/dir/ TODO: what if I want to delete copied reports? x.x.x. Testing ABRT's Crash Detection After completing all the steps of the configuration process, the basic setup is finished. To test that this setup works properly use the kill -s SEGV PID command to terminate a process on a client system. For example, start a sleep process and terminate it with the kill command in the following way: $ sleep 100 &  2823 $ kill -s SEGV 2823 ABRT should detect a simulated crash shortly after executing the kill command. Check that the crash was detected by ABRT on the client system (this can be checked by examining the appropriate syslog file, by running the abrt-cli --list --full command, or by examining the crash dump created in the /var/spool/abrt directory), copied to the server system, unpacked on the server system and can be seen and acted upon using abrt-cli or abrt-gui on the server system.