Last modified 4 years ago Last modified on 03/04/13 12:10:05
		Automatic Bug Reporting Tool (ABRT)

		x.x. Overview

ABRT is the Automatic Bug Reporting Tool. ABRT consists of a daemon,
abrtd, which runs silently in the background most of the time. It springs
into action when an application crashes, or kernel oops is detected.
It then collects the relevant crash data such as a core file
if there is one, the crashing application's command line parameters,
and other data of forensic utility.

ABRT currently supports detection of crashes in applications
written in the C/C++ and Python languages, and kernel oopses.

ABRT is capable of reporting problems to a remote issue tracker.
Reporting can be configured to happen automatically at every detected
crash, or problem data can be stored locally, reviewed, reported,
and deleted manually by a user.

Implemented reporting tools can send problem data to a Bugzilla
database, RHTSupport site, upload it using FTP/SCP, email it,
or write to a file.

The part of ABRT which handles already existing problem data
(as opposed to, for example, creation of new problem data)
has been factored out into a separate subproject, libreport.
It provides a generic mechanism for analyzing and reporting
problems, and it is used by projects other than ABRT.

However, ABRT and libreport operation and configuration
are closely integrated, therefore we discuss them as one here.

		x.x. Packages

Libreport packages provide the following:

    Configurational files (/etc/libreport/*)
    C library (/usr/lib/*)
    report-cli tool
    report-gtk tool
    report-newt tool
    Python binding (/usr/lib/python*/site-packages/report/*)
    reporter-bugzilla tool and corresponding configuration files
    reporter-rhtsupport tool and corresponding configuration files
    reporter-upload tool and corresponding configuration files
    reporter-mailx tool and corresponding configuration files
    reporter-kerneloops tool and corresponding configuration files
    reporter-print tool and corresponding configuration files
    report tool (compatibility with old report package)

ABRT packages provide the following:

    C library used by other abrt packages (/usr/lib/libabrt*.so*)
    abrtd, abrt-server, abrt-action-save-package-data, abrt-handle-upload
    service files for abrt-ccpp service
    CCpp.conf, event configuration for analyzing of C/C++ crashes
    service files for abrt-oops service
    service files for abrt-oops service
    abrt-action-analyze-python tool
    Python hook integration (/usr/lib/python*/site-packages/*)
    abrt-retrace-client tool and corresponding configuration files
    icons, .desktop etc
    (FIXME: why there are two sets of icons?)
    Meta-package: by installing it, you will install all of the above.
    The package itself is empty.

		x.x. Installing ABRT

In order to use ABRT, first ensure the abrt-desktop package is installed on your system by
running, as root:
	# yum install abrt-desktop
For more information on installing packages with Yum, refer to Section "Installing

abrtd is typically configured to start up at boot time. You can check that it is
is running using the following command, as root:
	# service abrtd status
	abrt (pid 1559) is running...

If you receive "abrtd is stopped", you can start the abrtd service by running, as root:
	# service abrtd start
	Starting abrt daemon:  [ OK ]

You can ensure that the abrtd service initializes at startup time by running the following command, as
	# chkconfig abrtd on

If you want ABRT to catch C/C++ crashes, follow similar steps to check or configure abrt-ccpp

If you want ABRT to detect kernel oopses, follow similar steps to check or configure abrt-oops
service. Note that this service can't catch oopses which hang or reboot the machine immediately.

ABRT's applet can be started by hand by running the abrt-applet program as a normal user
when logged into your desktop session, or by arranging for it to be started when the GUI session is
initialized. For example, on the GNOME desktop, this can be configured in System → Preferences →
Startup Applications.

= Configure kernel panic detection =
 1. edit /etc/grub.conf and append crashkernel=128M to the line starting with "kernel"
 2. install package abrt-addon-vmcore
 3. reboot the machine

Detected kernel crashes will be stored in /var/spool/abrt as any other
detected problem

		x.x. Running ABRT

ABRT saves problem data in "problem report" format.

Problem report is stored as a directory with files named
analyzer, architecture, coredump, cmdline, executable, kernel,
os_release, reason, time, uid etc. During analysis, more files
can be created, such as backtrace.

		x.x.x. Using the Graphical User Interface

When problem data is saved, a broadcast D-Bus message is sent about it.
If abrt-applet is running, it receives this message and displays a red alarm icon
in the Notification Area. You can open the GUI application by clicking on this icon.
Alternatively, you can open the ABRT GUI application by clicking Applications ->
System Tools -> Automatic Bug Reporting Tool.

...<more detailed GUI explanation>...

		x.x.x. Using the Command Line Interface

Problem data saved by abrtd can be viewed, reported, and deleted using the command line interface.

To get a list of not yet reported problems, simply execute abrt-cli list:

	# abrt-cli list
	Directory:      /var/spool/abrt/pyhook-2011-05-06-15:13:13-3782
	count:          2
	executable:     /usr/bin/gnome-tweak-tool
	package:        gnome-tweak-tool-3.0.3-1.fc15
	time:           Fri 06 May 2011 03:13:13 PM CEST
	uid:            0
	Directory:      /var/spool/abrt/ccpp-2011-05-05-10:04:02-1893
	count:          1
	executable:     /usr/bin/gnome-control-center
	package:        control-center-
	time:           Thu 05 May 2011 10:04:02 AM CEST
	uid:            500

This output contains basic information about every problem. The UID field shows the ID of the user
which ran the program that caused the crash. The package field shows the name and version of the
package that contains the program, and the executable field shows the location of the binary
or script that crashed. The count field indicates how many times the same crash happened.

To report a certain problem, enter abrt-cli --report DIR or abrt-cli --r DIR, where
DIR is a directory with problem data.

	# abrt-cli report /var/spool/abrt/ccpp-2011-05-05-10:04:02-1893
	How you would like to analyze the problem?
	 1) Local GNU Debugger
	Select analyzer: _

ABRT analyzes the problem. This might take a while. After this, abrt-cli opens a text editor
with the content of the report. You can see what is being reported, and you can fill in
instructions on how to reproduce the problem and other comments.
You should also check the backtrace, because the backtrace might be sent to a public server
and viewed by anyone, depending on the reporter settings.

When you are done with editing data, save your changes and close the editor.
You will be asked which of the configured reporters you want to use to send the report.

	How would you like to report the problem?
         1) Logger
         2) Red Hat Customer Support (rhel only)
 	 3) Bugzilla (It's not available in rhel by default)
	Select reporter(s): _

If you do not want to report a certain problem, you can delete it.
To delete a certain problem data, use "abrt-cli rm DIR" command.

Note that ABRT performs a detection of duplicate crashes by comparing new crashes with all locally
saved crashes. For a repeating crash, ABRT requires you to act upon it only once. However, if you
delete the problem data of that crash, the next time this specific crash occurs, ABRT will treat it as a
new one: ABRT will alert you about it, prompt you to fill in a description, and report it.
This can be redundant. To avoid having ABRT to annoy you with repeated notification
about a recurring crash, don't delete its problem data and let duplicate detection to work its magic.

		x.x. Configuring ABRT

		x.x.x. /etc/libreport/*.conf

Problem report goes through "events" in its life. For example:
(1) it is created, (2) analyzed, (3) reported to Bugzilla.

The set of possible events is not fixed - they can be added as
needed by the admin.

The main configuration file which specifies how problem reports
should be processed is /etc/libreport/report_event.conf.

report_event.conf file supplied by standard installation
contains the following include directive:

    include events.d/*.conf

which pulls in configuration added by other packages.

Upon installation, ABRT and libreport packages add their respective
configuration files into /etc/libreport/events.d/.

Processing of a problem report is done by "running events" on it.

/etc/libreport/report_event.conf and files included from it describe
what should be done on each event. Example:

EVENT=report_Bugzilla analyzer=Python
	reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf

This method of configuration superseeds many aspects
of abrt.conf in ABRT 1.x

The following event names are used by ABRT and libreport:

    post-create: run by abrtd on newly created problem data directories.
        If after running this event abrtd will detect that new
        problem data has the same UUID (hash) element
        as one of the existing ones, it will delete the new problem.

    analyze_SFX: used to process collected data. For example,
        analyze_LocalGDB event runs gdb on core dump and produces backtrace.
        abrt-gui shows the list of analyze events for the user to choose.

    collect_SFX: used to collect additional information.
        abrt-gui shows the list of collect events
        and lets user select which ones user wants to run.

    report_SFX: used to report the problem.
        abrt-gui shows the list of report events for the user to choose.

Programs which use libreport internally may define and use other event names.

System administrator's task is to define actions which should be taken
when these events are run, and/or to define additional analyze, collect
and report events if this is needed.

/etc/libreport/report_event.conf and files included from it
consist of directives and rules.

Directives start with a reserved word. Currently, there is only one
directive, "include". Its format is "include FILE".
It causes files which match FILE to be read and
parsed as if they are inserted textually where this directive
occurs. FILE can use shell pattern metacharacters (*,?,etc) to
specify multiple files. Relative paths are interpreted relative
to current file.

Rule starts with a line with non-space leading character.
All subsequent lines which start with space or tab form one rule.

Rules specify which programs to run on the dump directory.
Each rule may have conditions to be checked before the program is run.

Conditions have form VAR=VAL, VAR!=VAL or VAL~=REGEX, where VAR
is either word "EVENT" or a name of dump directory element to be checked
(for example, "executable", "package", hostname" etc).

If all conditions match, the remaining part of the rule
(the "program" part) is run in the shell.
All shell language constructs are valid.
All stdout and stderr output is captured and passed to ABRT
and possibly to abrt's frontends and shown to the user.

If the program terminates with nonzero exit code,
the event processing is considered unsuccessful and is stopped.
Last captured output line, if any, is considered to be
the error message indicating the reason of the failure,
and may be used by ABRT as such.

If the program terminates successfully, next rule is read
and processed. This process is repeated until there are no more
matching rules.

		x.x.x. /etc/abrt/*.conf

The following configuration directives are supported in /etc/abrt/abrt.conf:

	WatchCrashdumpArchiveDir = /var/spool/abrt-upload/

This directive is commented out by default. Enable (uncomment) it if you want abrtd to auto-
unpack crashdump tarballs which appear in the specified directory — in this case /var/spool/
abrt-upload/ — (for example, uploaded via ftp, scp, etc.). You must ensure that whatever
directory you specify in this directive exists and is writable for abrtd. abrtd will not create it
automatically. If you are using SElinux consult abrt_selinux manpage (man abrt_selinux).

	MaxCrashReportsSize = <size_in_megabytes>

This option sets the amount of storage space, in megabytes, used by ABRT to store all crash
information from all users. The default setting is 1000 MB. Once the quota specified here has
been exceeded, ABRT will continue catching crashes, and in order to make room for the new crash
dumps, it will delete the oldest and largest ones.

The following configuration directives are supported in /etc/abrt/abrt-action-save-package-data.conf:

	OpenGPGCheck = YES/NO

Setting the OpenGPGCheck directive to yes (the default setting) tells ABRT to only analyze and
handle crashes in applications provided by packages which are signed by the GPG keys whose
locations are listed in the /etc/abrt/gpg_keys file. Setting OpenGPGCheck to no tells ABRT
to catch crashes in programs from all packages.

	BlackList = nspluginwrapper, valgrind, strace[, MORE_PACKAGES]

Crashes in packages and binaries listed after the BlackList directive will not be handled by
ABRT. If you want ABRT to ignore other packages and binaries, list them here separated by

	ProcessUnpackaged = YES/NO

This directive tells ABRT whether to process crashes in executables that do not belong to any

	BlackListedPaths = /usr/share/doc/*, */example*

Crashes in executables in these paths will be ignored by ABRT.

The following configuration directives are supported in /etc/abrt/plugins/CCpp.conf:

	MakeCompatCore = YES/NO

Specifies whether ABRT's core catching hook should also create a core file
in a way similar to the case when ABRT is not installed (typically in the
current directory of the crashed program, and only if ulimit -c setting allows it).
Default is "yes".

	SaveBinaryImage = YES/NO

Specifies whether ABRT's core catching hook should save the binary's image
in addition to core dump. It may be useful if you want to be better equipped
for debugging of crashes which occurred in the binaries which have been deleted.
Default is "no".

		x.x.x. Configuring Automatic Reporting

ABRT can be configured to automatically report any detected issues or crashes,
without user interaction. This can be achieved by changing analyze and report rules
into post-create rules. For example, standard installation contains the following

	EVENT=report_Bugzilla analyzer=Python
	    test -f component || abrt-action-save-package-data
	    reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf

By replacing "EVENT=report_Bugzilla" by "EVENT=post-create",
you can instruct ABRT to report Python crashes to Bugzilla immediately
and without user interaction. (Note that post-create event is run
by abrtd, which usually runs under root and the required credentials must be set in 
the conf file).

		x.x.x. Uploading/reporting using proxy servers

reporter-bugzilla and reporter-upload tools respect http_proxy
and ftp_proxy environment variables.

When you run them as part of a reporting event, environment variables
are inherited from the process which performs reporting: usually, ABRT's
GUI or cli. Therefore you can use traditional method of specifying HTTP
or FTP proxy servers by setting these variables in your working environment.

If you arrange these tools to be as part of a post-create event,
they are run as children of abrtd process. In this case, you should either
modify abrtd's environment, or modify the rules to set these variables.

	EVENT=post-create analyzer=Python
	    test -f component || abrt-action-save-package-data
	    export http_proxy=http://proxy.server:8888/
	    reporter-bugzilla -c /etc/abrt/plugins/Bugzilla.conf

		x.x. Configuring Centralized Crash Collection

You can set up ABRT so that crash reports are collected from multiple systems and sent to a
dedicated system for further processing. This is useful when an administrator does not want to
log into hundreds of systems and manually check for crashes found by ABRT. In order to use this
method, you need to install the libreport-plugin-reportuploader plugin (yum install

The steps to configure ABRT's centralized crash collection are:

	x.x.x Complete the following steps on a dedicated system ("server system"):

Create a directory to which you want the crash reports to be uploaded to. Usually, /var/
spool/abrt-upload/ is used (the rest of the document assumes you are using this directory).
Make sure this directory is writable by the abrt user.

In the /etc/abrt/abrt.conf configuration file, set the WatchCrashdumpArchiveDir
directive to the following:

	WatchCrashdumpArchiveDir = /var/spool/abrt-upload/

Choose your preferred upload mechanism; for example, FTP or SCP. For more information
on how to configure FTP, refer to Section "FTP". For more information on how to configure
SCP, refer to Section "Using the scp Utility".
For security reasons, make sure that uploads can only be performed by a specific user and
with a password. The rest of the document assumes that the username used for uploads is
USERNAME and the password is PASSWORD. If you do not already have a suitable username
which can be used to perform uploads under, you may use the abrt user which already exists
on every system where ABRT is installed.
It is advisable to check whether your upload method works. For example, if you use ftp,
upload a file using an interactive FTP client:

	$ ftp
	ftp> open SERVERNAME
	Password: PASSWORD
	ftp> cd /var/spool/abrt-upload
	250 Operation successful
	ftp> put TESTFILE
	ftp> quit

Check whether TESTFILE appeared in the correct directory on the server system.

It is advisable to check and modify the MaxCrashReportsSize directive
in /etc/abrt/abrt.conf. It needs to be set to a larger value if the expected
volume of crash data is larger than the default 1000 MB.

Consider whether, and where, you want to generate backtrace of C/C++ crashes.

If you don't want to generate backtrace at all, or if you decide to generate
it locally on the machine where crash occurred, you need to disable
backtrace generation on the server. In standard ABRT installation,
backtrace is generated by this rule in ccpp_events.conf:

EVENT=analyze_LocalGDB analyzer=CCpp --core=coredump -o build_ids &&
        abrt-action-install-debuginfo-to-abrt-cache --size_mb=4096 &&
        abrt-action-generate-backtrace &&

By adding condition "remote!=1", you can make sure these rules don't apply
for uploaded problem data. (Uploaded data has remote=1, all other data
has no "remote" element).

Consider whether, and where, you want to collect package information
("package" and "component" elements in problem data). If you plan
to sent your problem reports to Bugzilla or RHTSupport,
this information is mandatory.

Out-of-the-box configuration will run abrt-action-save-package-data
immediately after crash. In most cases this is acceptable. If it is not,
modify or remove the rule which runs abrt-action-save-package-data.

The ProcessUnpackaged directive in /etc/abrt/abrt-action-save-package-data.conf
may need to be set to "yes": server system may lack packages
installed on the systems which upload problem data. If uploaded C/C++ crash
happened in a packages which is not installed on server machine and
ProcessUnpackaged is set to "no", abrt-action-save-package-data will
decide to delete this problem data.

If you decided to collect package information on client machines,
another alternative exists: modify the rule which runs

	EVENT=post-create component=            

so that it doesn't run on data uploaded from remote machines:

	EVENT=post-create remote!=1 component=            

	x.x.x. Complete the following steps on every client system
	which will use the central management method.

If you don't want to generate backtrace at all, or if you decide to generate
it on a server, you need to disable backtrace generation on clients.
This can be achieved by deleting or commenting out of the corresponding rules
in ccpp_events.conf.

If you decided to not collect package information on client machines,
delete or comment out the rule which runs abrt-action-save-package-data.

Add a rule to upload problem reports to the server.
For example, if you want to automatically and immediately upload
all problem reports as soon as they are detected, you can use the following rule:

        reporter-upload -u scp://

If you want to make upload to be one of the reporters for
the user who uses abrt-cli or abrt-gui on the client machine, you can use
rule similar to this one:

        reporter-upload -u scp://

TODO: what if I want to delete copied reports?

	x.x.x. Testing ABRT's Crash Detection

After completing all the steps of the configuration process, the basic setup is finished. To test that this
setup works properly use the kill -s SEGV PID command to terminate a process on a client
system. For example, start a sleep process and terminate it with the kill command in the following

	$ sleep 100 &
	[1] 2823
	$ kill -s SEGV 2823

ABRT should detect a simulated crash shortly after executing the kill command. Check that the crash was
detected by ABRT on the client system (this can be checked by examining the appropriate syslog file,
by running the abrt-cli --list --full command, or by examining the crash dump created in
the /var/spool/abrt directory), copied to the server system, unpacked on the server system and
can be seen and acted upon using abrt-cli or abrt-gui on the server system.