wiki:WikiStart
Last modified 5 years ago Last modified on 05/25/09 01:33:17

"squeal" (formerly "show") is a mashup of SQL and the command-line. It lets you issue SQL-like queries upon files in the filesystem.

For example, here's a simple command-line way of querying log files, to see who's been viewing this web site:

$ squeal "count(*)", host from /var/log/httpd/*access_log* group by host;

The above command-line example is querying multiple files at once, parsing them, and aggregating results for you.

Viewing tabular data

You can use "squeal" in shell pipelines, to read and write tabular data in various formats. Here's an example of reading the local RPM database and outputting it in HTML format:

$ squeal --format=html name, version, release, arch from rpm > test.html

If it's not part of a shell pipeline, it displays a text-based result browser, where you can scroll horizontally and vertically through the results. (Specifically, a "curses" UI appears if stdout is directly connected to a tty and you're not specifying format conversion) Screenshot of curses-based UI, showing result of "show from rpm order by name"

Examples

Querying the Apache Logs

How much has each user been downloading?

$ squeal host, "count(*)", "total(size)" from /var/log/httpd/*access_log* group by host;

What are all the 404s?

$ squeal distinct request from /var/log/httpd/*access_log* where status = 404

Look at number/size of requests in the apache logs:

$ squeal filename, "count(*)", "total(size)" from /var/log/httpd/*access_log* group by filename order by "total(size)" desc
                       filename|count(*)|total(size)|
-------------------------------+--------+-----------+
/var/log/httpd/ssl_access_log.4|    1921| 12824849.0|
    /var/log/httpd/access_log.3|     222|  6207367.0|
/var/log/httpd/ssl_access_log.3|     741|  2210799.0|
    /var/log/httpd/access_log.4|     268|   626711.0|
/var/log/httpd/ssl_access_log.1|       8|    13351.0|
/var/log/httpd/ssl_access_log.2|       5|     7305.0|
    /var/log/httpd/access_log.2|       4|     6995.0|
    /var/log/httpd/access_log.1|       2|      288.0|

Yum logs

Querying yum logs (dealing with changes in format internally):

[root@brick ~]# squeal from /var/log/yum.log* where 'name like "kernel%"' limit 5
           time|    event|           name|  arch|epoch|  version|     release|        filename|
---------------+---------+---------------+------+-----+---------+------------+----------------+
Feb 14 20:00:03|Installed|kernel-firmware|noarch| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 20:00:28|  Updated| kernel-headers|  i386| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 20:15:11|Installed|   kernel-devel|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 21:05:53|Installed|         kernel|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 21:12:41|Installed|     kernel-PAE|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|

Querying an RPM database

Querying the host's RPM database, finding packages by name with more than one installed:

[david@brick ~]$ squeal name, "count(*)" from rpm group by name having "count(*)>1"
                     name|count(*)|
-------------------------+--------+
               gpg-pubkey|       4|
jakarta-commons-validator|       2|
 java-1.6.0-openjdk-devel|       2|
                   kernel|       3|
             kernel-devel|       2|
               kernel-xen|       3|
                  libgnat|       2|
                  openssl|       2|

Looking in RPM database by vendor:

[david@brick ~]$ squeal vendor, "count(*)" from rpm group by vendor
            vendor|count(*)|
------------------+--------+
              None|      12|
    Fedora Project|    2042|
              Koji|       2|

Browsing RPM package files

$ squeal "total(size)" from ~/rpmbuild/RPMS/noarch/squeal-0.4-1.fc10.noarch.rpm
total(size)|
-----------+
   171407.0|

Looking in sys logs

$ squeal "count(*)", source from /var/log/messages* group by source order by "count(*)" desc
count(*)|source              |
--------+--------------------+
1633    |kernel              |
1324    |NetworkManager      |
98	|ntpd                |
70	|avahi-daemon        |
63	|dhclient            |
48	|setroubleshoot      |
39	|dnsmasq             |
29	|nm-system-settings  |
27	|bluetoothd          |
14	|/usr/sbin/gpm       |
13	|acpid               |
10	|init                |
9	|pcscd               |
9	|pulseaudio          |
6	|gnome-keyring-ask   |
6	|gnome-keyring-daemon|
6	|gnome-session       |
6	|rsyslogd            |
5	|rpc.statd           |
4	|vpnc                |
3	|gdm-session-worker  |
2	|auditd              |
2	|console-kit-daemon  |
2	|libvirtd            |
2	|rpcbind             |
1	|nm-dispatcher.action|
1	|restorecond         |
$ squeal /var/log/secure* where message like \"%authentication failure%\"

System configuration files

$ squeal /etc/yum.repos.d/*.repo where gpgcheck != \'"1"\'
$ squeal /etc/xinetd.d/*
$ squeal /etc/aliases
$ squeal /etc/inittab
$ squeal /etc/passwd where shell !=\'/sbin/nologin\'

Browsing archive files

Can query the contents of .zip, .tar, .tar.gz and .tar.bz2:

$ squeal name, size from ~/rpmbuild/SOURCES/squeal-0.4.tar.gz order by size desc

Browsing a tcpdump/Wireshark log

$ squeal "count(*)", "total(length)", src_mac, dst_mac from test.pcap group by src_mac, dst_mac

Getting the code

Anonymous git: git://git.fedorahosted.org/squeal.git e.g. git clone git://git.fedorahosted.org/squeal.git

Authenticated git: ssh://git.fedorahosted.org/git/squeal.git

Detailed web user interface to source: http://git.fedorahosted.org/git/squeal.git

Attachments