Console Login

Server Log Analysis with AWStats on CentOS 6 and Apache

The problem

Page-tagging analytics and the server disagree. A hosted tracker reports one traffic level while the load average, the bandwidth graph and the error log suggest another. The tracker is only one view of the traffic, and it leaves several things out:

  • Visitors whose browser blocks or does not run the tracking script never appear in it.
  • Requests that never render a page do not appear either. That covers 404 responses, hotlinked images, large file downloads, feed readers and scrapers.
  • The tracker cannot show which URLs consume bandwidth or which clients hammer the server.

The server's access log is the complete record of what was requested and what was answered. Reading it by hand with grep and awk does not scale beyond a few hundred lines, so a log analyzer is needed. A second problem follows once one is installed: it can produce wrong or empty reports if the log format does not match its configuration, and it can slow the machine that serves the site.

Why it happens

Script-based counting versus request logging

A JavaScript tracker counts only the browsers that download and execute its snippet and then send a beacon to a third party. The web server logs every request it handles, whatever the client is, and it does so whether or not a script runs. The two data sets measure different things and are not expected to match. Neither is exact: logs include robots and cached or proxied hits, and scripts miss non-script clients.

Mismatched formats

AWStats is a Perl script that parses a text log line by line. It needs to be told the exact field layout. A line that does not fit the declared layout is counted as a corrupted record, and if every line is corrupted the report stays empty or meaningless.

Proxy setups hide the client address

With Nginx in front of Apache as a reverse proxy, Apache sees the connection from the proxy, normally 127.0.0.1. Its log then lists the proxy address for every request. Country, host and visitor reports built from that log are useless unless the real client address is restored (for example with mod_rpaf on Apache 2.2) or the analysis reads the proxy's own log.

Parsing cost

Parsing a large log reads a lot of data sequentially and updates AWStats' data files. On a busy single machine this competes for disk and CPU with the web server unless it is scheduled and limited.

Source of truth

The AWStats documentation defines the setup and the configuration directives:

  • AWStats setup documentation confirms that the config file is created by copying awstats.model.conf. It says to check that LogFormat is 1 for the NCSA Apache combined (ELF/XLF) format, to set SiteDomain to the main domain, and to set LogFile to the web server log path. It also states that LogFormat must match the analyzed log, and that records not matching it are reported as corrupted.
  • AWStats configuration directives lists LogFile, LogType, LogFormat, SiteDomain, HostAliases and AllowToUpdateStatsFromBrowser. It confirms that records for hosts in neither SiteDomain nor HostAliases are discarded, that value 1 means Apache combined format, and that the common format (value 4) is also accepted, with fewer reportable details.
  • AWStats FAQ repeats that Apache must write the combined format when LogFormat=1 is used, and it explains how to override LogFile on the command line to process a different file.

Nginx and Apache both document the combined format, and the two are field-compatible, so LogFormat=1 applies to either server's combined log.

Ways to fix it

The goal is trustworthy, regularly updated reports from the server's own logs. Several methods exist:

MethodStrengthsWeaknesses
AWStats with a cron-driven update and CGI reportsPerl and a web server only; reports are generated from the logs; incremental updatesCGI must be exposed and protected; Nginx has no built-in CGI
AWStats with static HTML outputNo CGI at runtime; works behind Nginx; easy to protect with basic authenticationPages are regenerated rather than live; more disk used for the pages
Ad-hoc shell analysis (awk, sort, uniq -c)No install; ideal for one question, such as top 404 URLsNo history, no browser or country breakdown
A hosted script tracker aloneNo server loadMisses non-script clients and non-page requests; visitor data leaves the server
Offline analysis on a copy of the logsNo parsing load on the production hostRequires log transfer and one more machine to maintain

The two proxy cases also have sub-options: restore the client IP on Apache, or analyze the Nginx access log, which already records the real client address.

Why this fix

AWStats driven from cron against the web server's combined log is the choice here, for these reasons:

  • It is packaged for the target systems (EPEL for CentOS 6, the main archive for Debian 6 and Ubuntu 12.04), so it needs no build step.
  • Updates are incremental: it keeps its own state and skips old records, so each run costs little after the first.
  • Reports cover exactly what the script tracker cannot see: status codes, bytes per file type, robots and referrers.
  • The data stays on the operator's own host. It still contains client IP addresses, so retention and access need a deliberate policy. Self-hosting removes the third-party transfer but not the duty to handle the logs carefully.

For the proxy layout, the simplest reliable variant is to analyze the log of the front-most server, because it sees the real client address. Restoring the address on Apache is the alternative when the Apache log has to be used. The report front end is CGI under Apache. On an Nginx-only host, static pages are the practical option.

Step by step

The commands below are a procedure to apply as root, not captured output. Hostnames and paths are examples, and package paths differ slightly between distributions. Check each step before continuing.

1. Install the package

On CentOS 6, AWStats is not in the base repositories. Enable the EPEL repository first by installing its epel-release package for release 6 as described by the EPEL project, then install:

yum install awstats

On Debian 6 or Ubuntu 12.04:

apt-get update
apt-get install awstats

2. Make the web server write the combined format

On Apache 2.2 the combined format is predefined in the stock configuration. Make sure the site's log uses it:

CustomLog logs/access_log combined

Reload and confirm the log lines now end with a quoted referrer and a quoted user agent:

service httpd reload
tail -n 3 /var/log/httpd/access_log

On Nginx, combined is predefined too:

access_log /var/log/nginx/access.log combined;

3. Restore the client address behind a proxy (only if Apache sits behind Nginx)

In the Nginx proxy location, pass the address:

proxy_set_header X-Real-IP $remote_addr;

On Apache, mod_rpaf then replaces the connection address with that header, restricted to the proxy's own address:

RPAFenable On
RPAFproxy_ips 127.0.0.1
RPAFheader X-Real-IP

Alternatively skip this step and point AWStats at the Nginx log in step 4.

4. Create the site configuration

The file name must match the -config value used later:

cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.example.com.conf
vi /etc/awstats/awstats.example.com.conf

Set at least these directives:

LogFile="/var/log/httpd/access_log"
LogType=W
LogFormat=1
SiteDomain="example.com"
HostAliases="www.example.com localhost 127.0.0.1"
AllowToUpdateStatsFromBrowser=0
DirData="/var/lib/awstats"

Make sure the DirData directory exists and is writable by the user that runs the update. The log must be readable by that same user.

5. Run the first update by hand

Run it once manually and watch the summary for "corrupted records". A high count means the format does not match:

perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=example.com -update

6. Schedule the update at low priority

Edit the crontab with crontab -e. Run hourly, off the hour, and lower the CPU and disk priority so the web server wins any contention:

10 * * * * nice -n 19 ionice -c3 /usr/bin/perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=example.com -update > /dev/null

Run one last update before each log rotation, for example in the prerotate section of the logrotate stanza, so no lines are rotated away unread.

7. Publish the reports behind authentication (Apache 2.2)

htpasswd -c /etc/awstats/.htpasswd statsuser
Alias /awstats-icon/ /usr/share/awstats/wwwroot/icon/
ScriptAlias /awstats/ /usr/share/awstats/wwwroot/cgi-bin/
<Directory /usr/share/awstats/wwwroot/cgi-bin/>
    Options ExecCGI
    AllowOverride None
    AuthType Basic
    AuthName "Statistics"
    AuthUserFile /etc/awstats/.htpasswd
    Require valid-user
</Directory>

Set DirIcons="/awstats-icon" in the site configuration, reload Apache and open /awstats/awstats.pl?config=example.com. Serve it over HTTPS if the password should not cross the network in clear text.

8. Nginx-only alternative: static pages

Generate the pages from cron after the update and serve the directory with Nginx and basic authentication:

mkdir -p /var/www/stats
perl /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=example.com -month=all -year=2012 -dir=/var/www/stats -awstatsprog=/usr/share/awstats/wwwroot/cgi-bin/awstats.pl

9. What to read in the reports

  • HTTP status codes: repeated 404s point to broken links or probing.
  • File types and downloads: large archives or images that dominate bandwidth are candidates for a separate host or caching rules. Hotlinking shows up in the referrer reports.
  • Robots and unusual clients: AWStats labels known robots. Abusive address ranges can be blocked with a firewall rule or an deny directive in the web server configuration.

Review

What the method delivers

Incremental updates from a correct combined-format log give a complete picture of requests, including those that no script reports. Low-priority scheduling keeps the parse away from request handling. Keeping AllowToUpdateStatsFromBrowser at 0 and putting the CGI behind authentication limits who can see or trigger anything.

Limits

  • Log numbers count requests, not people. Caches, shared proxies, NAT and robots all distort visitor counts, so the figures need interpreting rather than comparing one to one with a script tracker.
  • The country report depends on DNS lookups or a GeoIP plugin. Without one, the geography is limited.
  • Parsing is a single Perl process. Very large logs make the first run slow. Run that first run off-peak, or split the history with logresolvemerge.pl and process it in pieces.
  • The log is personal data in practice, since it holds IP addresses. Rotate and delete old logs on a schedule and restrict read access.
  • Exposed CGI scripts need to be kept patched. Keep the package current from the distribution or EPEL.

When to choose another method

Use shell one-liners for a single question, such as the top missing URLs. Use a copy of the logs on another machine if the production host is already short of disk or CPU. Keep a script tracker alongside the logs when per-page behaviour such as events or conversions matters, because logs cannot see what happens inside the page.