The problem
Page-tagging analytics and the server disagree. A hosted tracker reports one traffic level while the load average, the bandwidth graph and the error log suggest another. The tracker is only one view of the traffic, and it leaves several things out:
- Visitors whose browser blocks or does not run the tracking script never appear in it.
- Requests that never render a page do not appear either. That covers 404 responses, hotlinked images, large file downloads, feed readers and scrapers.
- The tracker cannot show which URLs consume bandwidth or which clients hammer the server.
The server's access log is the complete record of what was requested and what was answered. Reading it by hand with grep and awk does not scale beyond a few hundred lines, so a log analyzer is needed. A second problem follows once one is installed: it can produce wrong or empty reports if the log format does not match its configuration, and it can slow the machine that serves the site.
Why it happens
Script-based counting versus request logging
A JavaScript tracker counts only the browsers that download and execute its snippet and then send a beacon to a third party. The web server logs every request it handles, whatever the client is, and it does so whether or not a script runs. The two data sets measure different things and are not expected to match. Neither is exact: logs include robots and cached or proxied hits, and scripts miss non-script clients.
Mismatched formats
AWStats is a Perl script that parses a text log line by line. It needs to be told the exact field layout. A line that does not fit the declared layout is counted as a corrupted record, and if every line is corrupted the report stays empty or meaningless.
Proxy setups hide the client address
With Nginx in front of Apache as a reverse proxy, Apache sees the connection from the proxy, normally 127.0.0.1. Its log then lists the proxy address for every request. Country, host and visitor reports built from that log are useless unless the real client address is restored (for example with mod_rpaf on Apache 2.2) or the analysis reads the proxy's own log.
Parsing cost
Parsing a large log reads a lot of data sequentially and updates AWStats' data files. On a busy single machine this competes for disk and CPU with the web server unless it is scheduled and limited.
Source of truth
The AWStats documentation defines the setup and the configuration directives:
- AWStats setup documentation confirms that the config file is created by copying
awstats.model.conf. It says to check thatLogFormatis1for the NCSA Apache combined (ELF/XLF) format, to setSiteDomainto the main domain, and to setLogFileto the web server log path. It also states thatLogFormatmust match the analyzed log, and that records not matching it are reported as corrupted. - AWStats configuration directives lists
LogFile,LogType,LogFormat,SiteDomain,HostAliasesandAllowToUpdateStatsFromBrowser. It confirms that records for hosts in neitherSiteDomainnorHostAliasesare discarded, that value1means Apache combined format, and that the common format (value4) is also accepted, with fewer reportable details. - AWStats FAQ repeats that Apache must write the combined format when
LogFormat=1is used, and it explains how to overrideLogFileon the command line to process a different file.
Nginx and Apache both document the combined format, and the two are field-compatible, so LogFormat=1 applies to either server's combined log.
Ways to fix it
The goal is trustworthy, regularly updated reports from the server's own logs. Several methods exist:
| Method | Strengths | Weaknesses |
|---|---|---|
| AWStats with a cron-driven update and CGI reports | Perl and a web server only; reports are generated from the logs; incremental updates | CGI must be exposed and protected; Nginx has no built-in CGI |
| AWStats with static HTML output | No CGI at runtime; works behind Nginx; easy to protect with basic authentication | Pages are regenerated rather than live; more disk used for the pages |
Ad-hoc shell analysis (awk, sort, uniq -c) | No install; ideal for one question, such as top 404 URLs | No history, no browser or country breakdown |
| A hosted script tracker alone | No server load | Misses non-script clients and non-page requests; visitor data leaves the server |
| Offline analysis on a copy of the logs | No parsing load on the production host | Requires log transfer and one more machine to maintain |
The two proxy cases also have sub-options: restore the client IP on Apache, or analyze the Nginx access log, which already records the real client address.
Why this fix
AWStats driven from cron against the web server's combined log is the choice here, for these reasons:
- It is packaged for the target systems (EPEL for CentOS 6, the main archive for Debian 6 and Ubuntu 12.04), so it needs no build step.
- Updates are incremental: it keeps its own state and skips old records, so each run costs little after the first.
- Reports cover exactly what the script tracker cannot see: status codes, bytes per file type, robots and referrers.
- The data stays on the operator's own host. It still contains client IP addresses, so retention and access need a deliberate policy. Self-hosting removes the third-party transfer but not the duty to handle the logs carefully.
For the proxy layout, the simplest reliable variant is to analyze the log of the front-most server, because it sees the real client address. Restoring the address on Apache is the alternative when the Apache log has to be used. The report front end is CGI under Apache. On an Nginx-only host, static pages are the practical option.
Step by step
The commands below are a procedure to apply as root, not captured output. Hostnames and paths are examples, and package paths differ slightly between distributions. Check each step before continuing.
1. Install the package
On CentOS 6, AWStats is not in the base repositories. Enable the EPEL repository first by installing its epel-release package for release 6 as described by the EPEL project, then install:
yum install awstats
On Debian 6 or Ubuntu 12.04:
apt-get update
apt-get install awstats
2. Make the web server write the combined format
On Apache 2.2 the combined format is predefined in the stock configuration. Make sure the site's log uses it:
CustomLog logs/access_log combined
Reload and confirm the log lines now end with a quoted referrer and a quoted user agent:
service httpd reload
tail -n 3 /var/log/httpd/access_log
On Nginx, combined is predefined too:
access_log /var/log/nginx/access.log combined;
3. Restore the client address behind a proxy (only if Apache sits behind Nginx)
In the Nginx proxy location, pass the address:
proxy_set_header X-Real-IP $remote_addr;
On Apache, mod_rpaf then replaces the connection address with that header, restricted to the proxy's own address:
RPAFenable On
RPAFproxy_ips 127.0.0.1
RPAFheader X-Real-IP
Alternatively skip this step and point AWStats at the Nginx log in step 4.
4. Create the site configuration
The file name must match the -config value used later:
cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.example.com.conf
vi /etc/awstats/awstats.example.com.conf
Set at least these directives:
LogFile="/var/log/httpd/access_log"
LogType=W
LogFormat=1
SiteDomain="example.com"
HostAliases="www.example.com localhost 127.0.0.1"
AllowToUpdateStatsFromBrowser=0
DirData="/var/lib/awstats"
Make sure the DirData directory exists and is writable by the user that runs the update. The log must be readable by that same user.
5. Run the first update by hand
Run it once manually and watch the summary for "corrupted records". A high count means the format does not match:
perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=example.com -update
6. Schedule the update at low priority
Edit the crontab with crontab -e. Run hourly, off the hour, and lower the CPU and disk priority so the web server wins any contention:
10 * * * * nice -n 19 ionice -c3 /usr/bin/perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=example.com -update > /dev/null
Run one last update before each log rotation, for example in the prerotate section of the logrotate stanza, so no lines are rotated away unread.
7. Publish the reports behind authentication (Apache 2.2)
htpasswd -c /etc/awstats/.htpasswd statsuser
Alias /awstats-icon/ /usr/share/awstats/wwwroot/icon/
ScriptAlias /awstats/ /usr/share/awstats/wwwroot/cgi-bin/
<Directory /usr/share/awstats/wwwroot/cgi-bin/>
Options ExecCGI
AllowOverride None
AuthType Basic
AuthName "Statistics"
AuthUserFile /etc/awstats/.htpasswd
Require valid-user
</Directory>
Set DirIcons="/awstats-icon" in the site configuration, reload Apache and open /awstats/awstats.pl?config=example.com. Serve it over HTTPS if the password should not cross the network in clear text.
8. Nginx-only alternative: static pages
Generate the pages from cron after the update and serve the directory with Nginx and basic authentication:
mkdir -p /var/www/stats
perl /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=example.com -month=all -year=2012 -dir=/var/www/stats -awstatsprog=/usr/share/awstats/wwwroot/cgi-bin/awstats.pl
9. What to read in the reports
- HTTP status codes: repeated 404s point to broken links or probing.
- File types and downloads: large archives or images that dominate bandwidth are candidates for a separate host or caching rules. Hotlinking shows up in the referrer reports.
- Robots and unusual clients: AWStats labels known robots. Abusive address ranges can be blocked with a firewall rule or an
denydirective in the web server configuration.
Review
What the method delivers
Incremental updates from a correct combined-format log give a complete picture of requests, including those that no script reports. Low-priority scheduling keeps the parse away from request handling. Keeping AllowToUpdateStatsFromBrowser at 0 and putting the CGI behind authentication limits who can see or trigger anything.
Limits
- Log numbers count requests, not people. Caches, shared proxies, NAT and robots all distort visitor counts, so the figures need interpreting rather than comparing one to one with a script tracker.
- The country report depends on DNS lookups or a GeoIP plugin. Without one, the geography is limited.
- Parsing is a single Perl process. Very large logs make the first run slow. Run that first run off-peak, or split the history with
logresolvemerge.pland process it in pieces. - The log is personal data in practice, since it holds IP addresses. Rotate and delete old logs on a schedule and restrict read access.
- Exposed CGI scripts need to be kept patched. Keep the package current from the distribution or EPEL.
When to choose another method
Use shell one-liners for a single question, such as the top missing URLs. Use a copy of the logs on another machine if the production host is already short of disk or CPU. Keep a script tracker alongside the logs when per-page behaviour such as events or conversions matters, because logs cannot see what happens inside the page.