
Monitoring Interface for Apache 1.3 & MRTG
Daily Average Graph
www.snert.com
- apache2mrtg.pl
- mod_watch
- mod_watch.pl
-
- Welcome
- Software
-
Description
This module will watch and collect the bytes, requests, and documents in & out per virtual
host, file owner, remote-ip address, directory or location, and the web server as a whole. This
module was designed for use with MRTG,
which will make nice graphical representations of the data, but is general
enough that it can be applied to other purposes, as the raw data is accessed by a URL.
This module supports mod_vhost_alias and mod_gzip.
Configuration
The directives below can be added to the general Apache configuration file,
httpd.conf .
- SetHandler watch-flush
- Context: global, <Directory>, <Location>, <VirtualHost>
When WatchStorage uses the hash table, this handler allows a remote script to request that the
shared hash table be flushed and saved to disk for the purpose of backing up the data, without
having to restart Apache. This handler is not required for normal operation. It should only be made
available when required and protected by Apache access directives to allow only the localhost to
make this request.
- SetHandler watch-info
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler is used to display a line of plain text containing the
following space separated fields:
ifName ifUptime ifInOctets ifOutOctets ifRequests ifDocuments ifActive ifOutRate
The ifName is the target name of the request made in a URL.
The remaining fields are all numerical. ifUptime is the
web server's uptime in seconds. The following is required
in the global context of the httpd.conf file in order for mod_watch.pl
to function:
This will permit the following URL forms to get the virtual-host, file-owner, server,
and spool-file, data respectively:
The token between the tilde (~) and the following slash (/) can be the name of
any key that appears in the watch-table list and is the
only way to obtain an entry when watching by-port.
- SetHandler watch-list
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler is similar to watch-info, but presents a
plain text list of all the entries that would appear in watch-table.
- SetHandler watch-copyright
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler permits me to verify the version and copyright of an installation
(see License part 2). This is only required to meet
the License requirements for commercial redistribution:
- SetHandler watch-table
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler is intended for debugging and analysis of the shared memory hash table
performance and weenie files:
- WatchDocuments mime-type ...
- Context: global
A list of MIME types considered to be documents, for the purpose of counting.
For example "text/html" would be considered a document, which loads page elements
such as images, style sheets, audio files, etc. which are not documents, but
elements of one. An asterisk in the mime-type string will act as a wildcard
for zero or more of anything. This directive will override the default list:
application/*word |
|
application/*excel |
|
application/pdf |
application/postscript |
application/rtf |
text/enriched |
text/html |
text/plain |
text/richtext |
text/xml |
- WatchHashTableSize size
- Context: global
A positive number specifying the size of the shared memory hash table used for the
WatchStorage hash policy. The default is 991 (a prime
number, but not necessary to be so) .
- WatchNotLogged handler ...
- Context: global
The combination of MRTG and mod_watch.pl can result in a significant number of
uninteresting entries concerning watch-info to
appear in the access log. This directive provides a means by which to
suppress those entries. By default everything is logged.
Specify one or more of the following handler names
for which logging should be suppressed:
watch-copyright, watch-info, watch-list, watch-table, all
NOTE this directive could affect the correct operation of other modules
that follow after mod_watch in the execution sequence, and should be
used with some care.
A better way to achive the same behaviour would be to alter the
httpd.conf file with something similar to the following:
- WatchNetwork exclude ip/mask ...
- WatchNetwork include ip/mask ...
- Context: global
Add one or more network IP/mask pairs to either the exclusion or inclusion list.
Requests coming from a network or host that is a member of the exclusion list
and is not a member of the inclusion list are not counted. Otherwise the requests
will be counted as outlined in WatchStorage.
The ip/mask can be specified in one of several ways
and is the same as for Apache's Allow or Deny directives (domain or host
name strings are not supported):
Form |
|
Example |
|
Equivalent |
a | 10 | 10.0.0.0/8 |
a.b | 169.254 | 169.254.0.0/16 |
a.b.c | 192.168.1 | 192.168.1.0/24 |
a.b.c.d | 127.0.0.1 | 127.0.0.1/32 |
a.b.c.d/n | 192.168.1.0/24 | 192.168.1 |
a.b.c.d/A.B.C.D | 192.168.1.0/255.255.255.0 | 192.168.1.0/24 |
- WatchStorage files
- WatchStorage hash
- Context: global
The WatchStorage directive specifies the perfered
storage policy for maintaining the counters.
The files and hash
storage policies both use the WatchSpoolDirectory
to store "weenie" text files. The filename of each text file is either the
virtual host name, file owner (login ID), the remote-IP address,
or the string "SERVER". The contents of each text file is a
string of ASCII text containing several space separated numerical fields given below.
The hash storage policy is the default storage
policy.
Name |
|
Type |
|
Description |
ifInOctets | 32-bit Counter | bytes received from requests (except chunked input) |
ifOutOctets | 32-bit Counter | bytes sent in response to requests |
ifRequests | 32-bit Counter | number of requests (raw hits) received |
ifDocuments | 32-bit Counter | number of documents sent |
ifActive | 16-bit Signed | number of active (concurrent) connections |
ifOutRate | Float | approx. 5 minute average of bytes sent per second |
periodOctets | 32-bit Unsigned | bytes sent during the current 5 minute period |
periodMarker | 32-bit Unsigned | start time() of the current 5 minute period |
When updating a set of counters, they are first read from the specified
storage medium. The number of octets in and out for a request are added
to the current counters, and the number of requests is incremented by one.
The number of documents is incremented by one, if the response MIME type
matches one of the MIME types specified for the
WatchDocuments directive. The number of active connections is incremented
at tne start of a request and decremented at the end of the same request.
The counters are then returned from whence
they came. The counters, ifInOctets, ifOutOctets, ifRequest, ifDocuments,
are allowed to roll over.
The ifOutRate, periodOctets, and periodMarker are experimental fields and
may change in future versions. The ifOutRate value is computed on the first
request after a minimum 5 minute wait period. The periodOctets counts the
number of octets sent for each 5 minute period. The periodMarker is the
the value returned from time() and marks the start of current
5 minute period.
The difference between files and
hash storage policies is that the
files policy will read/write to disk several
sets of counters (see WatchWhat) for each request.
The hash storage policy attempts to improve
performance by reducing disk access through the use of a shared memory
hash table, occasionally swapping one or more sets of counters out to disk.
- WatchSpoolFile filename
- Context: <Directory>, <Location>
Used to watch by directory or location. Each filename
must be unique amongst all the virtual hosts, file owners, IP addresses, and other
WatchSpoolFile directives, otherwise the behaviour is unspecified.
The filename is used to store runtime data about
a location within the WatchSpoolDirectory and is used
to extract this data using a URL of the form:
http://www.snert.com/~filename/watch-info
The filename will also appear as an entry in the
watch-table list.
- WatchSpoolDirectory directory
- Context: global
An absolute or server-root relative path to a directory where any support and
runtime files will be kept. This directory must be readable and writeable by the Apache child
processes. The default is /var/spool/mod_watch .
- WatchWhat item ...
- Context: global
mod_watch, by default, will collect information for the virtual host, file owner, and
the server as a whole for each request made. For performance reasons,
it might not be necessary to watch all these things. Specify one or more of the
following to select which items to watch:
all |
|
nothing |
virtual-host |
virtual-host-by-port |
file-owner |
file-owner-by-port |
remote-ip |
remote-ip-by-port |
server |
server-by-port |
headers-in |
headers-out |
content-length |
|
The by-port variants collect and separate the information
according to the port number, instead of grouping all requests regardless of port.
For example, virtual-host-by-port would consider requests for
http://www.snert.com/ different from https://www.snert.com/ and so would
separate the information into "www.snert.com:80" and "www.snert.com:443".
This also holds true if Apache is listening on other ports. Specifying both variants
is allowed and would provide both a break down by port and the sum on all ports.
Specifying headers-in or headers-out
will add the length of the headers of the request and response to the
ifInOctets and ifOutOctets counters repectively. These are not on by default,
because Apache does not count them in the acceess log and may skew comparisions
made between web log analysis tools and the MRTG data.
Specifying content-length will count the value of
the Content-Length header on POST requests. Note however that this value can be
falsified by malicious people in order to deliberately skew the graphs and
so its value is suspect. There is no way to guarantee that a CGI or module
actually read the input data.
Specifying all, remote-ip, or
remote-ip-by-port should
be used with care as monitoring remote-IP addresses can generate large volumes of data
and impact web server performance. Note that the values of ifInOctets and ifOutOctets
reflect data received from and sent to the remote-IP address.
Download
Installation
Unpack and install MRTG prior to installing this module.
Unpack this module into its distribution directory.
Before building the mod_watch binary, please review and alter the
configuration settings found at the top of the Makefile. Note that
Apache's "apxs" utility is required to have already been installed prior
to building either static or dynamic versions of mod_watch.
The fastest and simplest way to install mod_watch, is as an Apache
DSO library. This presumes that Apache has been compiled and built
with mod_so (see Apache documentation):
cd (path to)/mod_watch-3.18
make install-dynamic
Some system administrators prefer to statically link their Apache
modules into Apache itself. This requires more effort on the part
of the system administrator. To start the process do the following
and follow any additional instructions supplied by the Makefile:
cd (path to)/mod_watch-3.18
make install-static
In both dynamic and static installations, the Makefile will install the
following scripts into /usr/local/sbin (see SCRIPTDIR in the
Makefile):
apache2mrtg.pl,
mod_watch.pl
Also the directory /var/spool/mod_watch (see SPOOLDIR in the
Makefile) will be created to hold the weenie files managed by mod_watch.
The following must be added to the global server configuration
within httpd.conf in order for mod_watch.pl
to work (without modification):
<IfModule mod_watch.c>
# Allows the URL used to query virtual host data:
#
# http://www.snert.com/watch-info
#
<Location /watch-info>
SetHandler watch-info
</Location>
# Allows the URL used to query file owner and web
# server data:
#
# http://www.snert.com/~achowe/watch-info
# http://www.snert.com/~SERVER/watch-info
#
<Location /~*/watch-info>
SetHandler watch-info
</Location>
# Intended for debugging and analysis of shared memory
# hash table and weenie files:
#
# http://www.snert.com/watch-table
#
<Location /watch-table>
SetHandler watch-table
</Location>
# The following allows me to verify the version
# and copyright for the purposes of support:
#
<Location /watch-copyright>
SetHandler watch-copyright
Order allow,deny
Allow from snert.biz snert.com snert.info snert.net snert.org
</Location>
</IfModule>
Use apache2mrtg.pl to parse the
httpd.conf file to create an MRTG configuration script for all the
defined virtual hosts. The output can be appended to or included in an
existing mrtg.cfg file. There currently is no support to
generate MRTG configurations for virtual hosts derived from
mod_vhosts_alias or local user names.
perl apache2mrtg.pl (path to)/httpd.conf >web.cfg
Use this generated configuration with MRTG for a cron job that executes regularly
to generate the graphs (see MRTG installation documentation). This configuration
will periodically make calls to mod_watch.pl to obtain
the MRTG target name, uptime, bytes in & out, requests in, and documents out.
Stop/start the web server:
apachectl stop; apachectl start
Please note that "apachectl restart" does not always appear to load correctly
a DSO module that uses shared memory and semaphores. For this reason it is recommended
that you stop/start the web server instead.
Once the Apache web server starts receiving requests, all the bytes
received in requests, where a Content-Length header
is defined and all the bytes sent in responses are
added to the current 32-bit counters and stored.
If the MRTG output is properly
configured, you should be able to use a URL to see assorted graphs concerning data
transfer rates of virtual hosts, file owners, and the web server.
Notes
Sample MRTG output & configurations for:
- Octets In & Out
- The follow MRTG configuration is produced by
apache2mrtg.pl:
Title[www.snert.com]: www.snert.com Data Traffic
Target[www.snert.com]: `/usr/local/sbin/mod_watch.pl http://www.snert.com/watch-info`
MaxBytes[www.snert.com]: 1250000
PageTop[www.snert.com]: <h2>www.snert.com Data Traffic</h2>
- Requests & Documents
- The following MRTG configuration was done by hand to produce Request &
Document traffic:
Options[www.snert.com,req]: growright, perhour
Title[www.snert.com,req]: www.snert.com Request & Document Traffic
Target[www.snert.com,req]: `/usr/local/sbin/mod_watch.pl -f ifRequests,ifDocuments http://www.snert.com/watch-info`
MaxBytes[www.snert.com,req]: 1250000
PageTop[www.snert.com,req]: <h2>www.snert.com Request & Document Traffic</h2>
YLegend[www.snert.com,req]: Hits per Hour
ShortLegend[www.snert.com,req]: H/h
Legend1[www.snert.com,req]: Requests (raw hits) received per Hour
Legend2[www.snert.com,req]: Documents sent per Hour
Legend3[www.snert.com,req]: Maximal 5 Minute Incoming Request Traffic
Legend4[www.snert.com,req]: Maximal 5 Minute Outgoing Document Traffic
LegendI[www.snert.com,req]: Hits:
LegendO[www.snert.com,req]: Docs:
mod_watch has been built and/or tested on the following platforms
(please email me your platform if you've
made it work on something different):
- Cobalt Qube 1, RedHat Linux 5.1, Apache 1.3.22
-
Use the default (System V) configuration.
- FreeBSD 4.2, 4.4, Apache 1.3.22
-
Use the default (System V) configuration. Not yet tested with POSIX API.
From <benoitdm at citeglobe dot com>: "If Apache is running in a jail on
FreeBSD 4.4, you must setup jail.sysvipc_allowed=1 to allow use of shared
memory in the restricted environment. This OID is a kernel configuration
option, through you can set it up in /etc/rc.conf or on the fly with sysctl."
- HP-UX, Apache 1.3.22
-
Use the default (System V) configuration.
- Max OS X 10.1, Apache 1.3.24
-
Specify the following macros in the Makefile:
DEF=-DSERIALIZATION_API=FCNTL_API -DSHARED_MEMORY_API=MMAP_ANON
RANLIB=ranlib
- SunOS 5.7 (or better), Apache 1.3.22
-
mod_watch/3.0:
To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE.
Modify the top of the Makefile to enable the following C defines and link
with the correct libraries:
DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
LIB=-lrt
- Solaris 5.6, Apache 1.3.22
-
mod_watch/3.0:
To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE.
Modify the top of the Makefile to enable the following C defines and link
with the correct libraries:
DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
LIB=-lposix4
Help in French & English for MRTG and mod_watch.
Thank you to the folks at
Security Space for including mod_watch in their
monthly Apache Module Report.
ifOutRate introduced in 3.14 is experimental and subject to change.
- apache2mrtg.pl
- mod_watch
- mod_watch.pl
-
- Welcome
- Software
-
Copyright 2001, 2003 by
Anthony C Howe
All rights reserved.
light bulbs since 6 December 2001
|