|
HTML rendering created 2024-06-26 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|
|
NAME | SYNOPSIS | DESCRIPTION | INSTALLATION | CONFIGURATION | CAVEATS | FILES | PCP ENVIRONMENT | SEE ALSO | COLOPHON |
|
|
|
PMDAWEBLOG(1) General Commands Manual PMDAWEBLOG(1)
pmdaweblog - performance metrics domain agent (PMDA) for Web
server logs
$PCP_PMDAS_DIR/weblog/pmdaweblog [-Cp] [-d domain] [-h helpfile]
[-i port] [-l logfile] [-n idlesec] [-S num] [-t delay] [-u
socket] [-U username] configfile
pmdaweblog is a Performance Metrics Domain Agent (PMDA(3)) that
scans Web server logs to extract metrics characterizing Web
server activity. These performance metrics are then made
available through the infrastructure of the Performance Co-Pilot
(PCP).
The configfile specifies which Web servers are to be monitored,
their associated access logs and error logs, and a regular-
expression based scheme for extracting detailed information about
each Web access. This file is maintained as part of the PMDA
installation and/or de-installation by the scripts Install and
Remove in the directory $PCP_PMDAS_DIR/weblog. For more details,
refer to the section below covering installation.
Once started, pmdaweblog monitors a set of log files and in
response to a request for information, will process any new
information that has been appended to the log files, similar to a
tail(1). There is also periodic "catch up" to process new
information from all log files, and a scheme to detect the
rotation of log files.
Like all other PMDAs, pmdaweblog is launched by pmcd(1) using
command line options specified in $PCP_PMCDCONF_PATH - the
Install script will prompt for appropriate values for the command
line options, and update $PCP_PMCDCONF_PATH.
A brief description of the pmdaweblog command line options
follows:
-C Check the configuration and exit.
-d domain
Specify the domain number. It is absolutely crucial that
the performance metrics domain number specified here is
unique and consistent. That is, domain should be
different for every PMDA on the one host, and the same
domain number should be used for the pmdaweblog PMDA on
all hosts.
For most installations, the default domain as encapsulated
in the file $PCP_PMDAS_DIR/weblog/domain.h will suffice.
For alternate values, check $PCP_PMCDCONF_PATH for the
domain values already in use on this host, and the file
$PCP_VAR_DIR/pmns/stdpmid contains a repository of ``well
known'' domain assignments that probably should be
avoided.
-h helpfile
Get the help text from the supplied helpfile rather than
from the default location.
-i port
Communicate with pmcd(1) on the specified Internet port
(which may be a number or a name).
-l logfile
Location of the log file. By default, a log file named
weblog.log is written in the current directory of pmcd(1)
when pmdaweblog is started, i.e. $PCP_LOG_DIR/pmcd. If
the log file cannot be created or is not writable, output
is written to the standard error instead.
-n idlesec
If a Web server log file has not been modified for idlesec
seconds, then the file will be closed and re-opened. This
is the only way pmdaweblog can detect any asynchronous
rotation of the logs by Web server administrative scripts.
The default period is 20 seconds. This value may be
changed dynamically using pmstore(1) to modify the value
of the performance metric web.config.check.
-p Communicate with pmcd(1) via a pipe.
-S num Specify the maximum number of Web servers per sproc. It
may be desirable (from a latency and load balancing
perspective) or necessary (due to file descriptor limits)
to delegate responsibility for scanning the Web server log
files to several sprocs. pmdaweblog will ensure that each
sproc handles the log files for at most num Web servers.
The default value is 80 Web servers per sproc.
-t delay
To avoid the need to scan a lot of information from the
Web server logs in response to a single request for
performance metrics, all log files will be checked at
least once every delay seconds. The default is 15
seconds. This value may by changed dynamically using
pmstore(1) to modify the value of the performance metric
web.config.catchup.
-u socket
Communicate with pmcd(1) via the given Unix domain socket.
-U User account under which to run the agent. The default is
the unprivileged "pcp" account in current versions of PCP,
but in older versions the superuser account ("root") was
used by default.
The PCP framework allows metrics to be collected on one host and
monitored from another. These hosts are referred to as collector
and monitor hosts, respectively. A host may be both a collector
and a monitor.
Collector hosts require the installation of the agent, while
monitoring hosts require no agent installation at all.
For collector hosts do the following as root:
# cd $PCP_PMDAS_DIR/weblog
# ./Install
The installation procedure prompts for a default or non-default
installation. A default installation will search for known
server configurations and automatically configure the PMDA for
any server log files that are found. A non-default installation
will step through each server, prompting the user for other
server configurations and arguments to pmdaweblog. The end
result of a collector installation is to build a configuration
file that is passed to pmdaweblog via the configfile argument.
If you want to undo the installation, do the following as root:
# cd $PCP_PMDAS_DIR/weblog
# ./Remove
pmdaweblog is launched by pmcd(1) and should never be executed
directly. The Install and Remove scripts notify pmcd(1) when the
agent is installed or removed.
The configuration file for the weblog PMDA is an ASCII file that
can be easily modified. Empty lines and lines beginning with '#'
are ignored. All other lines must be either a regular expression
or server specification.
Regular expressions, which are used on both the access and error
log files, must be of the form:
regex regexName regexp
or
regex_posix regexName ordering regexp_posix
The regexName is a word which uniquely identifies the regular
expression. This is the reference used in the server
specification. The regexp for access logs is in the format
described for regcmp(3). The regexp_posix for access logs is in
the format described for regcomp(3). The argument ordering is
explained below. The Posix form should be available on all
platforms.
The regular expression requires the specification of up to four
arguments to be extracted from each line of a Web server access
log, depending on the type of server. In the most common case
there are two arguments representing the method and the size.
For the non- Posix version, argument $0 should contain the
method: GET, HEAD , POST or PUT. The method PUT is treated as a
synonym for POST, and anything else is categorized as OTHER.
The second argument, $1, should contain the size of the request.
A size of ``-'' or `` '' is treated as unknown.
Argument $3 should contain the status code returned to the client
browser and argument $4 should contain the status code returned
to the server from a remote host. These latter two arguments are
used for caching servers and must be specified as a pair (or $3
will be ignored). For further information on status codes, refer
to the web site
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html .
Some legal non- Posix regex expression specifications for
monitoring an access log are:
# pattern for CERN, NCSA, Netscape etc Access Logs
regex CERN ] "([A-Za-z][-A-Za-z]+)$0 .*" [-0-9]+ ([-0-9]+)$1
# pattern for FTP Server access logs (normally in SYSLOG)
regex SYSLOG_FTP ftpd[.*]: ([gp][-A-Za-z]+)$0( )$1
There is 1 special types of access logs with the RegexName SQUID.
This formats extract 4 parameters but since the Squid log file
uses text-based status codes, it is handled as a special case.
In the examples below, NS_PROXY parses the Netscape/W3C Common
Extended Log Format and SQUID parses the default Squid Object
Cache format log file.
# pattern for Netscape Proxy Server Extended Logs
regex NS_PROXY ] "([A-Za-z][-A-Za-z]+)$0 .*" ([-0-9]+)$2 \
([-0-9]+)$1 ([-0-9]+)$3
# pattern for Squid Cache logs
regex SQUID [0-9]+.[0-9]+[ ]+[0-9]+ [a-zA-Z0-9.]+ \
([_A-Z]+)$3([0-9]+)$2 ([0-9]+)$1 ([A-Z]+)$0
The regexp for the error logs does not require any arguments,
only a match. Some legal expressions are:
# pattern for CERN, NCSA, Netscape etc Error Logs
regex CERN_err .
# pattern for FTP Server error logs (normally in SYSLOG)
regex SYSLOG_FTP_err FTP LOGIN FAILED
If POSIX compliant regular expressions are used, additional
information is required since the order of parameters cannot be
specified in the regular expression. For backwards
compatibility, the common case of two parameters the order may be
specified as method,size or size,method In the general case, the
ordering is specified by one of the following methods:
n1,n2,n3,n4
where nX is a digit between 1 and 4. Each comma-separated
field represents (in order) the argument number for
method,size,client_status,server_status
- Used for cases like the error logs where the content is
ignored.
As for the non- Posix format, the SQUID RegexName is treated as a
special case to match the non-numerical status codes.
Some legal Posix regex expression specifications for monitoring
an access log are:
# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN method,size ][ \]+"([A-Za-z][-A-Za-z]+) \
[^"]*" [-0-9]+ ([-0-9]+)
# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN 1,2 ][ \]+"([A-Za-z][-A-Za-z]+) \
[^"]*" [-0-9]+ ([-0-9]+)
# pattern for FTP Server access logs (normally in SYSLOG)
regex_posix SYSLOG_FTP method,size ftpd[.*]: \
([gp][-A-Za-z]+)( )
# pattern for Netscape Proxy Server Extended Logs
regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A-Za-z][-A-Za-z]+) \
[^"]*" ([-0-9]+) ([-0-9]+) ([-0-9]+)
# pattern for Squid Cache logs
regex_posix SQUID 4,3,2,1 [0-9]+.[0-9]+[ ]+[0-9]+ \
[a-zA-Z0-9.]+ ([_A-Z]+)([0-9]+) ([0-9]+) ([A-Z]+)
# pattern for CERN, NCSA, Netscape etc Error Logs
regex_posix CERN_err - .
# pattern for FTP Server error logs (normally in SYSLOG)
regex_posix SYSLOG_FTP_err - FTP LOGIN FAILED
A Web server can be specified using this syntax:
server serverName on|off accessRegex accessFile errorRegex errorFile
The serverName must be unique for each server, and is the name
given to the instance for the associated performance metrics.
See PMAPI(3) for a discussion of PCP instance domains. The on or
off flag indicates whether the server is to be monitored when the
PMDA is installed. This can altered dynamically using pmstore(1)
for the metric web.perserver.watched, which has one instance for
each Web server named in configfile.
Two files are monitored for each Web server, the access and the
error log. Each file requires the name of a previously declared
regular expression, and a file name. The log files specified for
each server do not have to exist when the weblog PMDA is
installed. The PMDA will continue to check for non-existent log
files and open them when possible. Some legal server
specifications are:
# Netscape Server on Port 80 at IP address 127.55.555.555
server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors
# FTP Server.
server ftpd on SYSLOG_FTP /var/log/messages SYSLOG_FTP_err /var/log/messages
Specifying regular expressions with an incorrect number of
arguments, anything other than 2 for access logs, and none for
error logs, may cause the PMDA to behave incorrectly and even
crash. This is due to limitations in the interface of regex(3).
$PCP_PMDAS_DIR/weblog
installation directory for the weblog PMDA
$PCP_PMDAS_DIR/weblog/Install
installation script for the weblog PMDA
$PCP_PMDAS_DIR/weblog/Remove
de-installation script for the weblog PMDA
$PCP_LOG_DIR/pmcd/weblog.log
default log file for error reporting
$PCP_PMCDCONF_PATH
pmcd configuration file that specifies the command line
options to be used when pmdaweblog is launched
$PCP_LOG_DIR/NOTICES
log of PMDA installations and removals
$PCP_VAR_DIR/config/web/weblog.conf
likely location of the weblog PMDA configuration file
$PCP_DOC_DIR/pcpweb/index.html
the online HTML documentation for PCPWEB
Environment variables with the prefix PCP_ are used to
parameterize the file and directory names used by PCP. On each
installation, the file /etc/pcp.conf contains the local values
for these variables. The $PCP_CONF variable may be used to
specify an alternative configuration file, as described in
pcp.conf(5).
pmcd(1), pmchart(1), pmdawebping(1), pminfo(1), pmstore(1),
pmview(1), tail(1), weblogvis(1), webvis(1), PMAPI(3), PMDA(3)
and regcmp(3).
This page is part of the PCP (Performance Co-Pilot) project.
Information about the project can be found at
⟨http://www.pcp.io/⟩. If you have a bug report for this manual
page, send it to pcp@groups.io. This page was obtained from the
project's upstream Git repository
⟨https://github.com/performancecopilot/pcp.git⟩ on 2024-06-14.
(At that time, the date of the most recent commit that was found
in the repository was 2024-06-14.) If you discover any rendering
problems in this HTML version of the page, or you believe there
is a better or more up-to-date source for the page, or you have
corrections or improvements to the information in this COLOPHON
(which is not part of the original manual page), send a mail to
man-pages@man7.org
Performance Co-Pilot PCP PMDAWEBLOG(1)
|
HTML rendering created 2024-06-26 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|