XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
NAME
xferstats - compiles information about file transfers from logfiles
SYNOPSIS
xferstats [- | + ] [-c <config file>] [-T <number>] [[-+]C] [[-+]H]
[[-+]n] [[-+]s] [[-+]r] [[-+]a] [[-+]g] [[-+]i] [[-+]u] [[-+]h] [[-+]m]
[[-+]w] [[-+]M] [[-+]f [number]] [[-+]d [number]] [[-+]t [number]] [[-
+]O [number]] [[-+]o [number]] [+L <number>] [+A] [-D <number>] [+D
<number>] [+P <prefix>] [+B <timestamp>] [+E <timestamp>] [-v] [--
version] [--help] [--] [filename(s)]
DESCRIPTION
xferstats is a utility that parses wu-ftpd, ncftpd, and (special)
apache xferlogs (although it's not difficult to add support for other
log types) and gives various information and totals about the data
transferred.
OPTIONS
+ The single plus tells xferstats to get the xferlog from stdin.
For example:
cat xferlog | xferstats +
- The single hyphen tells xferstats to get the xferlog from a file.
While this is the default behaviour, the configuration file may
change the default, and this option allows you to override it.
For example:
xferstats - /var/log/xferlog
-c configuration file
Specify an alternate path and filename for the configuration
file. The packaged default is /usr/local/etc/xferstats.cfg but
your administrator may have changed this. A new default can be
set by changing the appropriate variable in xferstats.h and
recompiling.
-T log type
Specify the type of log file to be parsed. Currently, only three
types are defined, wu-ftpd for wu-ftpd-style logs, ncftpd for
ncftpd-style logs, and apache for special Apache logs. Note: you
can also use "wu-ftp", "wuftp", "wuftpd", or "ncftp". In order
for xferstats to parse Apache logfiles, you must use a special
LogFile line: LogFormat "%{%a %b %d %X %Y}t %T %h %b %f %u".
xferstats may or may not support regular Apache logfiles in the
future. If this is something you want, please let me know.
-v, --version
Display what version of xferstats you're running.
- 1 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
--help
Display a quick summary of the command line options.
-- Use a -- (double hyphen) to indicate that you are finished giving
parameters and will follow with a filename. This is particularly
useful when you want to specify a filename that begins with a +
or -
TOGGLE OPTIONS
The options in this section can be either forced on with +[option] or
forced off with -[option]. Options in this section, like all other
options, will always override any settings in the xferstats
configuration file. Any arguments to toggle options should only be
used when enabling an option. Supplying an argument when disabling an
option will generate an error.
C When enabled, xferstats will try to weed out corrupt log lines by
strictly checking to make sure that the data is valid. If you
--enable-debug_flow at compile time, xferstats will notify you
when corrupt lines are detected.
H When enabled, xferstats will generate HTML tables and graphs
instead of text tables. In order to actually generate tables
instead of broken graphic icons, you must put the graph JPGs in
the directory pointed to by the GRAPH_PATH configuration command.
For more information, see the CONFIGURATION FILE section.
n When enabled, xferstats will refrain from adding <HTML>, <BODY>,
and <TITLE> tags, so that the output can be inserted into pages
containing custom headers.
s When enabled, xferstats will output HTML as a single page
(filename "xferstats.html") instead of a page for each section.
This option also disables <HTML> and <BODY> tags, although it
does produce a <TITLE> tag unless the NO_HTML_HEADERS option is
also specified.
r When enabled, xferstats will generate statistics about real user
connections (but not necessarily only real user connections)
a When enabled, xferstats will generate statistics about anonymous
connections (but not necessarily only anonymous connections)
g When enabled, xferstats will generate statistics about guest
connections (but not necessarily only guest connections)
i When enabled, xferstats will generate statistics about inbound
(the product of a PUT) files (but not necessarily only inbound
files)
- 2 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
u When enabled, xferstats will generate statistics about outbound
(the product of a GET) files (but not necessarily only outbound
files)
h When enabled, xferstats will generate a report listing traffic by
hour
m When enabled, xferstats will generate a report listing traffic by
month (January, February, etc.)
w When enabled, xferstats will generate a report listing traffic by
day of the week (Monday, Tuesday, etc.)
M When enabled, xferstats will generate a report listing traffic by
day of the month (1st, 2nd, 3rd, etc.)
f [number]
When enabled, xferstats will generate a report listing traffic by
file. You can supply an optional argument that will determine
how many file records xferstats displays (0 (zero) disables the
limit)
d [number]
When enabled, xferstats will generate a report listing traffic by
directory. You can supply an optional argument that will
determine how many directory records xferstats displays (0 (zero)
disables the limit). See the -D and +D options, as well as the
DIR_DEPTH and DIR_FILTER configuration commands for more
information.
t [number]
When enabled, xferstats will generate a report listing traffic by
top-level domain. You can supply an optional argument that will
determine how many TLD records xferstats displays (0 (zero)
disables the limit)
O [number]
When enabled, xferstats will generate a report listing traffic by
domain. You can supply an optional argument that will determine
how many domain records xferstats displays (0 (zero) disables the
limit)
o [number]
When enabled, xferstats will generate a report listing traffic by
host. You can supply an optional argument that will determine
how many host records xferstats displays (0 (zero) disables the
limit)
OTHER OPTIONS
- 3 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
+A Include all users, generate all reports
+L <number>
Limit the daily report to the last <number> days
-D <number>
Depth of path detail for directories (default 3)
+D <directory>
Generate reports on files located only in <directory>. For
example:
+D /pub will report only on paths under /pub. Specifying
multiple +D options will instruct xferstats to include data from
all specified directories.
+P <pathprefix>
Strip the <prefix> from any pathnames that begin with it. This
is the same as specifying a STRIP_PREFIX configuration file
command. Keep in mind that the DIR_FILTER (or +D) is checked
before the STRIP_PREFIX (or +P) when using these.
+B <timestamp>
Providing a +B argument instructs xferstats to only generate data
on transfers occuring after that time. The time stamp format is
"Jan DD HH:MM:ss YYYY". One may combine +B and +E to generate a
report about a specific window in time.
+E <timestamp>
Providing a +E argument instructs xferstats to only generate data
on transfers occuring before that time. The time stamp format is
"Jan DD HH:MM:ss YYYY". One may combine +B and +E to generate a
report about a specific window in time.
The default compile-time setting for the xferlog location is
/var/log/xferlog but your administrator may have changed this.
If xferstats reports an error, try specifying the filename as the
last command line parameter.
CONFIGURATION FILE
The default configuration file path is /usr/local/etc/xferstats.cfg
although your administrator may have changed this.
Any text after a '#' is ignored, as long as the '#' begins a line or
is prepended by whitespace.
Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive, although
some arguments may be.
- 4 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
LOGFILE <filename>
sets the default log file to process. It can be overridden by
adding "<filename>" at the end of the command line options. You
may specify as many LOGFILE lines as you like; they will be
processed in the order specified (although the order processed
should not change the output). Example:
LOGFILE /var/log/xferlog LOGFILE /var/log/xferlog.1 LOGFILE
/var/log/xferlog.2
STRICT_CHECK
takes no arguments. When enabled, xferstats will try to weed out
corrupt log lines by strictly checking to make sure that the data
is valid. If you --enable-debug_flow at compile time, xferstats
will notify you when corrupt lines are detected. It can be
enabled with "+C" or disabled with "-C" on the command line.
ANON_TRAFFIC
takes no arguments. When enabled, xferstats will generate
statistics about anonymous connections (but not necessarily only
anonymous connections). It can be enabled with "+a" or disabled
with "-a" on the command line.
GUEST_TRAFFIC
takes no arguments. When enabled, xferstats will generate
statistics about guest connections (but not necessarily only
guest connections). This option has no effect when parsing non-
wuftpd logfiles. It can be enabled with "+g" or disabled with
"-g" on the command line.
REAL_TRAFFIC
takes no arguments. When enabled, xferstats will generate
statistics about real user connections (but not necessarily only
real user connections). It can be enabled with "+r" or disabled
with "-r" on the command line.
INBOUND
takes no arguments. When enabled, xferstats will generate
statistics about inbound (the product of a PUT) files (but not
necessarily only inbound files). It can be enabled with "+i" or
disabled with "-i" on the command line.
OUTBOUND
takes no arguments. When enabled, xferstats will generate
statistics about outbound (the product of a GET) files (but not
necessarily only outbound files). It can be enabled with "+u" or
disabled with "-u" on the command line.
HOURLY_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic per hour. It can be enabled with "+h" or
- 5 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
disabled with "-h" on the command line.
DOW_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by day of the week (Monday, Tuesday, etc).
It can be enabled with "+w" or disabled with "-w" on the command
line.
DOM_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by day of the month (1st, 2nd, etc). It
can be enabled with "+M" or disabled with "-M" on the command
line.
TLD_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by top-level domain (com, edu, net, us,
uk, etc). It can be enabled with "+t <number>" or disabled with
"-t" on the command line.
DOMAIN_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by domain (foo.com, blah.edu, etc). It
can be enabled with "+O <number>" or disabled with "-O" on the
command line.
HOST_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by host. It can be enabled with "+o" or
disabled with "-o" on the command line.
DIR_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by directory. The directory depth that
xferstats will keep track of can be controlled by the DIR_DEPTH
configuration command or with the "-l <depth>" command line
option. It can be enabled with "+d" or disabled with "-d" on the
command line.
FILE_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic by file. It can be enabled with "+f" or
disabled with "-f" on the command line.
MONTHLY_REPORT
takes no arguments. When enabled, xferstats will generate a
report listing traffic per month (January, February, etc). It
can be enabled with "+m" or disabled with "-m" on the command
line.
- 6 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
HTML_OUTPUT
takes no arguments. When enabled, xferstats will output all
reports into individual HTML files. Future versions will contain
a configuration option to set the directory in which to place the
HTML files. For now they are placed in the present working#
directory. It can be enabled with "+H" or disabled with "-H" on
the command line.
USE_STDIN
takes no arguments. When enabled, xferstats will, by default,
read the logfile from the standard input instead of from a file
on disk. It can be enabled with "+" or disabled with "-" on the
command line.
LOG_TYPE
takes one argument, the type of log that will be processed by
default. The argument can be either a number or a case
insensitive string and are as follows:
wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"
ncftpd log : "2" or "ncftp" or "ncftpd"
Apache log : "3" or "apache"
Therefore:
LOG_TYPE 1
LOG_TYPE wu-ftp
LOG_TYPE wu-ftpd
LOG_TYPE wuftp
LOG_TYPE wuftpd
are all equivalent.
Future versions will contain support for additional logfile
types. It can be set on the command line with "-T <number>"
DIR_DEPTH
takes one argument, the number of subdirectories deep that
xferstats should go when generating statistics by directory.
This configuration command has no effect when DIR_REPORT has not
been enabled. DIR_DEPTH has no upper bound. A DIR_DEPTH of '0'
(zero) tells xferstats to process all subdirectories. The default
value is 3. It can be set on the command line with "-D
<number>". Example:
- 7 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
DIR_DEPTH 3
DIR_FILTER
takes one argument, the directory that xferstats should report
on. xferstats will discard any data that does not have a path
that begins with the value set by DIR_FILTER. This value is
case-sensitive, as it is part of a path name. Specifying
multiple statements will instruct xferstats to include data from
all specified directories. It can be set on the command line
with "+D <path>". Example:
DIR_FILTER /pub/foobar
NUMBER_FILE_STATS
takes one argument, the number of files to report on in the file
statistics section. It must be a number greater than or equal to
zero, with zero denoting no limit. It can be set on the command
line with "+f <number>". The default is 50.
NUMBER_FILE_STATS 50
NUMBER_DIR_STATS
takes one argument, the number of directories to report on in the
directory statistics section. It must be a number greater than
or equal to zero, with zero denoting no limit. It can be set on
the command line with "+d <number>". The default is 50.
NUMBER_DAILY_STATS
takes one argument, the number of days to report on in the daily
statistics section. It must be a number greater than or equal to
zero, with zero denoting no limit. It can be set on the command
line with "+L <number>". The default is 50.
Note: the daily statistics section will always be displayed.
NUMBER_TLD_STATS
takes one argument, the number of top-level domains to report on
in the TLD statistics section. It must be a number greater than
or equal to zero, with zero denoting no limit. It can be set on
the command line with "+t <number>". The default is 50.
NUMBER_DOMAIN_STATS
takes one argument, the number of top-level domains to report on
in the domain statistics section. It must be a number greater
than or equal to zero, with zero denoting no limit. It can be
set on the command line with "+O <number>". The default is 50.
NUMBER_HOST_STATS
takes one argument, the number of hosts to report on in the host
statistics section. It must be a number greater than or equal to
zero, with zero denoting no limit. It can be set on the command
- 8 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
line with "+o <number>". The default is 50.
MAX_REPORT_SIZE
takes one argument, the maximum size of any one output table.
This is particularly useful to keep xferstats from generating
gigantic tables that make graphical browsers choke. It will
split tables to be no larger than# this value. It must be
greater than or equal to ten, or zero, zero denoting no limit.
It can be set on the command line with "+R <number>". The
default is 30.
GRAPH_PATH
takes one argument, the location of the JPGs used to draw graphs
in the HTML output. It can be an absolute path (ie,
http://foo.bar.com/graphs/) or a relative path (ie, ../graphs/).
You must include a trailing '/' or it will not work. This option
cannot be set on the command line. The default is nothing (which
will make the browser look for the graphs in the directory with
the HTML). Example:
GRAPH_PATH ../graphs/
NO_HTML_HEADERS
takes no arguments. When enabled, xferstats will refrain from
adding <HTML>, <BODY>, and <TITLE> tags, so that the output can
be inserted into pages containing custom headers. It can be
enabled with "+n" or disabled with "-n" on the command line.
REFRESH
takes one argument, the number of seconds after being displayed
that the browser should refresh it. Unspecified or zero
indicates no refresh.
SINGLE_PAGE
takes no arguments. When enabled, xferstats will output HTML as
a single page (entitled "xferstats.html") instead of a page for
each section. This option also disables <HTML> and <BODY> tags,
although it does produce a <TITLE> tag unless the NO_HTML_HEADERS
option is also specified. It can be enabled with "+s" or disabled
with "-s" on the command line.
STRIP_PREFIX
takes one argument. When specified, xferstats will strip the
given string from the beginning of any paths. Keep in mind when
using this option with DIR_FILTER that the DIR_FILTER is applied
-before- stripping, so that you must take this into account when
specifying the DIR_FILTER string. This option cannot be set on
the command line. Example:
STRIP_PREFIX /home/ftp
- 9 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
The next 11 entries are all basically the same--they define how the
reports are to be sorted.
0 -- Sort by the number of bytes downloaded
1 -- Sort by the number of files downloaded
2 -- Sort by the "name". This varies by report; for the file,
directory, TLD, etc reports, it sorts them alphabetically; for
the file size report it lists the file sizes in alphabetical
order; for the daily report, it lists the days in order. You get
the idea. 3 -- Sort by average throughput, in K/s
FILE_SORT_PREF, DIR_SORT_PREF, DOMAIN_SORT_PREF, TLD_SORT_PREF,
HOST_SORT_PREF, DOM_SORT_PREF, DOW_SORT_PREF, HOURLY_SORT_PREF,
DAILY_SORT_PREF, SIZE_SORT_PREF, MONTHLY_SORT_PREF
CHUNK_INPUT
takes one argument. When specified, xferstats will parse the
logfiles in chunks of the given size (in number of lines),
process them, and continue. This will keep it from trying to
parse the entire log at once, using up a lot of memory for large
logs. Specifying too small a value will cause xferstats to waste
a lot of time switching tasks, while too large a value may slow
it down as your system swaps. Toy with this value a bit and see
what works best for you. If left unspecified, xferstats will
continue to parse the logs in one pass.
FILES
xferlog (generated by wu-ftpd or ncftpd)
/usr/local/etc/xferstats.cfg contains all of the default settings and
many settings which cannot be set from the command line.
If you installed the RPM version of xferstats, you'll find the JPGs
for the graphs in /usr/doc/xferstats-x.xx/graphs Insert the version
you're running in place of the x.xx.
BUGS
The JPG graph pieces included are not ideal; sometimes artifacts show
up. I will eventually get around to making new GIFs or something.
Please send bug reports to phil@off.net If the xferlog that you're
using is smaller than 1 megabyte, please send a gzipped copy, along
with the exact parameters you're using.
ACKNOWLEDGEMENTS
I greatly appreciate the suggestions and assistance of Nat Friedman
- 10 - Formatted: November 14, 2024
XFERSTATS(8) XFERSTATS(8)
31 Oct 2000
<ndf@mit.edu>.
Thanks to Mike Shaver <shaver@mozilla.org> and Jamie Zawinski
<jwz@netscape.com> for coughing up gigantic mozilla.org logs to aid my
testing.
Thanks to Phil Copeland <bryce@zen.ics.uwe.ac.uk> for getting on my
back constantly to add new stuff, fix nagging bugs, and speed it up.
Oh, and for using it, at least someone does :).
Extreme gratitude to Mark Hagler <hagler@hagler.org>, someone whom
I've left off of this list for too long. Mark was kind enough to
Amazon me a copy of O'Reillys _Pthreads Programming_ which was the
catalyst for the threaded version and really helped get me interested
in the project again. While the threaded version doesn't exist
anymore, the book is still terribly useful for other projects.
SEE ALSO
ftpd(8), ncftpd(8), xferlog(5)
LOCATION
xferstats is available from the following locations. The old
addresses are valid, but are deprecated! Please use the new addresses
whenever possible.
http://xferstats.off.net/
ftp://xferstats.off.net/pub/xferstats/
AUTHOR
xferstats is Copyright (c) 1997-2000 Phil Schwan <phil@off.net> and is
released under the terms and provisions of the GNU Public License.
- 11 - Formatted: November 14, 2024
