mirrordir(1) Linux mirrordir(1)
1998 November 1
NAME
pslogin - secure remote tcp login using strong stream cipher
encryption and diffie-hellman key exchanges.
forward - arbitrary tcp socket forwarding over a secure/encrypted
channel.
copydir, mirrordir - copy, mirror directory trees via a minimal set of
changes, locally or over FTP, or over a secure tcp connection.
recursdir - recurse through local or remote directories to
command/find files or create tar files.
SYNOPSIS
mirrordir
[-a, --access-times]
[-m, --strict-mtimes]
[--no-mtimes]
[--ignore-size]
[-A, --mtime-threshold ext]
[--time-offset [[+]|-][H]H[:MM]]
[-A, --always-write]
[-r, --restore-access]
[--no-chown]
[--no-chmod]
[-D, --only-delete]
[-b, -S, --backup-extension, --suffix ext]
[-N, --num-backups num]
[-O, --backup-outdate sec]
[-B, --block-size bytes]
[-M, --max-bytes num[K|M|G]]
[-s, --starting-file path]
[-i, --ignore-next-exclude]
[[-i] -X, --exclude path] [[-i] -X, --exclude path] ...
[[-i] -F, --exclude-from file] ...
[[-i] -G, --exclude-glob expr] [[-i] -G, --exclude-glob expr] ...
[[-i] -R, --exclude-regexp expr] [[-i] -R, --exclude-regexp expr] ...
[-C, --exclude-script [expr|file]] [-C, --exclude-script [expr|file]]
...
[-h, --help]
[-v, --verbose] [-v, --verbose] ...
[-V, --version]
[-k, --keep-files]
[-l, --no-hard-links]
[--follow-symlinks]
[-L, --strict-locking]
[-p, --password password]
[-P, --password-exact password]
[--test-login]
[--no-warn-first-login]
[--read-password-from-stdin]
- 1 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
[--allow-empty-ftp-dirs]
[--no-allow-empty-ftp-dirs]
[--netrc]
[--no-netrc]
[--proxy-host host]
[--secure]
[-z, --gzip]
[--gzip-backups]
[--case-insensitive]
[--to-lower]
[--to-upper]
[--no-use-passive-connections]
[-K, --key-size bits]
[--download-scripts]
[--tar-file filename]
[--tar-block-size N]
[-t, --dry-run, --test-only]
[--nice num] control mirror
mirrordir [-c | --copy-mode | --recurs-mode ]
-[abBCdDFGhklMmNOopRrstvVX] src [src ...] dest
copydir -[abBCdeFGhklMmNOopRrstvVX] src [src ...] dest
recursdir -[abBCdeFGhklMmNOopRrstvVX] src [src ...]
pslogin [--key-size bits]
[mc://][username@]hostname[:portnumber][/path]
But usually just
mirrordir [--exclude path] control mirror
copydir src [src ...] dest
recursdir src [src ...] [-C program]
pslogin [username@]hostname[:portnumber]
DESCRIPTION
mirrordir is a set of useful utilities for manipulating and mirroring
directories. Included is also the command pslogin - an alternative to
ssh(1), and forward(1) for forwarding arbitrary TCP socket connections
over encrypted secure channels.
mirrordir copies files that are different between the directories
control and mirror to the directory mirror. Files whose modification
times or sizes differ are copied. File permissions, ownerships,
modification times, access times (only if --access-times is used),
sticky bits, and device types are duplicated. Symlinks are duplicated
without any translation. Symlink modification and access times (of the
symlink itself, not the file it points to) are not preserved. Hard
linked files are merely copied. Creation times cannot be set with Unix
as far as I can see.
- 2 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
mirrordir is a DANGEROUS command because files or directories that
exist in mirror that don't exist in control are deleted. If control is
entirely empty, then all files and directories in mirror will be
deleted. If mirror is entirely empty, then all files and directories
in control will be copied.
In short, mirrordir forces mirror to be an exact replica of the
directory tree control in every possible detail suitable for purposes
of timed backup. It naturally descends into subdirectories to all
their depths. mirrordir tries to be as efficient as possible by making
the minimal set of changes necessary to mirror the directory.
Access time duplication is not usually required and creates
unnecessary load. Hence it is given as an option.
The directory control is left untouched. If --restore-access is given
then access times are reset to their original with each read.
If the the --strict-locking option is on, files in control that are
copied are locked for `shared reading'. This will ensure, if another
process is busy writing to that file, that the file is not copied in
its incomplete or corrupted state.
Usually mirrordir will not exit, but will give error messages to
stderr to report any problems, and then will continue.
The directory mirror or dest must exist, even if it is empty.
Before erasing all the files in a directory, mirrordir checks for the
file *--keep-me (where * is zero or one characters). If this file is
present it will abort with an error message. Hence such a file can be
created in all directories that you are fearful of being recursively
erased.
copydir is equivalent to mirrordir -ck --no-erase-directories ...
(although -c implies -k anyway), so copydir is very much like a
rigorous version of cp(1) where filenames can also be URLs, and only
outdated files are replaced. Use copydir instead of mirrordir for most
file transfers. Only use mirrordir, when you really want to delete
things.
recursdir is a further program that does nothing but descend into the
directories on the command line. It is equivalent to mirrordir --
recurs-mode ...
It was born after the -C option was added, and can be used as a more
rigorous version of find(1) and can also pack all the files it finds
into a tar file.
pslogin is yet a further program which has almost nothing to do with
the previous three. It envokes a secure login session using secure-
mcserv. It is equivalent to mirrordir --login-mode --secure ...
- 3 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
pslogin should be called logindir. See --login-mode below.
forward is yet a further program which has almost nothing to do with
first three. It can do forwarding of arbitrary services over a secure
channel. See forward(1) for details.
The importance of this package is that you can use URL's instead of
normal filenames, and hence manipulate files over a network. The URL
types currently supported are ftp:// and mc:// (http:// is not a
filesystem and therefore is not supported). mc:// is the Midnight
Commander filesystem and is served by the secure-mcserv daemon. It has
the advantage of serving cryptographically strong secure file
transfers and logins.
You can also use glob expressions in filenames for the recursdir and
copydir commands. These will be recursively expanded.
SECURITY AND ENCRYPTION
mirrordir supports strong stream cipher encryption and Diffie-Hellman
key exchanges with several possible key sizes. Secure connections work
with mc:// type connections. See the options --secure, --key-size, --
download-scripts. See the EXAMPLES section for demo's and the FILES
section for where public/private keys are stored.
OPTIONS
--help
Print out detailed help, then exit.
--verbose
Specifies verbose output of file modifications made to mirror.
This option can be given multiple times for greater verbosity.
Output is written to stdout.
--restore-access
Restore the access times of control with each read.
--access-times
Duplicate even the access times of control.
--always-write
Force rewrite of every file regardless of whether they appear to
be identical or not.
--recurs-mode
This is set by default with recursdir. Listed directories are
read recursively and nothing is done to them. This option is
useful with -C in order to execute shell commands and search for
files. Note the system(), exec() and popen() functions available
to the C interpretor.
- 4 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
--login-mode
This option is set by default with pslogin. This approximates
rlogin(1) using secure-mcserv as a server. It is a secure
encrypted connection intended as a replacement to ssh(1).
pslogin implies --secure as well. When using this option or when
using pslogin, only one path must be present on the command-line.
The path is of the form
[mc://][username@]hostname[:portnumber][/path].
Immediately after logging in, a cd /path will be executed at the
shell prompt. To determine if a prompt is available, pslogin
searches for a #, $ or > character. Should these not be found,
pslogin blocks indefinately. To avoid this behaviour, specify
/path as exactly /. This will leave you in your login directory.
Alternatively, change your shell prompt to include one of these
characters.
--copy-mode
This is set by default with copydir. Approximate the behaviour of
cp(1) as regards source and destination files. This option
implies --keep-files. It overwrites any existing file or
directory with the same name as a source file or directory, but
does not delete files or directories unnecessarily. Multiple
source files or directories may be given. The destination path
must be a directory.
--no-erase-directories
With this option, if a target directory conflicts with a source
file, and the directory is not empty, then an error message will
be printed and the program will abort. This is the default
behaviour for copydir.
--erase-directories
With this option, if a target directory conflicts with a source
file, then the target directory will be recursively deleted. This
is the default behaviour for mirrordir.
--allow-empty-ftp-dirs
Some ftp servers do not produce a . or .. directory. This makes
it appear as though you did not have permission to read the
directory. This option overrides this by assuming that such
completely empty directories are merely empty and do not have
errors. If you get unable to open directory: *: Permission denied
errors, then you can use this option. This option is enabled by
default (see next).
--no-allow-empty-ftp-dirs
Because we now cd to the directory to check for permission to
access it, the default behaviour is to allow empty directories.
--only-delete
Do not make any changes to mirror that will cause the total space
- 5 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
occupied by mirror to enlarge. This is a useful option when
backing up onto a drive that has limited space, where changes
have been made to control that might cause mirror to grow larger
during the duration of the transfer. Running mirrordir once with
this option and then once normally will ensure that the available
space is not overrun.
-i, --ignore-next-exclude
This option dictates that the next --exclude- type option must
completely overlook those files regardless of their existence or
non-existence in the mirror directory. It has the effect of a
return value of IGNORE for the --exclude-script option below.
This can be used to cause certain files to never be modified, for
example if you would like /etc/named.boot to never be modified,
use mirrordir /mnt/1 /mnt/2 -i --exclude /mnt/1/etc/named.boot.
Note that all paths must be specified in the control directory
and not the mirror directory, hence --exclude
/mnt/2/etc/named.boot won't work. This has the idiotic behaviour
that if you want to avoid erasing a file, you have to have that
file present in the control directory, even if it has zero
length.
--exclude path
Exclude file or directory path. A large number of excluded paths
on the command-line will slow performance. An excluded path will
be removed from the mirror tree if it already exists, in the same
way as any absent directory or file. Use the --ignore option to
ignore keep rather than delete these files.
If you have a long list of files to exclude, use the --exclude-
from option.
--exclude-glob glob
Exclude file or directory names matching glob style expression
glob. Matches the file without its full path. For shells, the
expression should be enclosed in appropriate quotes to prevent
substitutions.
--exclude-regexp regex
Exclude full pathnames of files or directories matching regex.
For shells, the expression should be enclosed in appropriate
quotes to prevent substitutions.
--exclude-script [expr|file]
Cause execution of the script expr for each file before doing
anything with that file. The script is a C style statement block,
terminating with a `return expression;'. The value of expression
can be one of INCLUDE, EXCLUDE, UNKNOWN or IGNORE (see -i above)
to explain what is to be done with that file. If expr does not
contain a semi-colon (;) then it is assumed to be the name of a
file - which is then loaded. In either case, the text is byte
- 6 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
compiled into reverse-polish notation for fast execution. This
option can be specified multiple times and the scripts will be
executed in order until a script returns something other than
UNKNOWN. A return value of UNKNOWN allows the remaining --
exclude- options on the command-line to take effect.
If you find that the interpretor incorrectly reports errors or
segfaults, please report the breaking script to me.
The scripting language itself is a subset of the full C
programming language. For example, the following is a valid
script:
/* PATH is the full name of the file including
its path, DIR is the directory, excluding
the trailing slash (/), CWD is the current
directory, and depth() returns the number
of forward slashes (/) less one. */
if (depth (DIR) - depth (CWD) > 3) {
printf ("%s: excluded\n", PATH);
return EXCLUDE;
} else
return INCLUDE;
The scripting language does not support the assignment operator,
and hence does not support user defined variables.
The following predefined macros are available. Note that the
expansion of the macros applies equally well to directories as to
files.
FILE current file without its path
NAME file name without its path or extension or trailing dot (.)
EXTENSION
file extension without its leading dot (.)
DIR directory without file-name or trailing slash (/)
PATH full file name with path
CWD current working directory
TIME current time in seconds
All of the logical, arithmetic and bitwise C operators are
supported. These are ( ) >= <= > < != == && || ! - + * / % & ^
and have the same meanings and precedences as in C.
- 7 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
The following further predefined macros are available. Each
returns an integer (type long int in C). These are based on a C
lstat (or stat if --follow-symlinks is used) on the file. See
stat(2) for a detailed explanation.
stat.st_dev - device
stat.st_ino - inode
stat.st_mode - permissions
stat.st_nlink - number of hard links
stat.st_uid - user id of owner
stat.st_gid - group id of owner
stat.st_rdev - device type
stat.st_size - file size in bytes
stat.st_blksize - block-size for file-system I/O
stat.st_blocks - number of blocks allocate
stat.st_atime - time of last accessed in seconds
stat.st_mtime - time of last modification in seconds
stat.st_ctime - time of creation
The following functions return boolean values:
strncmp(string1, string2, integer);
returns an integer less than, equal to, or greater than zero
if string1 is found, respectively, to be less than, to
match, or be greater than string2.
glob(glob, string);
returns zero if string matches glob expression glob. Try to
use only one glob expression in your code for efficiency of
the underlying implementation.
regexp(regexp, string);
returns zero if string matches regular expression regexp.
Try to use only one regular expression in your code for
efficiency of the underlying implementation.
strstr(string1, string2);
returns the first occurance of string2 in string1 up to the
length of string1, or zero if it did not occur.
The following functions also return a boolean value and are
analogous to the corresponding macros explained in stat(2). They
return non-zero if the specified condition is true.
S_ISLNK(integer); - file is a sym-link
S_ISREG(integer); - file is a regular file
S_ISDIR(integer); - file is a directory
S_ISCHR(integer); - file is a character device
S_ISBLK(integer); - file is a block device
S_ISFIFO(integer); - file is an fifo
S_ISSOCK(integer); - file is a socket
- 8 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
The following functions manipulate strings:
strcat(string1, string2);
returns the concatenation of string1 with string2. Note that
the + operator also concatenates strings.
depth(string);
returns one less than the number of forward slashes (/) in
string.
printf(format, ...);
behaves like printf(3) with an important exception: only
long int format specifiers should be used. The behaviour of
anything that results in conversion of other than a long int
is undefined. For example, use "%ld" instead of "%d". This
function prints to stdout.
The following functions do system calls:
system(command);
executes /bin/sh -c command, but unlike the C version, it
returns the exit code of the command. I.e. it executes a
single line of shell script, command.
exec(argv0, argv1, ...);
executes process argv0 with arguments argv1.... argv0 must
be a full path name. This is faster than system because it
need not envoke sh.
popen([string, ] shell_command);
like system, but returns the output of shell_command as a
string. If string is given, this writes string into the
standard input of shell_command and return zero on success.
The following further integer constants are available and are
analogous to the macros defined in stat.h and explained in
stat(2).
S_IFMT S_IFSOCK S_IFLNK S_IFREG S_IFBLK S_IFDIR S_IFCHR S_IFIFO
S_ISUID S_ISGID S_ISVTX S_IRWXU S_IRUSR S_IWUSR S_IXUSR S_IRWXG
S_IRGRP S_IWGRP S_IXGRP S_IRWXO S_IROTH S_IWOTH S_IXOTH
One of the the following constants should be returned using the
return keyword, and imply to the caller as follows. If nothing is
returned, the return value is assumed to be UNKNOWN.
UNKNOWN
didn't know what to do, continue with other --exclude-
options
- 9 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
INCLUDE
include the file
IGNORE
do nothing with the file regardless of its existence or
non-existence in the mirror directory
EXCLUDE
consider the file to be non-existent in the control
directory and hence must be removed from the mirror
directory (this does not override the option --keep-files)
The following perform flow control analogous to C:
The if clause causes statement1, statement2, etc. to be executed
if integer is true (i.e. non-zero), or otherwise causes
statement1, statement2, etc. to be executed. The else {...} part
is optional.
if (integer) {
statement1;
statement2;
.
.
.
} else {
statementA;
statementB;
.
.
.
}
The return clause gives a value back to mirrordir and causes the
script to exit:
return expression;
The exit function cause mirrordir to exit with the specified exit
code.
exit(integer);
C scripts would typically be used to exclude types of files. Note
that this is an excessive implementation of a scripting language,
and all the features are not meant for general use. A typical
script will, for example, do nothing more than cause device files
to be excluded:
if (S_ISSOCK(stat.st_mode) || S_ISFIFO(stat.st_mode)
|| S_ISBLK(stat.st_mode) || S_ISCHR(stat.st_mode)) {
- 10 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
return EXCLUDE;
} else
return INCLUDE;
C scripts can also be used to search for files with the --
recurs-mode option (same as the recursdir command):
/* removes all core files */
/* This example has been moved to the EXAMPLES section. */
--exclude-from file
Exclude from a list of paths listed in the file file. Empty lines
and comment lines (with a # as the first character of the line)
are ignored. This list of files is sorted and binary searched,
so if you have lots of filenames to exclude, it is best to
include them here for performance. This option can be specified
multiple times with different files. BUG: the last path in file
must end with a newline.
--backup-extension level
Create backups of files before deleting or replacing them.
extension is a C style format string e.g. .ORIG.%d (be careful of
shell substitutions with the %). level is the highest number of
revision to keep. extension is appended to the filename, the
oldest file having the highest number.
--backup-outdate sec
Delete backup files older than sec seconds.
--nice num
Be nice to other processes by sleeping occasionally. num is a
small integer. --nice causes the process to sleep for as long as
it is active, times a factor of num. Hence a value of 1 will
(very roughly) double the time it takes to do a copy, and a value
of 3 will quadruple the time. This can be used where one would
like timed backups to place less load on the CPU. --nice may not
be available on your system.
--no-chmod
Normally the permissions of files are set. If you have restricted
access and cannot change permissions, than this can be used to
disable setting of permissions.
--no-chown
Normally the ownerships of files are set. If you have restricted
access and cannot change ownerships, than this can be used to
disable setting of ownerships.
--mtime-threshold sec
This is the deviation in mtime that is allowed for a file before
it is overwritten. If you have mirrored an ftp site, the mtimes
- 11 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
are accurate to within a minute only, thereafter an nfs mirror
will cause every file to be copied: you can then use --mtime-
threshold 60 to fix this.
--time-offset [[+]|-][H]H[:MM]
This sets the time offset of any vfs (i.e. non-local) directory.
For instance, I am 8 hours east of New York, so when I mirror
from New York, I use --time-offset -08:00.
--test-only, --dry-run
Do not make any changes. If used with --verbose, this will show
what changes would be made. This is an effective way of comparing
directory trees. This is untested - i.e. I don't know whether
this option actually makes any changes or not!
--skip-symlinks
Symlinks are treated as though they were not read - hence if they
are found in the mirror directory then they are deleted.
--keep-files
Don't remove files from mirror, even if they don't exist in
control. This makes mirrordir somewhat like cp(1).
--no-hard-links
mirrordir mirrors hard-links properly unless this option is set,
in which case hard-links are copied as regular files.
--follow-symlinks
mirrordir mirrors symlinks properly unless this option is set, in
which case symlinks are copied as regular files. Useful for
mirroring the Debian tree. Note that this follows symlinks in
both the control and mirror directories, hence if symlinks
already exist in the mirror directory, these will be left as
symlinks. Note that --follow-symlinks implies --no-hard-links
also.
--strict-locking
Create shared read locks on files as they are read. This prevents
clashes especially when copying files from the mail directory -
mail programs will be trying to write to those files at the same
time as mirrordir is trying to to read from them. This option has
no effect with virtual file systems.
--max-bytes [[num[k|
Maximum number of bytes to write before giving the message,
`filled up all blocks - first file/dir not mirrored: path', which
is written to stdout. The remaining files are deleted from
mirror, but in their listed order - hence it is possible that the
archive will grow greater than num while mirrordir is still
running. You should make allowance for this eventuality by making
num smaller than the maximum available space. Also note that some
- 12 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
filesystems will give an error message `No space left on device'
before the device is completely full. You can restart mirrordir
with the option --starting-file path and hence continue with this
file on another volume. This enables mirrordir to back up across
different devices. num can be appended by k, M or G (case
insensitive) to specify kilobytes, megabytes or gigabytes
respectively. If any single file is greater than this number,
then an error message will be given. See also --block-size.
--password password
Specify the password for FTP and mc:// connections. The anonymous
password defaults to your login name @ your local machine name.
You will be prompted for other login passwords. As usual, you are
warned that including passwords in scripts is a security risk. It
is much better to put the password in your ~/.netrc file and then
not use the --no-netrc option; see man ftp for details.
--password-exact password]
Don't prepend a - to the anonymous password. With ftp anonymous
passwords, a - is usually prepended to the password string. I
don't know why the Midnight Commander vfs did this, but one user
had problems with it, hence this option is given to send the
password exactly as specified by password.
--test-login
When using --login-mode or pslogin, you may want to test access
non-interactively (eg. in a shell script). To do this you can run
pslogin with this option and then check its exit status. secure-
mcserv uses this to verify if the user can login on the password
server.
--no-warn-first-login
The first time you try a secure connection to a machine, no
public key exists on the local machine. There is hence no
security against a man-in-the-middle attack. A warning to this
effect is printed and the user is prompted if they want to
continue. This option disables this warning and goes ahead
regardless.
--read-password-from-stdin
Instead of specifying the password on the command line, you can
write it into the command via stdin. This is not the same as
typing the invisible password as one normally does, because it
can be used even if there is no terminal. This is useful for
usage within other programs using, say popen(3). secure-mcserv
uses this to verify if the user can login on the password server.
--netrc
Scan the ~/.netrc file. By default, this option is on.
- 13 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
--no-netrc
Turn off reading of ~/.netrc file.
--proxy-host host
Set the proxy for ftp downloads. Don't know how or if this works.
Consult the mc(1) man page for info in proxy support.
--secure
(This feature is BETA) I have implemented a secure socket layer
for mirrordir. It is enabled with this option and applies to
connections to secure-mcserv (i.e. using mc:// type URLs). The
secure socket library consists of libdiffie.a and a header file
diffie-socket.h. If you include diffie-socket.h after you include
sys/socket.h you can recompile the resulting program with all
normal sockets turned into secure sockets. (This applies to any
program that uses Unix socket calls, but is untested). The first
service supporting this is secure-mcserv which compiles and
installs by default. Hence you can use the mc:// file system with
the --secure option, provided the remote host is running secure-
mcserv (try secure-mcserv -h to for help). --secure uses a stream
cipher (much more secure and faster than block ciphers like DES)
with discrete logarithm key exchanges with public key server
authentication (Diffie-Hellman and p-NEW schemes). For the full
details, see the file diffie-socket.h in the source distribution.
The default key size is 512 bits. Note that if you are using gcc
you should compile mirrordir with the options -O3 -fomit-frame-
pointer -s -Wall to speed key generation.
--key-size bits
The default key size is 512 bits. Sizes require primes to be
generated and compiled into the file field.c, hence only those
primes listed in field.c are supported, which are at the moment
512, 768, 1024 and 1536. 768 is the recommended value for medium
security if you have slow computer. Otherwise 1536 is not an
unreasonably paranoid size for long term security. 512 can be
used if you are not worried about large corporations, well
connected hackers, or governments snooping your connections. Note
that the the stream cipher has a length of bits/2, which means
that you have more chance of being hit by a meteor while cashing
in at the state lottery, than cracking it. Be aware that a person
can sniff your connection and then keep the output for as long as
it takes to break it. In twenty years time a 1536 bit key will be
considered small. Note also that the discrete logarithm problem
(used here) is considered to be more difficult to solve than the
factorisation problem, hence the key is effectively a little
larger than for RSA. This is all my (rather uneducated) opinion.
--download-scripts
Mirrordir comes in two versions, an International version and a
US version (see --version). The US version contains no encryption
code whatsoever. Instead it downloads the needed algorithms from
- 14 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
encrypt.obsidian.co.za (in South Africa). These are written in a
fast, native, C-style interpreted language. There are four
scripts: one for the server Diffie-Hellman key exchange, one for
the client Diffie-Hellman key exchange, one for initialising the
stream cipher and one for actually performing the encryption
using the stream cipher. Mirrordir automatically downloads these
scripts when you try to use the security features. The option --
download-scripts however can be used to force a download at any
time. The International version contains compiled-in versions of
the stream cipher, hence only two scripts are used, which need
never be downloaded since they come with the distribution. No
speed penalty is incurred by having the Diffie-Hellman exchanges
in scripts, however, you will notice a significant speed
different when using the encryption scripts compared to
compiled-in encryption.
--version
Prints out the version number as well as whether this is an
International or US version of mirrordir. See --download-scripts.
-z, --gzip
Enable compression for mc:// connections. This actually envokes a
compressed socket layer at a lower level than the encryption.
Compression uses the libz library of gzip(1). The degree of
compression is dynamically set to minimise transfer time. It will
drop to no compression for fast ethernet connections, and will go
to maximum compression for slow modem connections. The algorithm
adjusts the compression level so that the time for a TCP write
call is between 2% and 5% of the time for the same amount of data
to be deflated (i.e. compressed).
--gzip-backups
Backups are usually just a copy of the file. With this option
they are compressed and a .gz is added to the default extension.
Note that if you specify your own extension using --backup-
extension, then it must have a .gz at the end for compression to
work.
--case-insensitive, --for-Robert-Seese
Ignores case in comparison of filenames and linknames. This
option is useful when interacting with certain brain-dead
operating systems. I am not sure if this option behaves correctly
under all circumstances.
--to-lower
--to-upper
Convert all new filenames to upper or lower case. When used with
--case-insensitive, it will apply only when creating new files.
When used without --case-insensitive, all files, existing or not,
will be converted to uppercase in the inefficient way of deleting
- 15 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
the old file and then copying the new file. This option is useful
when interacting with certain brain-dead operating systems. I am
not sure if this options behave correctly under all
circumstances.
--no-use-passive-connections
If you get an error message could not setup passive mode it is
likely because you need to enable this option. I don't really
understand what `passive' means, so don't ask me.
--tar-file filename
This is to be used only with recursdir. It creates a tar archive
in the same format as GNU tar(1) and stores it in filename.
Leading special prefixes and leading slashes are removed, i.e.
ftp://machine/dir/file becomes dir/file. If the first character
of the filename is a | character, the remaining text is
considered as a command through which the output is to be piped.
Hence a gzipped archive can be created for example with
recursdir ftp://machine/dir --tar-file '| gzip -d > foo.tar.gz'
--tar-block-size N
Sets the block size for tar output to 512 * N. These are the
units in which data is written to the archive. The default is 20.
This is significant only when writing to block devices. This must
not be confused with --block-size.
--block-size bytes
The default block size is 1024 bytes. File sizes are rounded up
to the nearest block when calculating the total number of blocks
consumed. If the actual block size is greater than assumed then
it is possible for fewer blocks to be counted than are actually
consumed during writing. Hence it is essential to specify a
block size greater than or equal to the actual block size when
using the option --max-bytes.
--strict-mtimes
When copying regular files, mirrordir normally only overwrites
the mirror file if it is older than the control file. This option
forces a copy if there is any discrepancy at all in the modified
times of the files.
--no-mtimes
Copy files only if their sizes differ. Ignore the file
modification time.
--ignore-size
Copy files depending on mtime, but ignore size differences.
--starting-file path
path can be a file or directory. Until path is read, files or
directories will be processed as though they were excluded files
- 16 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
(i.e. if they exist in mirror they will be deleted). Directories
that contain path will be created if they don't exist. After path
is read, files are mirrored as usual. path itself will also be
mirrored. If path or any of its subdirectories are not present,
then mirrordir exits immediately. This is the only time
mirrordir exits prematurely. This is to prevent the entire
filesystem from being erased on account of path never being
found.
FTP SUPPORT
Ftp transfers are supported using the Midnight Commander's Virtual
File System (VFS), see mc(1). In short, this means that full URLs as
well as local directories are supported. The following example
demonstrates:
mirrordir --verbose \
ftp://lava.obsidian.co.za/pub/mirrordir \
/home/mirrordir
or alternatively,
mirrordir --verbose /home/mirrordir \
ftp:// HREF=mailto:psheer@lava.obsidian.co.za>psheer@lava.obsidian.co.za/home/ftp/pub/mirrordir
also works, but will prompt me for a password first. If you are
uploading to an ftp server, you should not have the --strict-mtimes
options on, because modification times cannot be set via ftp, and
hence everything will be copied.
You will notice (with --verbose) that mirrordir repeatedly tries to
set modification and access times over ftp. I am going to leave these
messages in to remind users of the limitations of the VFS type in use.
These attempts do not appreciably detract from performance. You can
use mc:// instead although performance is poor when uploading with
this protocol. Downloading is always preferable to uploading.
In general you should NOT have an ftp upload in your cron jobs or use
ftp upload to keep directories in sync. Always download from the other
end when trying to keep directories in sync. Uploading is good for
once off uploads only.
EXAMPLES
Here are some nifty things you can do with mirrordir.
Pedantic minimalist copy
I have two source trees. I like to keep an old copy before I
apply a patch. I just do mkdir tree.OLD and then
mirrordir -v tree tree.OLD
If I run mirrordir again, then only a minimal change is made,
i.e. only the updated files are copied. (cp(1) can in fact do
this).
- 17 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
System backups
Some systems used timed backups onto tape archives. Others still
use RAID devices which constantly maintain an identical copy of a
partition. mirrordir provides a further alternative. You can
install two drives in a system - one for use and one for backups
- and place mirrordir in your cron(8) tables. The various backup
options can be set to make backups of files that have changed.
The backup directory might also be readable by users so that they
can see their backed up files. Previous versions of files would
also be available for users that might like to retrieve any older
instances of the file. Because mirrordir executes only the
absolutely minimal set of changes, it is extremely fast. It can
be run several times a day or even continuously with the --nice
option.
Backups can also be made by FTP onto a remote machine for
additional security against the destruction of the machine.
Backing up hourly between two machines:
On one machine called dar2 I have a cron job that runs every six
hours:
#!/bin/sh
# (this is just in case of any bugs I don't know about,
# but I don't think it is necessary)
killall -9 tee
killall mirrordir >& /dev/null
sleep 2
killall -9 mirrordir >& /dev/null
( \
date ; \
echo "mirrordir says (if it said nothing it is bad):" ; \
mirrordir mc://dar1:12346/ -p abcdefg /mnt/dar1/ \
-i --exclude-regexp '^mc://dar1:12346/var/lock/subsys/atd' \
--exclude-regexp '^mc://dar1:12346/proc/' \
--exclude-regexp '^mc://dar1:12346/mnt/[^/]*/.*$' \
-i --exclude-regexp '^mc://dar1:12346/boot/' \
-i --exclude-regexp '^mc://dar1:12346/etc/lilo.conf' \
-C \
'
if (S_ISDIR (stat.st_mode)) {
if (!regexp ("^mc://dar1:12346/[^/]*$", PATH))
printf ("Backing up: %s0, PATH);
}
' ; \
date ; \
echo "Done" ; \
) 2>&1 \
| tee --ignore-interrupts --append /var/log/mirrordir.log \
- 18 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
| mail -s 'dar1 backup results' psheer@obsidian.co.za
Secure transfers and logins
On the machine turing.co.uk execute
secure-mcserv -p 12345 -d
On some foreign machine
copydir --secure -K 512 -z \
mc:// HREF=mailto:alan@turing.co.uk>alan@turing.co.uk:12345/usr/src/linux/.config .
to securely copy a file using a security key of 512 bits and
using compression, or
pslogin mc:// HREF=mailto:alan@turing.co.uk>alan@turing.co.uk:12345/
to login to the machine securely.
Mirroring FTP sites
Ftp sites often disable their -R option on ls so that a
traditional mirror(1) will fail (?). mirrordir does not suffer
from this limitation.
mirrordir -v ftp://metalab.unc.edu/pub /home/ftp/pub
FTP transfers
Typical FTP transfers can be made easily with a single command
using the --copy-mode option. You can copy multiple files in both
directions and even between two ftp sites (albeit indirectly),
just like with cp(1). I use
copydir -v mirrordir-0.9.15.tar.gz \
mirrordir.lsm ftp://metalab.unc.edu/incoming/Linux
to upload mirrordir to the sunsite. Passwords for non-anonymous
ftp transfers are best placed in the ~/.netrc file according to
standard ftp convention, and the option --netrc used.
Alternatively use ftp:// HREF=mailto:myname@machine>myname@machine/ instead.
Finding files
recursdir / -C
'if (!glob ("*.c", FILE)) printf ("%s\n",
will print out all the C files on your system.
recursdir / -C
'if (S_ISCHR(stat.st_mode)) printf ("%s\n", PATH);'
will print out all character devices on your system.
Backup FTP sites onto tape
Use
recursdir ftp:// HREF=mailto:user@remote.machine>user@remote.machine/ \
--exclude-regexp '//[^/]*/proc/' --tar-file /dev/mt
to back-up a remote site onto tape.
Removing core files
This removes all core files from your system:
recursdir / -C '
long l;
- 19 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
if (strncmp (PATH, "/proc", 5)) {
if (S_ISREG (stat.st_mode) && !strcmp ("core", FILE)) {
if (strstr (popen ("file " + PATH), "ELF 32-bit LSB core")) {
l = l + stat.st_size;
printf ("removing: %s, cumu. total = %ldkB\n", PATH, l >> 10);
exec ("rm", "-f", PATH); /* could also use system() */
}
}
}
'
ENVIRONMENT VARIABLES
TMPDIR
The directory where you would like temporary files to be stored.
The ftp filesystem downloads files first into the this temporary
directory, and then copies the file into its correct place. See
BUGS below.
If TMPDIR is not specified, then it defaults to the directory of
the current file in progress.
RETURN VALUE
mirrordir returns the following:
0 Success.
1 Some kind of error occurred like a write error, a permissions
error or the like. In this case, the precise error would have
been written to stderr.
2 A file was in use and therefore could not be copied, but
otherwise a success. In this case the error `unable to open
control file for writing' would have been written to stderr. If
you grep for these in stderr, you can then run mirrordir with
just these files. See grep(1).
BUGS
Some versions of the atd daemon create a lock (or pid?) file with a
lock on the file that cause secure-mcserv to block indefinately. You
will have to kill atd or exclude (?) this file before hand to prevent
this.
Mirroring over ftp where there is some daylight savings correction,
seems to produce a one hour time offset. Use --time-offset as a
temporary work-around. I don't know if this is mirrordir's fault
though.
A bug that caused mirrordir to eat CPU and appear to halt has been
fixed.
- 20 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
A large number of command-line --exclude expressions will slow
performance. This is noted for the purposes of those wanting to
exclude a large list of files. This list should be placed in a text
file and excluded with the --exclude-from option.
Modification and access times of symlinks (not the files they point
to) are not duplicated.
Note that the previous limitation that hardlinked files are treated as
regular files has now been alleviated. The option --no-hard-links is
provided to emulate the behaviour of versions prior to 0.9.8.
No check is made to see if a hard-link would be created across
devices, in which case an appropriate error will be reported.
The ftp file-system downloads files first into the temporary
directory, as a file with the name *ftpfs*. This is a waste of space
but is the default behavior of the vfs library. If there is not enough
space in this directory, then mirrordir will probably hang. See
ENVIRONMENT VARIABLES above.
The C scripting language used to exclude files is an excessive
implementation.
FILES
~/.netrc
List of machines and corresponding passwords. See the option --
netrc.
/etc/ssocket/accept.cs
This script performs the key exchange and signature generation on
the server side of the connection.
/etc/ssocket/connect.cs
This script performs the key exchange and signature
authentication on the client side of the connection.
/etc/ssocket/arcinit.cs
This initialises the stream cipher encryption. (Not present in
International versions.)
/etc/ssocket/arcencrypt.cs
This performs the stream cipher encryption. (Not present in
International versions.)
/etc/ssocket/private/
This directory contains private keys for the host. Each key will
be held in a file: filenames are 512, 1024 etc. This is a
preferable alternative to databases of keys as it obviates the
need for a key management utility. Soon the Reiser (spelling)
filesystem will be standard and database files will become
- 21 - Formatted: November 14, 2024
mirrordir(1) Linux mirrordir(1)
1998 November 1
obsolete anyway.
/etc/ssocket/public/
Analogous to /etc/ssocket/private/ for public keys.
STANDARDS
mirrordir is an invention of its author and does not belong to any
operating system standard (although it should!).
AVAILABILITY
The latest version of the program can be found at either
ftp://metalab.unc.edu/pub/Linux/system/backup, or
ftp://lava.obsidian.co.za/pub/linux/mirrordir.
AUTHOR
Paul Sheer <psheer@obsidian.co.za> <psheer@icon.co.za>
SEE ALSO
mirror(1), pavuk(1), cp(1), scp(1), find(1), mc(1), ftp(1), ssh(1),
tar(1), rlogin(1), rlogind(8), forward
- 22 - Formatted: November 14, 2024
