diff --git a/INSTALL.md b/INSTALL.md index 5b23ed0e..1605ab43 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -26,7 +26,7 @@ build user (after installing python3's pip package): You can test if you've got it fixed by running (from the rsync checkout): -> ./md2man --test rsync-ssl.1.md +> ./md-convert --test rsync-ssl.1.md Alternately, you can avoid generating the manpages by fetching the very latest versions (that match the latest git source) from the [generated-files][6] dir. diff --git a/NEWS.md b/NEWS.md index 187f2fdb..e32600c5 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,62 @@ +# NEWS for rsync 3.2.5 (14 Aug 2022) + +## Changes in this version: + +### SECURITY FIXES: + +- Added some file-list safety checking that helps to ensure that a rogue + sending rsync can't add unrequested top-level names and/or include recursive + names that should have been excluded by the sender. These extra safety + checks only require the receiver rsync to be updated. When dealing with an + untrusted sending host, it is safest to copy into a dedicated destination + directory for the remote content (i.e. don't copy into a destination + directory that contains files that aren't from the remote host unless you + trust the remote host). Fixes CVE-2022-29154. + + - A fix for CVE-2022-37434 in the bundled zlib (buffer overflow issue). + +### BUG FIXES: + +- Fixed the handling of filenames specified with backslash-quoted wildcards + when the default remote-arg-escaping is enabled. + +- Fixed the configure check for signed char that was causing a host that + defaults to unsigned characters to generate bogus rolling checksums. This + made rsync send mostly literal data for a copy instead of finding matching + data in the receiver's basis file (for a file that contains high-bit + characters). + +- Lots of manpage improvements, including an attempt to better describe how + include/exclude filters work. + +- If rsync is compiled with an xxhash 0.8 library and then moved to a system + with a dynamically linked xxhash 0.7 library, we now detect this and disable + the XX3 hashes (since these routines didn't stabilize until 0.8). + +### ENHANCEMENTS: + +- The [`--trust-sender`](rsync.1#opt) option was added as a way to bypass the + extra file-list safety checking (should that be required). + +### PACKAGING RELATED: + +- A note to those wanting to patch older rsync versions: the changes in this + release requires the quoted argument change from 3.2.4. Then, you'll want + every single code change from 3.2.5 since there is no fluff in this release. + +- The build date that goes into the manpages is now based on the developer's + release date, not on the build's local-timezone interpretation of the date. + +### DEVELOPER RELATED: + +- Configure now defaults GETGROUPS_T to gid_t when cross compiling. + +- Configure now looks for the bsd/string.h include file in order to fix the + build on a host that has strlcpy() in the main libc but not defined in the + main string.h file. + +------------------------------------------------------------------------------ + # NEWS for rsync 3.2.4 (15 Apr 2022) ## Changes in this version: @@ -4482,6 +4541,7 @@ | RELEASE DATE | VER. | DATE OF COMMIT\* | PROTOCOL | |--------------|--------|------------------|-------------| +| 14 Aug 2022 | 3.2.5 | | 31 | | 15 Apr 2022 | 3.2.4 | | 31 | | 06 Aug 2020 | 3.2.3 | | 31 | | 04 Jul 2020 | 3.2.2 | | 31 | diff --git a/batch.c b/batch.c index a9711c56..accc4c6e 100644 --- a/batch.c +++ b/batch.c @@ -3,7 +3,7 @@ * * Copyright (C) 1999 Weiss * Copyright (C) 2004 Chris Shoemaker - * Copyright (C) 2004-2020 Wayne Davison + * Copyright (C) 2004-2022 Wayne Davison * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -194,7 +194,7 @@ static int write_opt(const char *opt, const char *arg) { int len = strlen(opt); int err = write(batch_sh_fd, " ", 1) != 1; - err = write(batch_sh_fd, opt, len) != len ? 1 : 0; + err = write(batch_sh_fd, opt, len) != len ? 1 : 0; if (arg) { err |= write(batch_sh_fd, "=", 1) != 1; err |= write_arg(arg); diff --git a/byteorder.h b/byteorder.h index 525eaba0..059cc708 100644 --- a/byteorder.h +++ b/byteorder.h @@ -2,7 +2,7 @@ * Simple byteorder handling. * * Copyright (C) 1992-1995 Andrew Tridgell - * Copyright (C) 2007-2020 Wayne Davison + * Copyright (C) 2007-2022 Wayne Davison * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -129,4 +129,3 @@ SIVAL(char *buf, int pos, uint32 val) { SIVALu((uchar*)buf, pos, val); } - diff --git a/checksum.c b/checksum.c index b723109c..fb8c0a09 100644 --- a/checksum.c +++ b/checksum.c @@ -62,6 +62,8 @@ struct name_num_obj valid_checksums = { int xfersum_type = 0; /* used for the file transfer checksums */ int checksum_type = 0; /* used for the pre-transfer (--checksum) checksums */ +static int initialized_choices = 0; + int parse_csum_name(const char *name, int len) { struct name_num_item *nni; @@ -79,6 +81,9 @@ int parse_csum_name(const char *name, int len) return CSUM_MD4_ARCHAIC; } + if (!initialized_choices) + init_checksum_choices(); + nni = get_nni_by_name(&valid_checksums, name, len); if (!nni) { @@ -574,7 +579,7 @@ void sum_update(const char *p, int32 len) } /* NOTE: all the callers of sum_end() pass in a pointer to a buffer that is - * MAX_DIGEST_LEN in size, so even if the csum-len is shorter that that (i.e. + * MAX_DIGEST_LEN in size, so even if the csum-len is shorter than that (i.e. * CSUM_MD4_ARCHAIC), we don't have to worry about limiting the data we write * into the "sum" buffer. */ int sum_end(char *sum) @@ -623,3 +628,31 @@ int sum_end(char *sum) return csum_len_for_type(cursum_type, 0); } + +void init_checksum_choices() +{ +#ifdef SUPPORT_XXH3 + char buf[32816]; + int j; + for (j = 0; j < (int)sizeof buf; j++) { + buf[j] = ' ' + (j % 96); + } + sum_init(CSUM_XXH3_64, 0); + sum_update(buf, 32816); + sum_update(buf, 31152); + sum_update(buf, 32474); + sum_update(buf, 9322); + if (XXH3_64bits_digest(xxh3_state) != 0xadbcf16d4678d1de) { + int t, f; + struct name_num_item *nni = valid_checksums.list; + for (t = f = 0; nni[f].name; f++) { + if (nni[f].num == CSUM_XXH3_64 || nni[f].num == CSUM_XXH3_128) + continue; + if (t != f) + nni[t++] = nni[f]; + } + nni[t].name = NULL; + } +#endif + initialized_choices = 1; +} diff --git a/compat.c b/compat.c index b8967969..b46eb199 100644 --- a/compat.c +++ b/compat.c @@ -400,7 +400,7 @@ static const char *getenv_nstr(int ntype) const char *env_str = getenv(ntype == NSTR_COMPRESS ? "RSYNC_COMPRESS_LIST" : "RSYNC_CHECKSUM_LIST"); /* When writing a batch file, we always negotiate an old-style choice. */ - if (write_batch) + if (write_batch) env_str = ntype == NSTR_COMPRESS ? "zlib" : protocol_version >= 30 ? "md5" : "md4"; if (am_server && env_str) { @@ -433,7 +433,7 @@ void validate_choice_vs_env(int ntype, int num1, int num2) nno->saw[CSUM_MD4_ARCHAIC] = nno->saw[CSUM_MD4_BUSTED] = nno->saw[CSUM_MD4_OLD] = nno->saw[CSUM_MD4]; if (!nno->saw[num1] || (num2 >= 0 && !nno->saw[num2])) { - rprintf(FERROR, "Your --%s-choice value (%s) was refused by the server.\n", + rprintf(FERROR, "Your --%s-choice value (%s) was refused by the server.\n", ntype == NSTR_COMPRESS ? "compress" : "checksum", ntype == NSTR_COMPRESS ? compress_choice : checksum_choice); exit_cleanup(RERR_UNSUPPORTED); @@ -523,6 +523,8 @@ static void negotiate_the_strings(int f_in, int f_out) { /* We send all the negotiation strings before we start to read them to help avoid a slow startup. */ + init_checksum_choices(); + if (!checksum_choice) send_negotiate_str(f_out, &valid_checksums, NSTR_CHECKSUM); diff --git a/configure.ac b/configure.ac index 24e383a9..d185b2d3 100644 --- a/configure.ac +++ b/configure.ac @@ -13,7 +13,8 @@ AC_CHECK_HEADERS(sys/fcntl.h sys/select.h fcntl.h sys/time.h sys/unistd.h \ netdb.h malloc.h float.h limits.h iconv.h libcharset.h langinfo.h mcheck.h \ sys/acl.h acl/libacl.h attr/xattr.h sys/xattr.h sys/extattr.h dl.h \ popt.h popt/popt.h linux/falloc.h netinet/in_systm.h netgroup.h \ - zlib.h xxhash.h openssl/md4.h openssl/md5.h zstd.h lz4.h sys/file.h) + zlib.h xxhash.h openssl/md4.h openssl/md5.h zstd.h lz4.h sys/file.h \ + bsd/string.h) AC_CHECK_HEADERS([netinet/ip.h], [], [], [[#include ]]) AC_HEADER_MAJOR_FIXED @@ -22,7 +23,7 @@ AC_CONFIG_SRCDIR([byteorder.h]) AC_CONFIG_HEADER(config.h) AC_PREREQ([2.69]) -PACKAGE_VERSION=`sed 's/.*"\(.*\)".*/\1/' <$srcdir/version.h` +PACKAGE_VERSION=`sed -n 's/.*RSYNC_VERSION.*"\(.*\)".*/\1/p' <$srcdir/version.h` AC_MSG_NOTICE([Configuring rsync $PACKAGE_VERSION]) @@ -624,7 +625,11 @@ fi AC_TYPE_UID_T AC_CHECK_TYPES([mode_t,off_t,size_t,pid_t,id_t]) -AC_TYPE_GETGROUPS +if test "$cross_compiling" = no; then + AC_TYPE_GETGROUPS +else + AC_DEFINE([GETGROUPS_T],[gid_t],[Define to the type of elements in the array set by `getgroups'. Usually this is either `int' or `gid_t'.]) +fi AC_CHECK_MEMBERS([struct stat.st_rdev, struct stat.st_mtimensec, struct stat.st_mtimespec.tv_nsec, @@ -1117,7 +1122,7 @@ else fi AC_CACHE_CHECK([for unsigned char],rsync_cv_SIGNED_CHAR_OK,[ -AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[]], [[signed char *s = ""]])],[rsync_cv_SIGNED_CHAR_OK=yes],[rsync_cv_SIGNED_CHAR_OK=no])]) +AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[]], [[signed char *s = (signed char *)""]])],[rsync_cv_SIGNED_CHAR_OK=yes],[rsync_cv_SIGNED_CHAR_OK=no])]) if test x"$rsync_cv_SIGNED_CHAR_OK" = x"yes"; then AC_DEFINE(SIGNED_CHAR_OK, 1, [Define to 1 if "signed char" is a valid type]) fi diff --git a/exclude.c b/exclude.c index 39073a0c..e5774420 100644 --- a/exclude.c +++ b/exclude.c @@ -25,16 +25,21 @@ extern int am_server; extern int am_sender; +extern int am_generator; extern int eol_nulls; extern int io_error; +extern int xfer_dirs; +extern int recurse; extern int local_server; extern int prune_empty_dirs; extern int ignore_perishable; +extern int relative_paths; extern int delete_mode; extern int delete_excluded; extern int cvs_exclude; extern int sanitize_paths; extern int protocol_version; +extern int trust_sender_args; extern int module_id; extern char curr_dir[MAXPATHLEN]; @@ -44,8 +49,11 @@ extern unsigned int module_dirlen; filter_rule_list filter_list = { .debug_type = "" }; filter_rule_list cvs_filter_list = { .debug_type = " [global CVS]" }; filter_rule_list daemon_filter_list = { .debug_type = " [daemon]" }; +filter_rule_list implied_filter_list = { .debug_type = " [implied]" }; int saw_xattr_filter = 0; +int trust_sender_args = 0; +int trust_sender_filter = 0; /* Need room enough for ":MODS " prefix plus some room to grow. */ #define MAX_RULE_PREFIX (16) @@ -292,6 +300,233 @@ static void add_rule(filter_rule_list *listp, const char *pat, unsigned int pat_ } } +/* If the wildcards failed, the remote shell might give us a file matching the literal + * wildcards. Since "*" & "?" already match themselves, this just needs to deal with + * failed "[foo]" idioms. + */ +static void maybe_add_literal_brackets_rule(filter_rule const *based_on, int arg_len) +{ + filter_rule *rule; + const char *arg = based_on->pattern, *cp; + char *p; + int cnt = 0; + + if (arg_len < 0) + arg_len = strlen(arg); + + for (cp = arg; *cp; cp++) { + if (*cp == '\\' && cp[1]) { + cp++; + } else if (*cp == '[') + cnt++; + } + if (!cnt) + return; + + rule = new0(filter_rule); + rule->rflags = based_on->rflags; + rule->u.slash_cnt = based_on->u.slash_cnt; + p = rule->pattern = new_array(char, arg_len + cnt + 1); + for (cp = arg; *cp; ) { + if (*cp == '\\' && cp[1]) { + *p++ = *cp++; + } else if (*cp == '[') + *p++ = '\\'; + *p++ = *cp++; + } + *p++ = '\0'; + + rule->next = implied_filter_list.head; + implied_filter_list.head = rule; + if (DEBUG_GTE(FILTER, 3)) { + rprintf(FINFO, "[%s] add_implied_include(%s%s)\n", who_am_i(), rule->pattern, + rule->rflags & FILTRULE_DIRECTORY ? "/" : ""); + } +} + +static char *partial_string_buf = NULL; +static int partial_string_len = 0; +void implied_include_partial_string(const char *s_start, const char *s_end) +{ + partial_string_len = s_end - s_start; + if (partial_string_len <= 0 || partial_string_len >= MAXPATHLEN) { /* too-large should be impossible... */ + partial_string_len = 0; + return; + } + if (!partial_string_buf) + partial_string_buf = new_array(char, MAXPATHLEN); + memcpy(partial_string_buf, s_start, partial_string_len); +} + +void free_implied_include_partial_string() +{ + if (partial_string_buf) { + free(partial_string_buf); + partial_string_buf = NULL; + } + partial_string_len = 0; /* paranoia */ +} + +/* Each arg the client sends to the remote sender turns into an implied include + * that the receiver uses to validate the file list from the sender. */ +void add_implied_include(const char *arg, int skip_daemon_module) +{ + filter_rule *rule; + int arg_len, saw_wild = 0, saw_live_open_brkt = 0, backslash_cnt = 0; + int slash_cnt = 1; /* We know we're adding a leading slash. */ + const char *cp; + char *p; + if (trust_sender_args) + return; + if (partial_string_len) { + arg_len = strlen(arg); + if (partial_string_len + arg_len >= MAXPATHLEN) { + partial_string_len = 0; + return; /* Should be impossible... */ + } + memcpy(partial_string_buf + partial_string_len, arg, arg_len + 1); + partial_string_len = 0; + arg = partial_string_buf; + } + if (skip_daemon_module) { + if ((cp = strchr(arg, '/')) != NULL) + arg = cp + 1; + else + arg = ""; + } + if (relative_paths) { + if ((cp = strstr(arg, "/./")) != NULL) + arg = cp + 3; + } else if ((cp = strrchr(arg, '/')) != NULL) { + arg = cp + 1; + } + if (*arg == '.' && arg[1] == '\0') + arg++; + arg_len = strlen(arg); + if (arg_len) { + if (strpbrk(arg, "*[?")) { + /* We need to add room to escape backslashes if wildcard chars are present. */ + for (cp = arg; (cp = strchr(cp, '\\')) != NULL; cp++) + arg_len++; + saw_wild = 1; + } + arg_len++; /* Leave room for the prefixed slash */ + rule = new0(filter_rule); + if (!implied_filter_list.head) + implied_filter_list.head = implied_filter_list.tail = rule; + else { + rule->next = implied_filter_list.head; + implied_filter_list.head = rule; + } + rule->rflags = FILTRULE_INCLUDE + (saw_wild ? FILTRULE_WILD : 0); + p = rule->pattern = new_array(char, arg_len + 1); + *p++ = '/'; + for (cp = arg; *cp; ) { + switch (*cp) { + case '\\': + if (cp[1] == ']') { + if (!saw_wild) + cp++; /* A \] in a non-wild filter causes a problem, so drop the \ . */ + } else if (!strchr("*[?", cp[1])) { + backslash_cnt++; + if (saw_wild) + *p++ = '\\'; + } + *p++ = *cp++; + break; + case '/': + if (p[-1] == '/') { /* This is safe because of the initial slash. */ + cp++; + break; + } + if (relative_paths) { + filter_rule const *ent; + int found = 0; + *p = '\0'; + for (ent = implied_filter_list.head; ent; ent = ent->next) { + if (ent != rule && strcmp(ent->pattern, rule->pattern) == 0) { + found = 1; + break; + } + } + if (!found) { + filter_rule *R_rule = new0(filter_rule); + R_rule->rflags = FILTRULE_INCLUDE | FILTRULE_DIRECTORY; + /* Check if our sub-path has wildcards or escaped backslashes */ + if (saw_wild && strpbrk(rule->pattern, "*[?\\")) + R_rule->rflags |= FILTRULE_WILD; + R_rule->pattern = strdup(rule->pattern); + R_rule->u.slash_cnt = slash_cnt; + R_rule->next = implied_filter_list.head; + implied_filter_list.head = R_rule; + if (DEBUG_GTE(FILTER, 3)) { + rprintf(FINFO, "[%s] add_implied_include(%s/)\n", + who_am_i(), R_rule->pattern); + } + if (saw_live_open_brkt) + maybe_add_literal_brackets_rule(R_rule, -1); + } + } + slash_cnt++; + *p++ = *cp++; + break; + case '[': + saw_live_open_brkt = 1; + *p++ = *cp++; + break; + default: + *p++ = *cp++; + break; + } + } + *p = '\0'; + rule->u.slash_cnt = slash_cnt; + arg = rule->pattern; + arg_len = p - arg; /* We recompute it due to backslash weirdness. */ + if (DEBUG_GTE(FILTER, 3)) + rprintf(FINFO, "[%s] add_implied_include(%s)\n", who_am_i(), rule->pattern); + if (saw_live_open_brkt) + maybe_add_literal_brackets_rule(rule, arg_len); + } + + if (recurse || xfer_dirs) { + /* Now create a rule with an added "/" & "**" or "*" at the end */ + rule = new0(filter_rule); + rule->rflags = FILTRULE_INCLUDE | FILTRULE_WILD; + if (recurse) + rule->rflags |= FILTRULE_WILD2; + /* We must leave enough room for / * * \0. */ + if (!saw_wild && backslash_cnt) { + /* We are appending a wildcard, so now the backslashes need to be escaped. */ + p = rule->pattern = new_array(char, arg_len + backslash_cnt + 3 + 1); + for (cp = arg; *cp; ) { + if (*cp == '\\') + *p++ = '\\'; + *p++ = *cp++; + } + } else { + p = rule->pattern = new_array(char, arg_len + 3 + 1); + if (arg_len) { + memcpy(p, arg, arg_len); + p += arg_len; + } + } + if (p[-1] != '/') + *p++ = '/'; + *p++ = '*'; + if (recurse) + *p++ = '*'; + *p = '\0'; + rule->u.slash_cnt = slash_cnt + 1; + rule->next = implied_filter_list.head; + implied_filter_list.head = rule; + if (DEBUG_GTE(FILTER, 3)) + rprintf(FINFO, "[%s] add_implied_include(%s)\n", who_am_i(), rule->pattern); + if (saw_live_open_brkt) + maybe_add_literal_brackets_rule(rule, p - rule->pattern); + } +} + /* This frees any non-inherited items, leaving just inherited items on the list. */ static void pop_filter_list(filter_rule_list *listp) { @@ -706,11 +941,12 @@ static void report_filter_result(enum logcode code, char const *name, filter_rule const *ent, int name_flags, const char *type) { + int log_level = am_sender || am_generator ? 1 : 3; + /* If a trailing slash is present to match only directories, * then it is stripped out by add_rule(). So as a special - * case we add it back in here. */ - - if (DEBUG_GTE(FILTER, 1)) { + * case we add it back in the log output. */ + if (DEBUG_GTE(FILTER, log_level)) { static char *actions[2][2] = { {"show", "hid"}, {"risk", "protect"} }; const char *w = who_am_i(); @@ -718,7 +954,7 @@ static void report_filter_result(enum logcode code, char const *name, : name_flags & NAME_IS_DIR ? "directory" : "file"; rprintf(code, "[%s] %sing %s %s because of pattern %s%s%s\n", - w, actions[*w!='s'][!(ent->rflags & FILTRULE_INCLUDE)], + w, actions[*w=='g'][!(ent->rflags & FILTRULE_INCLUDE)], t, name, ent->pattern, ent->rflags & FILTRULE_DIRECTORY ? "/" : "", type); } @@ -890,6 +1126,7 @@ static filter_rule *parse_rule_tok(const char **rulestr_ptr, } switch (ch) { case ':': + trust_sender_filter = 1; rule->rflags |= FILTRULE_PERDIR_MERGE | FILTRULE_FINISH_SETUP; /* FALL THROUGH */ diff --git a/flist.c b/flist.c index 1ba306bc..0e6bf782 100644 --- a/flist.c +++ b/flist.c @@ -73,6 +73,7 @@ extern int need_unsorted_flist; extern int sender_symlink_iconv; extern int output_needs_newline; extern int sender_keeps_checksum; +extern int trust_sender_filter; extern int unsort_ndx; extern uid_t our_uid; extern struct stats stats; @@ -83,8 +84,7 @@ extern char curr_dir[MAXPATHLEN]; extern struct chmod_mode_struct *chmod_modes; -extern filter_rule_list filter_list; -extern filter_rule_list daemon_filter_list; +extern filter_rule_list filter_list, implied_filter_list, daemon_filter_list; #ifdef ICONV_OPTION extern int filesfrom_convert; @@ -986,6 +986,19 @@ static struct file_struct *recv_file_entry(int f, struct file_list *flist, int x exit_cleanup(RERR_UNSUPPORTED); } + if (*thisname != '.' || thisname[1] != '\0') { + int filt_flags = S_ISDIR(mode) ? NAME_IS_DIR : NAME_IS_FILE; + if (!trust_sender_filter /* a per-dir filter rule means we must trust the sender's filtering */ + && filter_list.head && check_filter(&filter_list, FINFO, thisname, filt_flags) < 0) { + rprintf(FERROR, "ERROR: rejecting excluded file-list name: %s\n", thisname); + exit_cleanup(RERR_PROTOCOL); + } + if (implied_filter_list.head && check_filter(&implied_filter_list, FINFO, thisname, filt_flags) <= 0) { + rprintf(FERROR, "ERROR: rejecting unrequested file-list name: %s\n", thisname); + exit_cleanup(RERR_PROTOCOL); + } + } + if (inc_recurse && S_ISDIR(mode)) { if (one_file_system) { /* Room to save the dir's device for -x */ diff --git a/io.c b/io.c index cf94cee7..3f605d74 100644 --- a/io.c +++ b/io.c @@ -376,6 +376,7 @@ static void forward_filesfrom_data(void) free_xbuf(&ff_xb); if (ff_reenable_multiplex >= 0) io_start_multiplex_out(ff_reenable_multiplex); + free_implied_include_partial_string(); } return; } @@ -419,6 +420,7 @@ static void forward_filesfrom_data(void) while (s != eob) { if (*s++ == '\0') { ff_xb.len = s - sob - 1; + add_implied_include(sob, 0); if (iconvbufs(ic_send, &ff_xb, &iobuf.out, flags) < 0) exit_cleanup(RERR_PROTOCOL); /* impossible? */ write_buf(iobuf.out_fd, s-1, 1); /* Send the '\0'. */ @@ -434,6 +436,7 @@ static void forward_filesfrom_data(void) ff_lastchar = '\0'; else { /* Handle a partial string specially, saving any incomplete chars. */ + implied_include_partial_string(sob, s); flags &= ~ICB_INCLUDE_INCOMPLETE; if (iconvbufs(ic_send, &ff_xb, &iobuf.out, flags) < 0) { if (errno == E2BIG) @@ -450,13 +453,17 @@ static void forward_filesfrom_data(void) char *f = ff_xb.buf + ff_xb.pos; char *t = ff_xb.buf; char *eob = f + len; + char *cur = t; /* Eliminate any multi-'\0' runs. */ while (f != eob) { if (!(*t++ = *f++)) { + add_implied_include(cur, 0); + cur = t; while (f != eob && *f == '\0') f++; } } + implied_include_partial_string(cur, t); ff_lastchar = f[-1]; if ((len = t - ff_xb.buf) != 0) { /* This will not circle back to perform_io() because we only get diff --git a/lib/sysacls.c b/lib/sysacls.c index a9c3f1bd..a5abe408 100644 --- a/lib/sysacls.c +++ b/lib/sysacls.c @@ -2,7 +2,7 @@ * Unix SMB/CIFS implementation. * Based on the Samba ACL support code. * Copyright (C) Jeremy Allison 2000. - * Copyright (C) 2007-2020 Wayne Davison + * Copyright (C) 2007-2022 Wayne Davison * * The permission functions have been changed to get/set all bits via * one call. Some functions that rsync doesn't need were also removed. @@ -175,7 +175,7 @@ int sys_acl_delete_def_file(const char *name) return acl_delete_def_file(name); } -int sys_acl_free_acl(SMB_ACL_T the_acl) +int sys_acl_free_acl(SMB_ACL_T the_acl) { return acl_free(the_acl); } @@ -185,7 +185,7 @@ int sys_acl_free_acl(SMB_ACL_T the_acl) * The interface to DEC/Compaq Tru64 UNIX ACLs * is based on Draft 13 of the POSIX spec which is * slightly different from the Draft 16 interface. - * + * * Also, some of the permset manipulation functions * such as acl_clear_perm() and acl_add_perm() appear * to be broken on Tru64 so we have to manipulate @@ -310,7 +310,7 @@ int sys_acl_delete_def_file(const char *name) return acl_delete_def_file((char *)name); } -int sys_acl_free_acl(SMB_ACL_T the_acl) +int sys_acl_free_acl(SMB_ACL_T the_acl) { return acl_free(the_acl); } @@ -457,7 +457,7 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) break; } ndefault = count - naccess; - + /* * if the caller wants the default ACL we have to copy * the entries down to the start of the acl[] buffer @@ -517,7 +517,7 @@ SMB_ACL_T sys_acl_get_fd(int fd) if (acl_d->acl[naccess].a_type & ACL_DEFAULT) break; } - + acl_d->count = naccess; return acl_d; @@ -532,7 +532,7 @@ int sys_acl_get_info(SMB_ACL_ENTRY_T entry, SMB_ACL_TAG_T *tag_type_p, uint32 *b if (*tag_type_p == SMB_ACL_USER || *tag_type_p == SMB_ACL_GROUP) *u_g_id_p = entry->a_id; - + return 0; } @@ -633,7 +633,7 @@ static int acl_sort(SMB_ACL_T acl_d) } return 0; } - + int sys_acl_valid(SMB_ACL_T acl_d) { return acl_sort(acl_d); @@ -755,11 +755,11 @@ int sys_acl_delete_def_file(const char *path) ret = acl(path, SETACL, acl_d->count, acl_d->acl); sys_acl_free_acl(acl_d); - + return ret; } -int sys_acl_free_acl(SMB_ACL_T acl_d) +int sys_acl_free_acl(SMB_ACL_T acl_d) { SAFE_FREE(acl_d); return 0; @@ -895,10 +895,10 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) int ndefault; /* # of default ACL entries */ if (hpux_acl_call_presence() == False) { - /* Looks like we don't have the acl() system call on HPUX. + /* Looks like we don't have the acl() system call on HPUX. * May be the system doesn't have the latest version of JFS. */ - return NULL; + return NULL; } if (type != SMB_ACL_TYPE_ACCESS && type != SMB_ACL_TYPE_DEFAULT) { @@ -949,7 +949,7 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) break; } ndefault = count - naccess; - + /* * if the caller wants the default ACL we have to copy * the entries down to the start of the acl[] buffer @@ -1109,9 +1109,9 @@ struct hpux_acl_types { * aclp - Array of ACL structures. * acl_type_count - Pointer to acl_types structure. Should already be * allocated. - * Output: + * Output: * - * acl_type_count - This structure is filled up with counts of various + * acl_type_count - This structure is filled up with counts of various * acl types. */ @@ -1123,28 +1123,28 @@ static void hpux_count_obj(int acl_count, struct acl *aclp, struct hpux_acl_type for (i = 0; i < acl_count; i++) { switch (aclp[i].a_type) { - case USER: + case USER: acl_type_count->n_user++; break; - case USER_OBJ: + case USER_OBJ: acl_type_count->n_user_obj++; break; - case DEF_USER_OBJ: + case DEF_USER_OBJ: acl_type_count->n_def_user_obj++; break; - case GROUP: + case GROUP: acl_type_count->n_group++; break; - case GROUP_OBJ: + case GROUP_OBJ: acl_type_count->n_group_obj++; break; - case DEF_GROUP_OBJ: + case DEF_GROUP_OBJ: acl_type_count->n_def_group_obj++; break; - case OTHER_OBJ: + case OTHER_OBJ: acl_type_count->n_other_obj++; break; - case DEF_OTHER_OBJ: + case DEF_OTHER_OBJ: acl_type_count->n_def_other_obj++; break; case CLASS_OBJ: @@ -1159,14 +1159,14 @@ static void hpux_count_obj(int acl_count, struct acl *aclp, struct hpux_acl_type case DEF_GROUP: acl_type_count->n_def_group++; break; - default: + default: acl_type_count->n_illegal_obj++; break; } } } -/* swap_acl_entries: Swaps two ACL entries. +/* swap_acl_entries: Swaps two ACL entries. * * Inputs: aclp0, aclp1 - ACL entries to be swapped. */ @@ -1189,25 +1189,25 @@ static void hpux_swap_acl_entries(struct acl *aclp0, struct acl *aclp1) } /* prohibited_duplicate_type - * Identifies if given ACL type can have duplicate entries or + * Identifies if given ACL type can have duplicate entries or * not. * * Inputs: acl_type - ACL Type. * - * Outputs: + * Outputs: * - * Return.. + * Return.. * * True - If the ACL type matches any of the prohibited types. * False - If the ACL type doesn't match any of the prohibited types. - */ + */ static BOOL hpux_prohibited_duplicate_type(int acl_type) { switch (acl_type) { case USER: case GROUP: - case DEF_USER: + case DEF_USER: case DEF_GROUP: return True; default: @@ -1217,7 +1217,7 @@ static BOOL hpux_prohibited_duplicate_type(int acl_type) /* get_needed_class_perm * Returns the permissions of a ACL structure only if the ACL - * type matches one of the pre-determined types for computing + * type matches one of the pre-determined types for computing * CLASS_OBJ permissions. * * Inputs: aclp - Pointer to ACL structure. @@ -1226,17 +1226,17 @@ static BOOL hpux_prohibited_duplicate_type(int acl_type) static int hpux_get_needed_class_perm(struct acl *aclp) { switch (aclp->a_type) { - case USER: - case GROUP_OBJ: - case GROUP: - case DEF_USER_OBJ: + case USER: + case GROUP_OBJ: + case GROUP: + case DEF_USER_OBJ: case DEF_USER: - case DEF_GROUP_OBJ: + case DEF_GROUP_OBJ: case DEF_GROUP: case DEF_CLASS_OBJ: - case DEF_OTHER_OBJ: + case DEF_OTHER_OBJ: return aclp->a_perm; - default: + default: return 0; } } @@ -1267,15 +1267,15 @@ static int hpux_acl_sort(int acl_count, int calclass, struct acl *aclp) #if !defined(HAVE_HPUX_ACLSORT) /* * The aclsort() system call is available on the latest HPUX General - * Patch Bundles. So for HPUX, we developed our version of acl_sort - * function. Because, we don't want to update to a new + * Patch Bundles. So for HPUX, we developed our version of acl_sort + * function. Because, we don't want to update to a new * HPUX GR bundle just for aclsort() call. */ struct hpux_acl_types acl_obj_count; int n_class_obj_perm = 0; int i, j; - + if (!acl_count) { DEBUG(10, ("Zero acl count passed. Returning Success\n")); return 0; @@ -1290,8 +1290,8 @@ static int hpux_acl_sort(int acl_count, int calclass, struct acl *aclp) hpux_count_obj(acl_count, aclp, &acl_obj_count); - /* There should be only one entry each of type USER_OBJ, GROUP_OBJ, - * CLASS_OBJ and OTHER_OBJ + /* There should be only one entry each of type USER_OBJ, GROUP_OBJ, + * CLASS_OBJ and OTHER_OBJ */ if (acl_obj_count.n_user_obj != 1 @@ -1313,15 +1313,15 @@ or DEF_USER_OBJ or DEF_GROUP_OBJ or DEF_OTHER_OBJ\n")); return -1; } - /* We now have proper number of OBJ and DEF_OBJ entries. Now sort the acl - * structures. + /* We now have proper number of OBJ and DEF_OBJ entries. Now sort the acl + * structures. * * Sorting crieteria - First sort by ACL type. If there are multiple entries of * same ACL type, sort by ACL id. * - * I am using the trivial kind of sorting method here because, performance isn't + * I am using the trivial kind of sorting method here because, performance isn't * really effected by the ACLs feature. More over there aren't going to be more - * than 17 entries on HPUX. + * than 17 entries on HPUX. */ for (i = 0; i < acl_count; i++) { @@ -1390,7 +1390,7 @@ static int acl_sort(SMB_ACL_T acl_d) } return 0; } - + int sys_acl_valid(SMB_ACL_T acl_d) { return acl_sort(acl_d); @@ -1405,11 +1405,11 @@ int sys_acl_set_file(const char *name, SMB_ACL_TYPE_T type, SMB_ACL_T acl_d) int ret; if (hpux_acl_call_presence() == False) { - /* Looks like we don't have the acl() system call on HPUX. + /* Looks like we don't have the acl() system call on HPUX. * May be the system doesn't have the latest version of JFS. */ errno=ENOSYS; - return -1; + return -1; } if (type != SMB_ACL_TYPE_ACCESS && type != SMB_ACL_TYPE_DEFAULT) { @@ -1538,11 +1538,11 @@ int sys_acl_delete_def_file(const char *path) ret = acl(path, ACL_SET, acl_d->count, acl_d->acl); sys_acl_free_acl(acl_d); - + return ret; } -int sys_acl_free_acl(SMB_ACL_T acl_d) +int sys_acl_free_acl(SMB_ACL_T acl_d) { free(acl_d); return 0; @@ -1723,7 +1723,7 @@ int sys_acl_delete_def_file(const char *name) return acl_delete_def_file(name); } -int sys_acl_free_acl(SMB_ACL_T acl_d) +int sys_acl_free_acl(SMB_ACL_T acl_d) { if (acl_d->freeaclp) { acl_free(acl_d->aclp); @@ -1834,12 +1834,12 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) } /* Get the acl using statacl */ - + DEBUG(10, ("Entering sys_acl_get_file\n")); DEBUG(10, ("path_p is %s\n", path_p)); file_acl = (struct acl *)SMB_MALLOC(BUFSIZ); - + if (file_acl == NULL) { errno=ENOMEM; DEBUG(0, ("Error in AIX sys_acl_get_file: %d\n", errno)); @@ -1931,9 +1931,9 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) * to be specified but, it's better than leaving it 0 */ acl_entry_link->entryp->ace_type = acl_entry->ace_type; - + acl_entry_link->entryp->ace_access = acl_entry->ace_access; - + memcpy(acl_entry_link->entryp->ace_id, idp, sizeof (struct ace_id)); /* The access in the acl entries must be left shifted by * @@ -1962,7 +1962,7 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) DEBUG(10, ("acl_entry = %d\n", acl_entry)); DEBUG(10, ("The ace_type is %d\n", acl_entry->ace_type)); - + acl_entry = acl_nxt(acl_entry); } } /* end of if enabled */ @@ -2014,12 +2014,12 @@ SMB_ACL_T sys_acl_get_file(const char *path_p, SMB_ACL_TYPE_T type) new_acl_entry->ace_access = file_acl->o_access << 6; idp->id_type = SMB_ACL_OTHER; break; - + case 1: new_acl_entry->ace_access = file_acl->u_access << 6; idp->id_type = SMB_ACL_USER_OBJ; break; - + default: return NULL; @@ -2048,7 +2048,7 @@ SMB_ACL_T sys_acl_get_fd(int fd) int rc = 0; /* Get the acl using fstatacl */ - + DEBUG(10, ("Entering sys_acl_get_fd\n")); DEBUG(10, ("fd is %d\n", fd)); file_acl = (struct acl *)SMB_MALLOC(BUFSIZ); @@ -2095,12 +2095,12 @@ SMB_ACL_T sys_acl_get_fd(int fd) DEBUG(10, ("acl_entry is %d\n", acl_entry)); DEBUG(10, ("acl_last(file_acl) id %d\n", acl_last(file_acl))); - + /* Check if the extended acl bit is on. * * If it isn't, do not show the * * contents of the acl since AIX intends * * the extended info to remain unused */ - + if (file_acl->acl_mode & S_IXACL){ /* while we are not pointing to the very end */ while (acl_entry < acl_last(file_acl)) { @@ -2115,7 +2115,7 @@ SMB_ACL_T sys_acl_get_fd(int fd) } idp = acl_entry->ace_id; - + /* Check if this is the first entry in the linked list. * * The first entry needs to keep prevp pointing to NULL * * and already has entryp allocated. */ @@ -2177,7 +2177,7 @@ SMB_ACL_T sys_acl_get_fd(int fd) DEBUG(10, ("acl_entry = %d\n", acl_entry)); DEBUG(10, ("The ace_type is %d\n", acl_entry->ace_type)); - + acl_entry = acl_nxt(acl_entry); } } /* end of if enabled */ @@ -2210,43 +2210,43 @@ SMB_ACL_T sys_acl_get_fd(int fd) } acl_entry_link->nextp = NULL; - + new_acl_entry = acl_entry_link->entryp; idp = new_acl_entry->ace_id; - + new_acl_entry->ace_len = sizeof (struct acl_entry); new_acl_entry->ace_type = ACC_PERMIT; idp->id_len = sizeof (struct ace_id); DEBUG(10, ("idp->id_len = %d\n", idp->id_len)); memset(idp->id_data, 0, sizeof (uid_t)); - + switch (i) { case 2: new_acl_entry->ace_access = file_acl->g_access << 6; idp->id_type = SMB_ACL_GROUP_OBJ; break; - + case 3: new_acl_entry->ace_access = file_acl->o_access << 6; idp->id_type = SMB_ACL_OTHER; break; - + case 1: new_acl_entry->ace_access = file_acl->u_access << 6; idp->id_type = SMB_ACL_USER_OBJ; break; - + default: return NULL; } - + acl_entry_link_head->count++; DEBUG(10, ("new_acl_entry->ace_access = %d\n", new_acl_entry->ace_access)); } acl_entry_link_head->count = 0; SAFE_FREE(file_acl); - + return acl_entry_link_head; } #endif @@ -2274,7 +2274,7 @@ int sys_acl_get_info(SMB_ACL_ENTRY_T entry, SMB_ACL_TAG_T *tag_type_p, uint32 *b SMB_ACL_T sys_acl_init(int count) { struct acl_entry_link *theacl = NULL; - + if (count < 0) { errno = EINVAL; return NULL; @@ -2383,9 +2383,9 @@ int sys_acl_valid(SMB_ACL_T theacl) } DEBUG(10, ("user_obj=%d, group_obj=%d, other_obj=%d\n", user_obj, group_obj, other_obj)); - + if (user_obj != 1 || group_obj != 1 || other_obj != 1) - return -1; + return -1; return 0; } @@ -2404,7 +2404,7 @@ int sys_acl_set_file(const char *name, SMB_ACL_TYPE_T acltype, SMB_ACL_T theacl) DEBUG(10, ("Entering sys_acl_set_file\n")); DEBUG(10, ("File name is %s\n", name)); - + /* AIX has no default ACL */ if (acltype == SMB_ACL_TYPE_DEFAULT) return 0; @@ -2449,7 +2449,7 @@ int sys_acl_set_file(const char *name, SMB_ACL_TYPE_T acltype, SMB_ACL_T theacl) errno = ENOMEM; DEBUG(0, ("Error in sys_acl_set_file is %d\n", errno)); return -1; - } + } memcpy(file_acl_temp, file_acl, file_acl->acl_len); SAFE_FREE(file_acl); @@ -2460,15 +2460,15 @@ int sys_acl_set_file(const char *name, SMB_ACL_TYPE_T acltype, SMB_ACL_T theacl) file_acl->acl_len += sizeof (struct acl_entry); acl_entry->ace_len = acl_entry_link->entryp->ace_len; acl_entry->ace_access = acl_entry_link->entryp->ace_access; - + /* In order to use this, we'll need to wait until we can get denies */ /* if (!acl_entry->ace_access && acl_entry->ace_type == ACC_PERMIT) acl_entry->ace_type = ACC_SPECIFY; */ acl_entry->ace_type = ACC_SPECIFY; - + ace_id = acl_entry->ace_id; - + ace_id->id_type = acl_entry_link->entryp->ace_id->id_type; DEBUG(10, ("The id type is %d\n", ace_id->id_type)); ace_id->id_len = acl_entry_link->entryp->ace_id->id_len; @@ -2496,7 +2496,7 @@ int sys_acl_set_fd(int fd, SMB_ACL_T theacl) uint user_id; uint acl_length; uint rc; - + DEBUG(10, ("Entering sys_acl_set_fd\n")); acl_length = BUFSIZ; file_acl = (struct acl *)SMB_MALLOC(BUFSIZ); @@ -2508,7 +2508,7 @@ int sys_acl_set_fd(int fd, SMB_ACL_T theacl) } memset(file_acl, 0, BUFSIZ); - + file_acl->acl_len = ACL_SIZ; file_acl->acl_mode = S_IXACL; @@ -2550,22 +2550,22 @@ int sys_acl_set_fd(int fd, SMB_ACL_T theacl) file_acl->acl_len += sizeof (struct acl_entry); acl_entry->ace_len = acl_entry_link->entryp->ace_len; acl_entry->ace_access = acl_entry_link->entryp->ace_access; - + /* In order to use this, we'll need to wait until we can get denies */ /* if (!acl_entry->ace_access && acl_entry->ace_type == ACC_PERMIT) acl_entry->ace_type = ACC_SPECIFY; */ - + acl_entry->ace_type = ACC_SPECIFY; - + ace_id = acl_entry->ace_id; - + ace_id->id_type = acl_entry_link->entryp->ace_id->id_type; DEBUG(10, ("The id type is %d\n", ace_id->id_type)); ace_id->id_len = acl_entry_link->entryp->ace_id->id_len; memcpy(&user_id, acl_entry_link->entryp->ace_id->id_data, sizeof (uid_t)); memcpy(ace_id->id_data, &user_id, sizeof (uid_t)); } - + rc = fchacl(fd, file_acl, file_acl->acl_len); DEBUG(10, ("errno is %d\n", errno)); DEBUG(10, ("return code is %d\n", rc)); @@ -2594,7 +2594,7 @@ int sys_acl_free_acl(SMB_ACL_T posix_acl) SAFE_FREE(acl_entry_link->prevp); SAFE_FREE(acl_entry_link->entryp); SAFE_FREE(acl_entry_link); - + return 0; } diff --git a/lib/sysacls.h b/lib/sysacls.h index 8865dae4..c0695974 100644 --- a/lib/sysacls.h +++ b/lib/sysacls.h @@ -3,7 +3,7 @@ * Version 2.2.x * Portable SMB ACL interface * Copyright (C) Jeremy Allison 2000 - * Copyright (C) 2007-2020 Wayne Davison + * Copyright (C) 2007-2022 Wayne Davison * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -232,7 +232,7 @@ struct new_acl_entry{ #define SMB_ACL_ENTRY_T struct new_acl_entry* #define SMB_ACL_T struct acl_entry_link* - + #define SMB_ACL_TAG_T unsigned short #define SMB_ACL_TYPE_T int diff --git a/main.c b/main.c index 58920a2d..9ebfbea7 100644 --- a/main.c +++ b/main.c @@ -104,7 +104,7 @@ extern char curr_dir[MAXPATHLEN]; extern char backup_dir_buf[MAXPATHLEN]; extern char *basis_dir[MAX_BASIS_DIRS+1]; extern struct file_list *first_flist; -extern filter_rule_list daemon_filter_list; +extern filter_rule_list daemon_filter_list, implied_filter_list; uid_t our_uid; gid_t our_gid; @@ -1076,6 +1076,7 @@ static int do_recv(int f_in, int f_out, char *local_name) } am_generator = 1; + implied_filter_list.head = implied_filter_list.tail = NULL; flist_receiving_enabled = True; io_end_multiplex_in(MPLX_SWITCHING); @@ -1500,6 +1501,8 @@ static int start_client(int argc, char *argv[]) char *dummy_host; int dummy_port = rsync_port; int i; + if (filesfrom_fd < 0) + add_implied_include(remote_argv[0], daemon_connection); /* For remote source, any extra source args must have either * the same hostname or an empty hostname. */ for (i = 1; i < remote_argc; i++) { @@ -1523,6 +1526,7 @@ static int start_client(int argc, char *argv[]) if (!rsync_port && !*arg) /* Turn an empty arg into a dot dir. */ arg = "."; remote_argv[i] = arg; + add_implied_include(arg, daemon_connection); } } diff --git a/md-convert b/md-convert index 41a1930a..19709c8d 100755 --- a/md-convert +++ b/md-convert @@ -115,7 +115,8 @@ NBR_SPACE = ('\xa0', r"\ ") FILENAME_RE = re.compile(r'^(?P(?P.+/)?(?P(?P[^/]+?)(\.(?P\d+))?)\.md)$') ASSIGNMENT_RE = re.compile(r'^(\w+)=(.+)') -QUOTED_RE = re.compile(r'"(.+?)"') +VER_RE = re.compile(r'^#define\s+RSYNC_VERSION\s+"(\d.+?)"', re.M) +TZ_RE = re.compile(r'^#define\s+MAINTAINER_TZ_OFFSET\s+(-?\d+(\.\d+)?)', re.M) VAR_REF_RE = re.compile(r'\$\{(\w+)\}') VERSION_RE = re.compile(r' (\d[.\d]+)[, ]') BIN_CHARS_RE = re.compile(r'[\1-\7]+') @@ -213,6 +214,7 @@ def find_man_substitutions(): env_subs['VERSION'] = '1.0.0' env_subs['bindir'] = '/usr/bin' env_subs['libdir'] = '/usr/lib/rsync' + tz_offset = 0 else: for fn in (srcdir + 'version.h', 'Makefile'): try: @@ -224,8 +226,10 @@ def find_man_substitutions(): with open(srcdir + 'version.h', 'r', encoding='utf-8') as fh: txt = fh.read() - m = QUOTED_RE.search(txt) + m = VER_RE.search(txt) env_subs['VERSION'] = m.group(1) + m = TZ_RE.search(txt) # the tzdata lib may not be installed, so we use a simple hour offset + tz_offset = float(m.group(1)) * 60 * 60 with open('Makefile', 'r', encoding='utf-8') as fh: for line in fh: @@ -241,7 +245,7 @@ def find_man_substitutions(): if var == 'srcdir': break - env_subs['date'] = time.strftime('%d %b %Y', time.localtime(mtime)) + env_subs['date'] = time.strftime('%d %b %Y', time.gmtime(mtime + tz_offset)).lstrip('0') def html_via_commonmark(txt): @@ -605,12 +609,12 @@ def die(*msg): if __name__ == '__main__': - parser = argparse.ArgumentParser(description="Output html and (optionally) nroff for markdown pages.", add_help=False) + parser = argparse.ArgumentParser(description="Convert markdown into html and (optionally) nroff. Each input filename must have a .md suffix, which is changed to .html for the output filename. If the input filename ends with .num.md (e.g. foo.1.md) then a nroff file is also output with the input filename's .md suffix removed (e.g. foo.1).", add_help=False) parser.add_argument('--test', action='store_true', help="Just test the parsing without outputting any files.") - parser.add_argument('--dest', metavar='DIR', help="Put files into DIR instead of the current directory.") + parser.add_argument('--dest', metavar='DIR', help="Create files in DIR instead of the current directory.") parser.add_argument('--debug', '-D', action='count', default=0, help='Output copious info on the html parsing. Repeat for even more.') parser.add_argument("--help", "-h", action="help", help="Output this help message and exit.") - parser.add_argument("mdfiles", nargs='+', help="The source .md files to convert.") + parser.add_argument("mdfiles", metavar='FILE.md', nargs='+', help="One or more .md files to convert.") args = parser.parse_args() try: diff --git a/options.c b/options.c index 93bdb237..4feeb7e0 100644 --- a/options.c +++ b/options.c @@ -27,6 +27,8 @@ extern int module_id; extern int local_server; extern int sanitize_paths; +extern int trust_sender_args; +extern int trust_sender_filter; extern unsigned int module_dirlen; extern filter_rule_list filter_list; extern filter_rule_list daemon_filter_list; @@ -64,6 +66,7 @@ int preserve_atimes = 0; int preserve_crtimes = 0; int omit_dir_times = 0; int omit_link_times = 0; +int trust_sender = 0; int update_only = 0; int open_noatime = 0; int cvs_exclude = 0; @@ -293,7 +296,7 @@ static struct output_struct debug_words[COUNT_DEBUG+1] = { DEBUG_WORD(DELTASUM, W_SND|W_REC, "Debug delta-transfer checksumming (levels 1-4)"), DEBUG_WORD(DUP, W_REC, "Debug weeding of duplicate names"), DEBUG_WORD(EXIT, W_CLI|W_SRV, "Debug exit events (levels 1-3)"), - DEBUG_WORD(FILTER, W_SND|W_REC, "Debug filter actions (levels 1-2)"), + DEBUG_WORD(FILTER, W_SND|W_REC, "Debug filter actions (levels 1-3)"), DEBUG_WORD(FLIST, W_SND|W_REC, "Debug file-list operations (levels 1-4)"), DEBUG_WORD(FUZZY, W_REC, "Debug fuzzy scoring (levels 1-2)"), DEBUG_WORD(GENR, W_REC, "Debug generator functions"), @@ -788,6 +791,7 @@ static struct poptOption long_options[] = { {"protect-args", 's', POPT_ARG_VAL, &protect_args, 1, 0, 0}, {"no-protect-args", 0, POPT_ARG_VAL, &protect_args, 0, 0, 0}, {"no-s", 0, POPT_ARG_VAL, &protect_args, 0, 0, 0}, + {"trust-sender", 0, POPT_ARG_VAL, &trust_sender, 1, 0, 0}, {"numeric-ids", 0, POPT_ARG_VAL, &numeric_ids, 1, 0, 0 }, {"no-numeric-ids", 0, POPT_ARG_VAL, &numeric_ids, 0, 0, 0 }, {"usermap", 0, POPT_ARG_STRING, 0, OPT_USERMAP, 0, 0 }, @@ -2465,6 +2469,11 @@ int parse_arguments(int *argc_p, const char ***argv_p) } } + if (trust_sender || am_server || read_batch) + trust_sender_args = trust_sender_filter = 1; + else if (old_style_args || filesfrom_host != NULL) + trust_sender_args = 1; + am_starting_up = 0; return 1; @@ -2492,12 +2501,17 @@ char *safe_arg(const char *opt, const char *arg) BOOL is_filename_arg = !opt; char *escapes = is_filename_arg ? SHELL_CHARS : WILD_CHARS SHELL_CHARS; BOOL escape_leading_dash = is_filename_arg && *arg == '-'; + BOOL escape_leading_tilde = 0; int len1 = opt && *opt ? strlen(opt) + 1 : 0; int len2 = strlen(arg); int extras = escape_leading_dash ? 2 : 0; char *ret; if (!protect_args && old_style_args < 2 && (!old_style_args || (!is_filename_arg && opt != SPLIT_ARG_WHEN_OLD))) { const char *f; + if (!trust_sender_args && *arg == '~' && (relative_paths || !strchr(arg, '/'))) { + extras++; + escape_leading_tilde = 1; + } for (f = arg; *f; f++) { if (strchr(escapes, *f)) extras++; @@ -2520,8 +2534,13 @@ char *safe_arg(const char *opt, const char *arg) else { const char *f = arg; char *t = ret + len1; + if (escape_leading_tilde) + *t++ = '\\'; while (*f) { - if (strchr(escapes, *f)) + if (*f == '\\') { + if (!is_filename_arg || !strchr(WILD_CHARS, f[1])) + *t++ = '\\'; + } else if (strchr(escapes, *f)) *t++ = '\\'; *t++ = *f++; } diff --git a/packaging/lsb/rsync.spec b/packaging/lsb/rsync.spec index 0bdcd833..0959c744 100644 --- a/packaging/lsb/rsync.spec +++ b/packaging/lsb/rsync.spec @@ -1,6 +1,6 @@ Summary: A fast, versatile, remote (and local) file-copying tool Name: rsync -Version: 3.2.4 +Version: 3.2.5 %define fullversion %{version} Release: 1 %define srcdir src @@ -79,8 +79,8 @@ rm -rf $RPM_BUILD_ROOT %dir /etc/rsync-ssl/certs %changelog -* Fri Apr 15 2022 Wayne Davison -Released 3.2.4. +* Sun Aug 14 2022 Wayne Davison +Released 3.2.5. * Fri Mar 21 2008 Wayne Davison Added installation of /etc/xinetd.d/rsync file and some commented-out diff --git a/packaging/release-rsync b/packaging/release-rsync index d484628c..5e54d06e 100755 --- a/packaging/release-rsync +++ b/packaging/release-rsync @@ -193,7 +193,9 @@ About to: with open(fn, 'r', encoding='utf-8') as fh: old_txt = txt = fh.read() if fn == 'version.h': - txt = f'#define RSYNC_VERSION "{version}"\n' + x_re = re.compile(r'^(#define RSYNC_VERSION).*', re.M) + msg = f"Unable to update RSYNC_VERSION in {fn}" + txt = replace_or_die(x_re, r'\1 "%s"' % version, txt, msg) elif '.spec' in fn: for var, val in specvars.items(): x_re = re.compile(r'^%s .*' % re.escape(var), re.M) diff --git a/receiver.c b/receiver.c index b3a69da0..93cf8efd 100644 --- a/receiver.c +++ b/receiver.c @@ -593,10 +593,13 @@ int recv_files(int f_in, int f_out, char *local_name) if (DEBUG_GTE(RECV, 1)) rprintf(FINFO, "recv_files(%s)\n", fname); - if (daemon_filter_list.head && (*fname != '.' || fname[1] != '\0') - && check_filter(&daemon_filter_list, FLOG, fname, 0) < 0) { - rprintf(FERROR, "attempt to hack rsync failed.\n"); - exit_cleanup(RERR_PROTOCOL); + if (daemon_filter_list.head && (*fname != '.' || fname[1] != '\0')) { + int filt_flags = S_ISDIR(file->mode) ? NAME_IS_DIR : NAME_IS_FILE; + if (check_filter(&daemon_filter_list, FLOG, fname, filt_flags) < 0) { + rprintf(FERROR, "ERROR: rejecting file transfer request for daemon excluded file: %s\n", + fname); + exit_cleanup(RERR_PROTOCOL); + } } #ifdef SUPPORT_XATTRS diff --git a/rsync-ssl.1.md b/rsync-ssl.1.md index f3f93718..a6f1e3d3 100644 --- a/rsync-ssl.1.md +++ b/rsync-ssl.1.md @@ -94,6 +94,11 @@ The ssl helper scripts are affected by the following environment variables: > rsync-ssl -aiv rsync://example.com:9874/mod/ dest +## THE SERVER SIDE + +For help setting up an SSL/TLS supporting rsync, see the [instructions in +rsyncd.conf](rsyncd.conf.5#SSL_TLS_Daemon_Setup). + ## SEE ALSO [**rsync**(1)](rsync.1), [**rsyncd.conf**(5)](rsyncd.conf.5) diff --git a/rsync.1.md b/rsync.1.md index 313025df..f29495f2 100644 --- a/rsync.1.md +++ b/rsync.1.md @@ -154,6 +154,47 @@ rsync daemon by leaving off the module name: See the following section for more details. +## SORTED TRANSFER ORDER + +Rsync always sorts the specified filenames into its internal transfer list. +This handles the merging together of the contents of identically named +directories, makes it easy to remove duplicate filenames. It can, however, +confuse someone when the files are transferred in a different order than what +was given on the command-line. + +If you need a particular file to be transferred prior to another, either +separate the files into different rsync calls, or consider using +[`--delay-updates`](#opt) (which doesn't affect the sorted transfer order, but +does make the final file-updating phase happen much more rapidly). + +## MULTI-HOST SECURITY + +Rsync takes steps to ensure that the file requests that are shared in a +transfer are protected against various security issues. Most of the potential +problems arise on the receiving side where rsync takes steps to ensure that the +list of files being transferred remains within the bounds of what was +requested. + +Toward this end, rsync 3.1.2 and later have aborted when a file list contains +an absolute or relative path that tries to escape out of the top of the +transfer. Also, beginning with version 3.2.5, rsync does two more safety +checks of the file list to (1) ensure that no extra source arguments were added +into the transfer other than those that the client requested and (2) ensure +that the file list obeys the exclude rules that were sent to the sender. + +For those that don't yet have a 3.2.5 client rsync (or those that want to be +extra careful), it is safest to do a copy into a dedicated destination +directory for the remote files when you don't trust the remote host. For +example, instead of doing an rsync copy into your home directory: + +> rsync -aiv host1:dir1 ~ + +Dedicate a "host1-files" dir to the remote content: + +> rsync -aiv host1:dir1 ~/host1-files + +See the [`--trust-sender`](#opt) option for additional details. + ## ADVANCED USAGE The syntax for requesting multiple files from a remote host is done by @@ -164,20 +205,24 @@ the hostname omitted. For instance, all these work: > rsync -aiv host::modname/file{1,2} host::modname/extra /dest/ > rsync -aiv host::modname/first ::modname/extra{1,2} /dest/ -In a modern rsync, you only need to quote or backslash-escape things like -spaces from the local shell but not also from the remote shell: +Really old versions of rsync (2.6.9 and before) only allowed specifying one +remote-source arg, so some people have instead relied on the remote-shell +performing space splitting to break up an arg into multiple paths. Such +unintuitive behavior is no longer supported by default (though you can request +it, as described below). + +Starting in 3.2.4, filenames are passed to a remote shell in such a way as to +preserve the characters you give it. Thus, if you ask for a file with spaces +in the name, that's what the remote rsync looks for: > rsync -aiv host:'a simple file.pdf' /dest/ -Really old versions of rsync only allowed specifying one remote-source arg, so -it required the remote side to split the args at a space. You can still get -this old-style arg splitting by using the [`--old-args`](#opt) option: - -> rsync -ai --old-args host:'dir1/file1 dir2/file2' /dest -> rsync -ai --old-args host::'modname/dir1/file1 modname/dir2/file2' /dest - -See that option's section for an [environment variable](#RSYNC_OLD_ARGS) that -can be exported to help old scripts. +If you use scripts that have been written to manually apply extra quoting to +the remote rsync args (or to require remote arg splitting), you can ask rsync +to let your script handle the extra escaping. This is done by either adding +the [`--old-args`](#opt) option to the rsync runs in the script (which requires +a new rsync) or exporting [RSYNC_OLD_ARGS](#)=1 and [RSYNC_PROTECT_ARGS](#)=0 +(which works with old or new rsync versions). ## CONNECTING TO AN RSYNC DAEMON @@ -270,6 +315,10 @@ example that uses the short version of the [`--rsh`](#opt) option: The "ssh-user" will be used at the ssh level; the "rsync-user" will be used to log-in to the "module". +In this setup, the daemon is started by the ssh command that is accessing the +system (which can be forced via the `~/.ssh/authorized_keys` file, if desired). +However, when accessing a daemon directly, it needs to be started beforehand. + ## STARTING AN RSYNC DAEMON TO ACCEPT CONNECTIONS In order to connect to an rsync daemon, the remote system needs to have a @@ -283,48 +332,18 @@ the daemon (including stand-alone and inetd configurations). If you're using one of the remote-shell transports for the transfer, there is no need to manually start an rsync daemon. -## SORTED TRANSFER ORDER - -Rsync always sorts the specified filenames into its internal transfer list. -This handles the merging together of the contents of identically named -directories, makes it easy to remove duplicate filenames, and may confuse -someone when the files are transferred in a different order than what was given -on the command-line. - -If you need a particular file to be transferred prior to another, either -separate the files into different rsync calls, or consider using -[`--delay-updates`](#opt) (which doesn't affect the sorted transfer order, but -does make the final file-updating phase happen much more rapidly). - ## EXAMPLES -Here are some examples of how I use rsync. +Here are some examples of how rsync can be used. -To backup my wife's home directory, which consists of large MS Word files and -mail folders, I use a cron job that runs +To backup a home directory, which consists of large MS Word files and mail +folders, a per-user cron job can be used that runs this each day: -> rsync -Cavz . arvidsjaur:backup +> rsync -aiz . bkhost:backup/joe/ -each night over a PPP connection to a duplicate directory on my machine -"arvidsjaur". +To move some files from a remote host to the local host, you could run: -To synchronize my samba source trees I use the following Makefile targets: - -> get: -> rsync -avuzb --exclude '*~' samba:samba/ . -> put: -> rsync -Cavuzb . samba:samba/ -> sync: get put - -This allows me to sync with a CVS directory at the other end of the connection. -I then do CVS operations on the remote machine, which saves a lot of time as -the remote CVS protocol isn't very efficient. - -I mirror a directory between my "old" and "new" ftp sites with the command: - -> rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge" - -This is launched from cron every few hours. +> rsync -aiv --remove-source-files rhost:/tmp/{file1,file2}.c ~/src/ ## OPTION SUMMARY @@ -446,6 +465,7 @@ has its own detailed description later in this manpage. --from0, -0 all *-from/filter files are delimited by 0s --old-args disable the modern arg-protection idiom --protect-args, -s no space-splitting; wildcard chars only +--trust-sender trust the remote sender's file list --copy-as=USER[:GROUP] specify user & optional group for the copy --address=ADDRESS bind address for outgoing socket to daemon --port=PORT specify double-colon alternate port number @@ -508,14 +528,22 @@ accepted: Rsync accepts both long (double-dash + word) and short (single-dash + letter) options. The full list of the available options are described below. If an option can be specified in more than one way, the choices are comma-separated. -Some options only have a long variant, not a short. If the option takes a -parameter, the parameter is only listed after the long variant, even though it -must also be specified for the short. When specifying a parameter, you can -either use the form `--option=param` or replace the '=' with whitespace. The -parameter may need to be quoted in some manner for it to survive the shell's -command-line parsing. Keep in mind that a leading tilde (`~`) in a filename is -substituted by your shell, so `--option=~/foo` will not change the tilde into -your home directory (remove the '=' for that). +Some options only have a long variant, not a short. + +If the option takes a parameter, the parameter is only listed after the long +variant, even though it must also be specified for the short. When specifying +a parameter, you can either use the form `--option=param`, `--option param`, +`-o=param`, `-o param`, or `-oparam` (the latter choices assume that your +option has a short variant). + +The parameter may need to be quoted in some manner for it to survive the +shell's command-line parsing. Also keep in mind that a leading tilde (`~`) in +a pathname is substituted by your shell, so make sure that you separate the +option name from the pathname using a space if you want the local shell to +expand it. + +[comment]: # (Some markup below uses a literal non-breakable space when a backtick string) +[comment]: # (needs to contain a space since markdown strips spaces from the start/end) [comment]: # (An OL starting at 0 is converted into a DL by the parser.) @@ -793,7 +821,7 @@ your home directory (remove the '=' for that). In order to make [`--delete`](#opt) compatible with incremental recursion, rsync 3.0.0 made [`--delete-during`](#opt) the default delete mode (which - was first first added in 2.6.4). + was first added in 2.6.4). One side-effect of incremental recursion is that any missing sub-directories inside a recursively-scanned directory are (by default) @@ -948,9 +976,8 @@ your home directory (remove the '=' for that). directory where the destination has a file, the transfer would occur regardless of the timestamps. - This option is a transfer rule, not an exclude, so it doesn't affect the - data that goes into the file-lists, and thus it doesn't affect deletions. - It just limits the files that the receiver requests to be transferred. + This option is a [TRANSFER RULE](#TRANSFER_RULES), so don't expect any + exclude side effects. A caution for those that choose to combine [`--inplace`](#opt) with `--update`: an interrupted transfer will leave behind a partial file on the @@ -1169,7 +1196,7 @@ your home directory (remove the '=' for that). transfer, the client is the sender, so specifying the option directly unmunges symlinks while specifying it as a remote option munges symlinks. - This option has no affect when sent to a daemon via [`--remote-option`](#opt) + This option has no effect when sent to a daemon via [`--remote-option`](#opt) because the daemon configures whether it wants munged symlinks via its "`munge symlinks`" parameter. @@ -1478,7 +1505,7 @@ your home directory (remove the '=' for that). This tells rsync to treat a device on the sending side as a regular file, allowing it to be copied to a normal destination file (or another device - if `--write-devices` was also specifed). + if `--write-devices` was also specified). This option is refused by default by an rsync daemon. @@ -1733,9 +1760,8 @@ your home directory (remove the '=' for that). [`--ignore-existing`](#opt) option, no files will be updated (which can be useful if all you want to do is delete extraneous files). - This option is a transfer rule, not an exclude, so it doesn't affect the - data that goes into the file-lists, and thus it doesn't affect deletions. - It just limits the files that the receiver requests to be transferred. + This option is a [TRANSFER RULE](#TRANSFER_RULES), so don't expect any + exclude side effects. 0. `--ignore-existing` @@ -1743,9 +1769,8 @@ your home directory (remove the '=' for that). destination (this does _not_ ignore existing directories, or nothing would get done). See also [`--ignore-non-existing`](#opt). - This option is a transfer rule, not an exclude, so it doesn't affect the - data that goes into the file-lists, and thus it doesn't affect deletions. - It just limits the files that the receiver requests to be transferred. + This option is a [TRANSFER RULE](#TRANSFER_RULES), so don't expect any + exclude side effects. This option can be useful for those doing backups using the [`--link-dest`](#opt) option when they need to continue a backup run that @@ -1874,13 +1899,25 @@ your home directory (remove the '=' for that). 0. `--delete-excluded` - In addition to deleting the files on the receiving side that are not on the - sending side, this tells rsync to also delete any files on the receiving - side that are excluded (see [`--exclude`](#opt)). See the [FILTER - RULES](#) section for a way to make individual exclusions behave this way - on the receiver, and for a way to protect files from `--delete-excluded`. - See [`--delete`](#opt) (which is implied) for more details on - file-deletion. + This option turns any unqualified exclude/include rules into server-side + rules that do not affect the receiver's deletions. + + By default, an exclude or include has both a server-side effect (to "hide" + and "show" files when building the server's file list) and a receiver-side + effect (to "protect" and "risk" files when deletions are occuring). Any + rule that has no modifier to specify what sides it is executed on will be + instead treated as if it were a server-side rule only, avoiding any + "protect" effects of the rules. + + A rule can still apply to both sides even with this option specified if the + rule is given both the sender & receiver modifer letters (e.g., `-f'-sr + foo'`). Receiver-side protect/risk rules can also be explicitly specified + to limit the deletions. This saves you from having to edit a bunch of + `-f'- foo'` rules into `-f'-s foo'` (aka `-f'H foo'`) rules (not to mention + the corresponding includes). + + See the [FILTER RULES](#) section for more information. See + [`--delete`](#opt) (which is implied) for more details on deletion. 0. `--ignore-missing-args` @@ -1941,9 +1978,8 @@ your home directory (remove the '=' for that). the numeric units or left unqualified to specify bytes. Feel free to use a fractional value along with the units, such as `--max-size=1.5m`. - This option is a transfer rule, not an exclude, so it doesn't affect the - data that goes into the file-lists, and thus it doesn't affect deletions. - It just limits the files that the receiver requests to be transferred. + This option is a [TRANSFER RULE](#TRANSFER_RULES), so don't expect any + exclude side effects. The first letter of a units string can be `B` (bytes), `K` (kilo), `M` (mega), `G` (giga), `T` (tera), or `P` (peta). If the string is a single @@ -2200,8 +2236,8 @@ your home directory (remove the '=' for that). 0. `--exclude=PATTERN` This option is a simplified form of the [`--filter`](#opt) option that - defaults to an exclude rule and does not allow the full rule-parsing syntax - of normal filter rules. + specifies an exclude rule and does not allow the full rule-parsing syntax + of normal filter rules. This is equivalent to specifying `-f'- PATTERN'`. See the [FILTER RULES](#) section for detailed information on this option. @@ -2212,13 +2248,20 @@ your home directory (remove the '=' for that). file are ignored, as are whole-line comments that start with '`;`' or '`#`' (filename rules that contain those characters are unaffected). + If a line begins with "`- `" (dash, space) or "`+ `" (plus, space), then + the type of rule is being explicitly specified as an exclude or an include + (respectively). Any rules without such a prefix are taken to be an exclude. + + If a line consists of just "`!`", then the current filter rules are cleared + before adding any further rules. + If _FILE_ is '`-`', the list will be read from standard input. 0. `--include=PATTERN` This option is a simplified form of the [`--filter`](#opt) option that - defaults to an include rule and does not allow the full rule-parsing syntax - of normal filter rules. + specifies an include rule and does not allow the full rule-parsing syntax + of normal filter rules. This is equivalent to specifying `-f'+ PATTERN'`. See the [FILTER RULES](#) section for detailed information on this option. @@ -2229,6 +2272,13 @@ your home directory (remove the '=' for that). file are ignored, as are whole-line comments that start with '`;`' or '`#`' (filename rules that contain those characters are unaffected). + If a line begins with "`- `" (dash, space) or "`+ `" (plus, space), then + the type of rule is being explicitly specified as an exclude or an include + (respectively). Any rules without such a prefix are taken to be an include. + + If a line consists of just "`!`", then the current filter rules are cleared + before adding any further rules. + If _FILE_ is '`-`', the list will be read from standard input. 0. `--files-from=FILE` @@ -2323,6 +2373,12 @@ your home directory (remove the '=' for that). behavior. The environment is always overridden by manually specified positive or negative options (the negative is `--no-old-args`). + Note that this option also disables the extra safety check added in 3.2.5 + that ensures that a remote sender isn't including extra top-level items in + the file-list that you didn't request. This side-effect is necessary + because we can't know for sure what names to expect when the remote shell + is interpreting the args. + This option conflicts with the [`--protect-args`](#opt) option. 0. `--protect-args`, `-s` @@ -2356,6 +2412,42 @@ your home directory (remove the '=' for that). Note that this option is incompatible with the use of the restricted rsync script (`rrsync`) since it hides options from the script's inspection. +0. `--trust-sender` + + This option disables two extra validation checks that a local client + performs on the file list generated by a remote sender. This option should + only be used if you trust the sender to not put something malicious in the + file list (something that could possibly be done via a modified rsync, a + modified shell, or some other similar manipulation). + + Normally, the rsync client (as of version 3.2.5) runs two extra validation + checks when pulling files from a remote rsync: + + - It verifies that additional arg items didn't get added at the top of the + transfer. + - It verifies that none of the items in the file list are names that should + have been excluded (if filter rules were specified). + + Note that various options can turn off one or both of these checks if the + option interferes with the validation. For instance: + + - Using a per-directory filter file reads filter rules that only the server + knows about, so the filter checking is disabled. + - Using the [`--old-args`](#opt) option allows the sender to manipulate the + requested args, so the arg checking is disabled. + - Reading the files-from list from the server side means that the client + doesn't know the arg list, so the arg checking is disabled. + - Using [`--read-batch`](#opt) disables both checks since the batch file's + contents will have been verified when it was created. + + This option may help an under-powered client server if the extra pattern + matching is slowing things down on a huge transfer. It can also be used to + work around a currently-unknown bug in the verification logic for a transfer + from a trusted sender. + + When using this option it is a good idea to specify a dedicated destination + directory, as discussed in the [MULTI-HOST SECURITY](#) section. + 0. `--copy-as=USER[:GROUP]` This option instructs rsync to use the USER and (if specified after a @@ -3017,7 +3109,7 @@ your home directory (remove the '=' for that). of "%i %n%L". See the [`--log-file-format`](#opt) option if you wish to override this. - Here's a example command that requests the remote side to log what is + Here's an example command that requests the remote side to log what is happening: > rsync -av --remote-option=--log-file=/tmp/rlog src/ dest/ @@ -3255,10 +3347,8 @@ your home directory (remove the '=' for that). directories when the sending rsync is recursively scanning a hierarchy of files using include/exclude/filter rules. - Note that the use of transfer rules, such as the [`--min-size`](#opt) - option, does not affect what goes into the file list, and thus does not - leave directories empty, even if none of the files in a directory match the - transfer rule. + This option can still leave empty directories on the receiving side if you + make use of [TRANSFER_RULES](#). Because the file-list is actually being pruned, this option also affects what directories get deleted when a delete is active. However, keep in @@ -3394,8 +3484,8 @@ your home directory (remove the '=' for that). include the destination. CAUTION: keep in mind that a source arg with a wild-card is expanded by the - shell into multiple args, so it is never safe to try to list such an arg - without using this option. For example: + shell into multiple args, so it is never safe to try to specify a single + wild-card arg to try to infer this option. A safe example is: > rsync -av --list-only foo* dest/ @@ -3721,27 +3811,146 @@ The options allowed when starting an rsync daemon are as follows: ## FILTER RULES -The filter rules allow for flexible selection of which files to transfer -(include) and which files to skip (exclude). The rules either directly specify -include/exclude patterns or they specify a way to acquire more include/exclude -patterns (e.g. to read them from a file). - -As the list of files/directories to transfer is built, rsync checks each name -to be transferred against the list of include/exclude patterns in turn, and the -first matching pattern is acted on: if it is an exclude pattern, then that file -is skipped; if it is an include pattern then that filename is not skipped; if -no matching pattern is found, then the filename is not skipped. - -Aside: because the interactions of filter rules can be complex, it is useful to -use the `--debug=FILTER` option if things aren't working the way you expect. -The level-1 output (the default if no level number is specified) mentions the -filter rule that is first matched by each file in the transfer. It also warns -if a filter rule has trailing whitespace. The level-2 output mentions a lot -more filter events, including the definition of each rule and the handling of -per-directory filter files. - -Rsync builds an ordered list of filter rules as specified on the command-line. -Filter rules have the following syntax: +The filter rules allow for custom control of several aspects of how files are +handled: + +- Control which files the sending side puts into the file list that describes + the transfer hierarchy +- Control which files the receiving side protects from deletion when the file + is not in the sender's file list +- Control which extended attribute names are skipped when copying xattrs + +The rules are either directly specified via option arguments or they can be +read in from one or more files. The filter-rule files can even be a part of +the hierarchy of files being copied, affecting different parts of the tree in +different ways. + +### SIMPLE INCLUDE/EXCLUDE RULES + +We will first cover the basics of how include & exclude rules affect what files +are transferred, ignoring any deletion side-effects. Filter rules mainly +affect the contents of directories that rsync is "recursing" into, but they can +also affect a top-level item in the transfer that was specified as a argument. + +The default for any unmatched file/dir is for it to be included in the +transfer, which puts the file/dir into the sender's file list. The use of an +exclude rule causes one or more matching files/dirs to be left out of the +sender's file list. An include rule can be used to limit the effect of an +exclude rule that is matching too many files. + +The order of the rules is important because the first rule that matches is the +one that takes effect. Thus, if an early rule excludes a file, no include rule +that comes after it can have any effect. This means that you must place any +include overrides somewhere prior to the exclude that it is intended to limit. + +When a directory is excluded, all its contents and sub-contents are also +excluded. The sender doesn't scan through any of it at all, which can save a +lot of time when skipping large unneeded sub-trees. + +It is also important to understand that the include/exclude rules are applied +to every file and directory that the sender is recursing into. Thus, if you +want a particular deep file to be included, you have to make sure that none of +the directories that must be traversed on the way down to that file are +excluded or else the file will never be discovered to be included. As an +example, if the directory "`a/path`" was given as a transfer argument and you +want to ensure that the file "`a/path/down/deep/wanted.txt`" is a part of the +transfer, then the sender must not exclude the directories "`a/path`", +"`a/path/down`", or "`a/path/down/deep`" as it makes it way scanning through +the file tree. + +When you are working on the rules, it can be helpful to ask rsync to tell you +what is being excluded/included and why. Specifying `--debug=FILTER` or (when +pulling files) `-M--debug=FILTER` turns on level 1 of the FILTER debug +information that will output a message any time that a file or directory is +included or excluded and which rule it matched. Beginning in 3.2.4 it will +also warn if a filter rule has trailing whitespace, since an exclude of "foo " +(with a trailing space) will not exclude a file named "foo". + +Exclude and include rules can specify wildcard [PATTERN MATCHING RULES](#) +(similar to shell wildcards) that allow you to match things like a file suffix +or a portion of a filename. + +A rule can be limited to only affecting a directory by putting a trailing slash +onto the filename. + +### SIMPLE INCLUDE/EXCLUDE EXAMPLE + +With the following file tree created on the sending side: + +> mkdir x/ +> touch x/file.txt +> mkdir x/y/ +> touch x/y/file.txt +> touch x/y/zzz.txt +> mkdir x/z/ +> touch x/z/file.txt + +Then the following rsync command will transfer the file "`x/y/file.txt`" and +the directories needed to hold it, resulting in the path "`/tmp/x/y/file.txt`" +existing on the remote host: + +> rsync -ai -f'+ x/' -f'+ x/y/' -f'+ x/y/file.txt' -f'- *' x host:/tmp/ + +Aside: this copy could also have been accomplished using the [`-R`](#opt) +option (though the 2 commands behave differently if deletions are enabled): + +> rsync -aiR x/y/file.txt host:/tmp/ + +The following command does not need an include of the "x" directory because it +is not a part of the transfer (note the traililng slash). Running this command +would copy just "`/tmp/x/file.txt`" because the "y" and "z" dirs get excluded: + +> rsync -ai -f'+ file.txt' -f'- *' x/ host:/tmp/x/ + +This command would omit the zzz.txt file while copying "x" and everything else +it contains: + +> rsync -ai -f'- zzz.txt' x host:/tmp/ + +### FILTER RULES WHEN DELETING + +By default the include & exclude filter rules affect both the sender +(as it creates its file list) +and the receiver (as it creates its file lists for calculating deletions). If +no delete option is in effect, the receiver skips creating the delete-related +file lists. This two-sided default can be manually overridden so that you are +only specifying sender rules or receiver rules, as described in the [FILTER +RULES IN DEPTH](#) section. + +When deleting, an exclude protects a file from being removed on the receiving +side while an include overrides that protection (putting the file at risk of +deletion). The default is for a file to be at risk -- its safety depends on it +matching a corresponding file from the sender. + +An example of the two-sided exclude effect can be illustrated by the copying of +a C development directory between 2 systems. When doing a touch-up copy, you +might want to skip copying the built executable and the `.o` files (sender +hide) so that the receiving side can build their own and not lose any object +files that are already correct (receiver protect). For instance: + +> rsync -ai --del -f'- *.o' -f'- cmd' src host:/dest/ + +Note that using `-f'-p *.o'` is even better than `-f'- *.o'` if there is a +chance that the directory structure may have changed. The "p" modifier is +discussed in [FILTER RULE MODIFIERS](#). + +One final note, if your shell doesn't mind unexpanded wildcards, you could +simplify the typing of the filter options by using an underscore in place of +the space and leaving off the quotes. For instance, `-f -_*.o -f -_cmd` (and +similar) could be used instead of the filter options above. + +### FILTER RULES IN DEPTH + +Rsync supports old-style include/exclude rules and new-style filter rules. The +older rules are specified using [`--include`](#opt) and [`--exclude`](#opt) as +well as the [`--include-from`](#opt) and [`--exclude-from`](#opt). These are +limited in behavior but they don't require a "-" or "+" prefix. An old-style +exclude rule is turned into a "`- name`" filter rule (with no modifiers) and an +old-style include rule is turned into a "`+ name`" filter rule (with no +modifiers). + +Rsync builds an ordered list of filter rules as specified on the command-line +and/or read-in from files. New style filter rules have the following syntax: > RULE [PATTERN_OR_FILENAME] > RULE,MODIFIERS [PATTERN_OR_FILENAME] @@ -3749,35 +3958,36 @@ Filter rules have the following syntax: You have your choice of using either short or long RULE names, as described below. If you use a short-named rule, the ',' separating the RULE from the MODIFIERS is optional. The PATTERN or FILENAME that follows (when present) -must come after either a single space or an underscore (\_). Here are the -available rule prefixes: +must come after either a single space or an underscore (\_). Any additional +spaces and/or underscores are considered to be a part of the pattern name. +Here are the available rule prefixes: -0. `exclude, '-'` specifies an exclude pattern. -0. `include, '+'` specifies an include pattern. -0. `merge, '.'` specifies a merge-file to read for more rules. -0. `dir-merge, ':'` specifies a per-directory merge-file. +0. `exclude, '-'` specifies an exclude pattern that (by default) is both a + `hide` and a `protect`. +0. `include, '+'` specifies an include pattern that (by default) is both a + `show` and a `risk`. +0. `merge, '.'` specifies a merge-file on the client side to read for more + rules. +0. `dir-merge, ':'` specifies a per-directory merge-file. Using this kind of + filter rule requires that you trust the sending side's filter checking, so + it has the side-effect mentioned under the [`--trust-sender`](#opt) option. 0. `hide, 'H'` specifies a pattern for hiding files from the transfer. -0. `show, 'S'` files that match the pattern are not hidden. + Equivalent to a sender-only exclude, so `-f'H foo'` could also be specified + as `-f'-s foo'`. +0. `show, 'S'` files that match the pattern are not hidden. Equivalent to a + sender-only include, so `-f'S foo'` could also be specified as `-f'+s + foo'`. 0. `protect, 'P'` specifies a pattern for protecting files from deletion. -0. `risk, 'R'` files that match the pattern are not protected. + Equivalent to a receiver-only exclude, so `-f'P foo'` could also be + specified as `-f'-r foo'`. +0. `risk, 'R'` files that match the pattern are not protected. Equivalent to a + receiver-only include, so `-f'R foo'` could also be specified as `-f'+r + foo'`. 0. `clear, '!'` clears the current include/exclude list (takes no arg) -When rules are being read from a file, empty lines are ignored, as are -whole-line comments that start with a '`#`' (filename rules that contain a hash -are unaffected). - -[comment]: # (Remember that markdown strips spaces from start/end of ` ... ` sequences!) -[comment]: # (Thus, the `x ` sequences below use a literal non-breakable space!) - -Note that the [`--include`](#opt) & [`--exclude`](#opt) command-line options do -not allow the full range of rule parsing as described above -- they only allow -the specification of include / exclude patterns plus a "`!`" token to clear the -list (and the normal comment parsing when rules are read from a file). If a -pattern does not begin with "`- `" (dash, space) or "`+ `" (plus, space), then -the rule will be interpreted as if "`+ `" (for an include option) or "`- `" -(for an exclude option) were prefixed to the string. A [`--filter`](#opt) -option, on the other hand, must always contain either a short or long rule name -at the start of the rule. +When rules are being read from a file (using merge or dir-merge), empty lines +are ignored, as are whole-line comments that start with a '`#`' (filename rules +that contain a hash character are unaffected). Note also that the [`--filter`](#opt), [`--include`](#opt), and [`--exclude`](#opt) options take one rule/pattern each. To add multiple ones, @@ -3785,121 +3995,90 @@ you can repeat the options on the command-line, use the merge-file syntax of the [`--filter`](#opt) option, or the [`--include-from`](#opt) / [`--exclude-from`](#opt) options. -## INCLUDE/EXCLUDE PATTERN RULES +### PATTERN MATCHING RULES -You can include and exclude files by specifying patterns using the "+", "-", -etc. filter rules (as introduced in the [FILTER RULES](#) section above). The -include/exclude rules each specify a pattern that is matched against the names -of the files that are going to be transferred. These patterns can take several -forms: +Most of the rules mentioned above take an argument that specifies what the rule +should match. If rsync is recursing through a directory hierarchy, keep in +mind that each pattern is matched against the name of every directory in the +descent path as rsync finds the filenames to send. -- if the pattern starts with a `/` then it is anchored to a particular spot in - the hierarchy of files, otherwise it is matched against the end of the - pathname. This is similar to a leading `^` in regular expressions. Thus - `/foo` would match a name of "foo" at either the "root of the transfer" (for - a global rule) or in the merge-file's directory (for a per-directory rule). - An unqualified `foo` would match a name of "foo" anywhere in the tree because - the algorithm is applied recursively from the top down; it behaves as if each - path component gets a turn at being the end of the filename. Even the - unanchored "sub/foo" would match at any point in the hierarchy where a "foo" - was found within a directory named "sub". See the section on ANCHORING - INCLUDE/EXCLUDE PATTERNS for a full discussion of how to specify a pattern - that matches at the root of the transfer. -- if the pattern ends with a `/` then it will only match a directory, not a - regular file, symlink, or device. -- rsync chooses between doing a simple string match and wildcard matching by - checking if the pattern contains one of these three wildcard characters: - '`*`', '`?`', and '`[`' . -- a '`*`' matches any path component, but it stops at slashes. -- use '`**`' to match anything, including slashes. -- a '`?`' matches any character except a slash (`/`). -- a '`[`' introduces a character class, such as `[a-z]` or `[[:alpha:]]`. -- in a wildcard pattern, a backslash can be used to escape a wildcard - character, but it is matched literally when no wildcards are present. This - means that there is an extra level of backslash removal when a pattern - contains wildcard characters compared to a pattern that has none. e.g. if - you add a wildcard to "`foo\bar`" (which matches the backslash) you would - need to use "`foo\\bar*`" to avoid the "`\b`" becoming just "b". -- if the pattern contains a `/` (not counting a trailing /) or a "`**`", then it - is matched against the full pathname, including any leading directories. If - the pattern doesn't contain a `/` or a "`**`", then it is matched only against - the final component of the filename. (Remember that the algorithm is applied - recursively so "full filename" can actually be any portion of a path from the - starting directory on down.) -- a trailing "`dir_name/***`" will match both the directory (as if "dir_name/" +The matching rules for the pattern argument take several forms: + +- If a pattern contains a `/` (not counting a trailing slash) or a "`**`" + (which can match a slash), then the pattern is matched against the full + pathname, including any leading directories within the transfer. If the + pattern doesn't contain a (non-trailing) `/` or a "`**`", then it is matched + only against the final component of the filename or pathname. For example, + `foo` means that the final path component must be "foo" while `foo/bar` would + match the last 2 elements of the path (as long as both elements are within + the transfer). +- A pattern that ends with a `/` only matches a directory, not a regular file, + symlink, or device. +- A pattern that starts with a `/` is anchored to the start of the transfer + path instead of the end. For example, `/foo/**` or `/foo/bar/**` match only + leading elements in the path. If the rule is read from a per-directory + filter file, the transfer path being matched will begin at the level of the + filter file instead of the top of the transfer. See the section on + [ANCHORING INCLUDE/EXCLUDE PATTERNS](#) for a full discussion of how to + specify a pattern that matches at the root of the transfer. + +Rsync chooses between doing a simple string match and wildcard matching by +checking if the pattern contains one of these three wildcard characters: '`*`', +'`?`', and '`[`' : + +- a '`?`' matches any single character except a slash (`/`). +- a '`*`' matches zero or more non-slash characters. +- a '`**`' matches zero or more characters, including slashes. +- a '`[`' introduces a character class, such as `[a-z]` or `[[:alpha:]]`, that + must match one character. +- a trailing `***` in the pattern is a shorthand that allows you to match a + directory and all its contents using a single rule. For example, specifying + "`dir_name/***`" will match both the "dir_name" directory (as if "`dir_name/`" had been specified) and everything in the directory (as if "`dir_name/**`" - had been specified). This behavior was added in version 2.6.7. - -Note that, when using the [`--recursive`](#opt) (`-r`) option (which is implied -by [`-a`](#opt)), every subdir component of every path is visited left to -right, with each directory having a chance for exclusion before its content. -In this way include/exclude patterns are applied recursively to the pathname of -each node in the filesystem's tree (those inside the transfer). The exclude -patterns short-circuit the directory traversal stage as rsync finds the files -to send. - -For instance, to include "`/foo/bar/baz`", the directories "`/foo`" and "`/foo/bar`" -must not be excluded. Excluding one of those parent directories prevents the -examination of its content, cutting off rsync's recursion into those paths and -rendering the include for "`/foo/bar/baz`" ineffectual (since rsync can't match -something it never sees in the cut-off section of the directory hierarchy). - -The concept path exclusion is particularly important when using a trailing '`*`' -rule. For instance, this won't work: - -> + /some/path/this-file-will-not-be-found -> + /file-is-included -> - * - -This fails because the parent directory "some" is excluded by the '`*`' rule, so -rsync never visits any of the files in the "some" or "some/path" directories. -One solution is to ask for all directories in the hierarchy to be included by -using a single rule: "`+ */`" (put it somewhere before the "`- *`" rule), and -perhaps use the [`--prune-empty-dirs`](#opt) option. Another solution is to add -specific include rules for all the parent dirs that need to be visited. For -instance, this set of rules works fine: - -> + /some/ -> + /some/path/ -> + /some/path/this-file-is-found -> + /file-also-included -> - * + had been specified). +- a backslash can be used to escape a wildcard character, but it is only + interpreted as an escape character if at least one wildcard character is + present in the match pattern. For instance, the pattern "`foo\bar`" matches + that single backslash literally, while the pattern "`foo\bar*`" would need to + be changed to "`foo\\bar*`" to avoid the "`\b`" becoming just "b". Here are some examples of exclude/include matching: -- "`- *.o`" would exclude all names matching `*.o` -- "`- /foo`" would exclude a file (or directory) named foo in the transfer-root - directory -- "`- foo/`" would exclude any directory named foo -- "`- /foo/*/bar`" would exclude any file named bar which is at two levels - below a directory named foo in the transfer-root directory -- "`- /foo/**/bar`" would exclude any file named bar two or more levels below a - directory named foo in the transfer-root directory -- The combination of "`+ */`", "`+ *.c`", and "`- *`" would include all - directories and C source files but nothing else (see also the - [`--prune-empty-dirs`](#opt) option) -- The combination of "`+ foo/`", "`+ foo/bar.c`", and "`- *`" would include - only the foo directory and foo/bar.c (the foo directory must be explicitly - included or it would be excluded by the "`*`") +- Option `-f'- *.o'` would exclude all filenames ending with `.o` +- Option `-f'- /foo'` would exclude a file (or directory) named foo in the + transfer-root directory +- Option `-f'- foo/'` would exclude any directory named foo +- Option `-f'- foo/*/bar'` would exclude any file/dir named bar which is at two + levels below a directory named foo (if foo is in the transfer) +- Option `-f'- /foo/**/bar'` would exclude any file/dir named bar that was two + or more levels below a top-level directory named foo (note that /foo/bar is + **not** excluded by this) +- Options `-f'+ */' -f'+ *.c' -f'- *'` would include all directories and .c + source files but nothing else +- Options `-f'+ foo/' -f'+ foo/bar.c' -f'- *'` would include only the foo + directory and foo/bar.c (the foo directory must be explicitly included or it + would be excluded by the "`- *`") -The following modifiers are accepted after a "`+`" or "`-`": +### FILTER RULE MODIFIERS + +The following modifiers are accepted after an include (+) or exclude (-) rule: - A `/` specifies that the include/exclude rule should be matched against the - absolute pathname of the current item. For example, "`-/ /etc/passwd`" would - exclude the passwd file any time the transfer was sending files from the - "/etc" directory, and "-/ subdir/foo" would always exclude "foo" when it is - in a dir named "subdir", even if "foo" is at the root of the current + absolute pathname of the current item. For example, `-f'-/ /etc/passwd'` + would exclude the passwd file any time the transfer was sending files from + the "/etc" directory, and "-/ subdir/foo" would always exclude "foo" when it + is in a dir named "subdir", even if "foo" is at the root of the current transfer. - A `!` specifies that the include/exclude should take effect if the pattern - fails to match. For instance, "`-! */`" would exclude all non-directories. + fails to match. For instance, `-f'-! */'` would exclude all non-directories. - A `C` is used to indicate that all the global CVS-exclude rules should be inserted as excludes in place of the "-C". No arg should follow. - An `s` is used to indicate that the rule applies to the sending side. When a - rule affects the sending side, it prevents files from being transferred. The - default is for a rule to affect both sides unless [`--delete-excluded`](#opt) - was specified, in which case default rules become sender-side only. See also - the hide (H) and show (S) rules, which are an alternate way to specify - sending-side includes/excludes. + rule affects the sending side, it affects what files are put into the + sender's file list. The default is for a rule to affect both sides unless + [`--delete-excluded`](#opt) was specified, in which case default rules become + sender-side only. See also the hide (H) and show (S) rules, which are an + alternate way to specify sending-side includes/excludes. - An `r` is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the `s` modifier for more info. See also the protect (P) and risk (R) rules, @@ -3915,7 +4094,7 @@ The following modifiers are accepted after a "`+`" or "`-`": xattr-matching rules are specified, a default xattr filtering rule is used (see the [`--xattrs`](#opt) option). -## MERGE-FILE FILTER RULES +### MERGE-FILE FILTER RULES You can merge whole files into your filter rules by specifying either a merge (.) or a dir-merge (:) filter rule (as introduced in the [FILTER RULES](#) @@ -4049,7 +4228,7 @@ $HOME/.cvsignore, and the value of $CVSIGNORE) you should omit the `-C` command-line option and instead insert a "-C" rule into your filter rules; e.g. "`--filter=-C`". -## LIST-CLEARING FILTER RULE +### LIST-CLEARING FILTER RULE You can clear the current include/exclude list by using the "!" filter rule (as introduced in the [FILTER RULES](#) section above). The "current" list is either @@ -4057,7 +4236,7 @@ the global list of rules (if the rule is encountered while parsing the filter options) or a set of per-directory rules (which are inherited in their own sub-list, so a subdirectory can use this to clear out the parent's rules). -## ANCHORING INCLUDE/EXCLUDE PATTERNS +### ANCHORING INCLUDE/EXCLUDE PATTERNS As mentioned earlier, global include/exclude patterns are anchored at the "root of the transfer" (as opposed to per-directory patterns, which are anchored at @@ -4112,7 +4291,7 @@ The easiest way to see what name you should filter is to just look at the output when using [`--verbose`](#opt) and put a / in front of the name (use the `--dry-run` option if you're not yet ready to copy any files). -## PER-DIRECTORY RULES AND DELETE +### PER-DIRECTORY RULES AND DELETE Without a delete option, per-directory rules are only relevant on the sending side, so you can feel free to exclude the merge files themselves without @@ -4158,6 +4337,38 @@ one of these commands: > rsync -avFF --delete host:src/dir /dest > ``` +## TRANSFER RULES + +In addition to the [FILTER RULES](#) that affect the recursive file scans that +generate the file list on the sending and (when deleting) receiving sides, +there are transfer rules. These rules affect which files the generator decides +need to be transferred without the side effects of an exclude filter rule. +Transfer rules affect only files and never directories. + +Because a transfer rule does not affect what goes into the sender's (and +receiver's) file list, it cannot have any effect on which files get deleted on +the receiving side. For example, if the file "foo" is present in the sender's +list but its size is such that it is omitted due to a transfer rule, the +receiving side does not request the file. However, its presence in the file +list means that a delete pass will not remove a matching file named "foo" on +the receiving side. On the other hand, a server-side exclude (hide) of the +file "foo" leaves the file out of the server's file list, and absent a +receiver-side exclude (protect) the receiver will remove a matching file named +"foo" if deletions are requested. + +Given that the files are still in the sender's file list, the +[`--prune-empty-dirs`](#opt) option will not judge a directory as being empty +even if it contains only files that the transfer rules omitted. + +Similarly, a transfer rule does not have any extra effect on which files are +deleted on the receiving side, so setting a maximum file size for the transfer +does not prevent big files from being deleted. + +Examples of transfer rules include the default "quick check" algorithm (which +compares size & modify time), the [`--update`](#opt) option, the +[`--max-size`](#opt) option, the [`--ignore-non-existing`](#opt) option, and a +few others. + ## BATCH MODE Batch mode can be used to apply the same set of updates to many identical @@ -4466,7 +4677,7 @@ file is included or excluded. 0. `RSYNC_SHELL` This environment variable is mainly used in debug setups to set the program - to use to run the program specified by [`RSYNC_CONNECT_PROG`]. See + to use to run the program specified by [`RSYNC_CONNECT_PROG`](#). See [CONNECTING TO AN RSYNC DAEMON](#) for full details. ## FILES diff --git a/rsync.c b/rsync.c index 102e72ed..cd288f57 100644 --- a/rsync.c +++ b/rsync.c @@ -642,7 +642,7 @@ int set_file_attrs(const char *fname, struct file_struct *file, stat_x *sxp, #ifdef SUPPORT_ACLS /* It's OK to call set_acl() now, even for a dir, as the generator * will enable owner-writability using chmod, if necessary. - * + * * If set_acl() changes permission bits in the process of setting * an access ACL, it changes sxp->st.st_mode so we know whether we * need to chmod(). */ diff --git a/rsync.h b/rsync.h index e5aacd25..1cc037c5 100644 --- a/rsync.h +++ b/rsync.h @@ -338,6 +338,9 @@ enum delret { # endif # include #endif +#ifdef HAVE_BSD_STRING_H +# include +#endif #ifdef HAVE_STRINGS_H # include #endif diff --git a/rsyncd.conf.5.md b/rsyncd.conf.5.md index c61765a7..8bcbec0a 100644 --- a/rsyncd.conf.5.md +++ b/rsyncd.conf.5.md @@ -1114,15 +1114,15 @@ SSL proxy. ## SSL/TLS Daemon Setup When setting up an rsync daemon for access via SSL/TLS, you will need to -configure a proxy (such as haproxy or nginx) as the front-end that handles the -encryption. +configure a TCP proxy (such as haproxy or nginx) as the front-end that handles +the encryption. - You should limit the access to the backend-rsyncd port to only allow the proxy to connect. If it is on the same host as the proxy, then configuring it to only listen on localhost is a good idea. -- You should consider turning on the `proxy protocol` parameter if your proxy - supports sending that information. The examples below assume that this is - enabled. +- You should consider turning on the `proxy protocol` rsync-daemon parameter if + your proxy supports sending that information. The examples below assume that + this is enabled. An example haproxy setup is as follows: @@ -1149,14 +1149,14 @@ An example nginx proxy setup is as follows: > ssl_certificate_key /etc/letsencrypt/example.com/privkey.pem; > > proxy_pass localhost:873; -> proxy_protocol on; # Requires "proxy protocol = true" +> proxy_protocol on; # Requires rsyncd.conf "proxy protocol = true" > proxy_timeout 1m; > proxy_connect_timeout 5s; > } > } > ``` -## EXAMPLES +## DAEMON CONFIG EXAMPLES A simple rsyncd.conf file that allow anonymous rsync to a ftp area at `/home/ftp` would be: @@ -1237,8 +1237,9 @@ Thanks to Karsten Thygesen for his many suggestions and documentation! ## AUTHOR -Rsync was written by Andrew Tridgell and Paul Mackerras. Many people have -later contributed to it. +Rsync was originally written by Andrew Tridgell and Paul Mackerras. Many +people have later contributed to it. It is currently maintained by Wayne +Davison. Mailing lists for support and development are available at . diff --git a/support/rrsync.1.md b/support/rrsync.1.md index 54980db0..a7365323 100644 --- a/support/rrsync.1.md +++ b/support/rrsync.1.md @@ -11,14 +11,16 @@ rrsync [-ro|-rw] [-munge] [-no-del] [-no-lock] DIR The single non-option argument specifies the restricted _DIR_ to use. It can be relative to the user's home directory or an absolute path. -The online version of this man page (that includes cross-linking of topics) +The online version of this manpage (that includes cross-linking of topics) is available at . ## DESCRIPTION A user's ssh login can be restricted to only allow the running of an rsync -transfer in one of two easy ways: forcing the running of the rrsync script -or forcing the running of an rsync daemon-over-ssh command. +transfer in one of two easy ways: + +* forcing the running of the rrsync script +* forcing the running of an rsync daemon-over-ssh command. To use the rrsync script, edit the user's `~/.ssh/authorized_keys` file and add a prefix like one of the following (followed by a space) in front of each @@ -47,13 +49,14 @@ ssh-key line that should be restricted: Then, ensure that the rsyncd.conf file is created with one or more module names with the appropriate path and option restrictions. If rsync's [`--config`](rsync.1#dopt) option is omitted, it defaults to `~/rsyncd.conf`. -See the `rsyncd.conf` man page for details of how to configure an rsync daemon. +See the [**rsyncd.conf**(5)](rsyncd.conf.5) manpage for details of how to +configure an rsync daemon. When using rrsync, there can be just one restricted dir per authorized key. A daemon setup, on the other hand, allows multiple module names inside the config file, each one with its own path setting. -The remainder of this man page is dedicated to using the rrsync script. +The remainder of this manpage is dedicated to using the rrsync script. ## OPTIONS @@ -119,11 +122,11 @@ The `~/.ssh/authorized_keys` file might have lines in it like this: ## SEE ALSO -[**rsync**(1)](rsync.1) +[**rsync**(1)](rsync.1), [**rsyncd.conf**(5)](rsyncd.conf.5) ## VERSION -This man page is current for version @VERSION@ of rsync. +This manpage is current for version @VERSION@ of rsync. ## CREDITS diff --git a/uidlist.c b/uidlist.c index 6100b503..99a34679 100644 --- a/uidlist.c +++ b/uidlist.c @@ -3,7 +3,7 @@ * * Copyright (C) 1996 Andrew Tridgell * Copyright (C) 1996 Paul Mackerras - * Copyright (C) 2004-2020 Wayne Davison + * Copyright (C) 2004-2022 Wayne Davison * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -210,7 +210,7 @@ static int is_in_group(gid_t gid) ngroups = getgroups(ngroups, gidset); /* The default gid might not be in the list on some systems. */ for (n = 0; n < ngroups; n++) { - if (gidset[n] == our_gid) + if ((gid_t)gidset[n] == our_gid) break; } if (n == ngroups) @@ -229,7 +229,7 @@ static int is_in_group(gid_t gid) last_in = gid; for (n = 0; n < ngroups; n++) { - if (gidset[n] == gid) + if ((gid_t)gidset[n] == gid) return last_out = 1; } return last_out = 0; diff --git a/usage.c b/usage.c index cbca83e3..df033c92 100644 --- a/usage.c +++ b/usage.c @@ -195,6 +195,8 @@ void print_rsync_version(enum logcode f) print_info_flags(f); + init_checksum_choices(); + rprintf(f, "Checksum list:\n"); get_default_nno_list(&valid_checksums, tmpbuf, sizeof tmpbuf, '('); rprintf(f, " %s\n", tmpbuf); diff --git a/version.h b/version.h index f97bb16d..82aa78f2 100644 --- a/version.h +++ b/version.h @@ -1 +1,2 @@ -#define RSYNC_VERSION "3.2.4" +#define RSYNC_VERSION "3.2.5" +#define MAINTAINER_TZ_OFFSET -7.0 diff --git a/zlib/inflate.c b/zlib/inflate.c index e43abd9e..e9840b67 100644 --- a/zlib/inflate.c +++ b/zlib/inflate.c @@ -740,8 +740,9 @@ int flush; if (copy > have) copy = have; if (copy) { if (state->head != Z_NULL && - state->head->extra != Z_NULL) { - len = state->head->extra_len - state->length; + state->head->extra != Z_NULL && + (len = state->head->extra_len - state->length) < + state->head->extra_max) { zmemcpy(state->head->extra + len, next, len + copy > state->head->extra_max ? state->head->extra_max - len : copy); diff -upN a/config.h.in b/config.h.in --- a/config.h.in +++ b/config.h.in @@ -73,6 +73,9 @@ /* Define to 1 if readdir() is broken */ #undef HAVE_BROKEN_READDIR +/* Define to 1 if you have the header file. */ +#undef HAVE_BSD_STRING_H + /* Define to 1 if vsprintf has a C99-compatible return value */ #undef HAVE_C99_VSNPRINTF diff -upN a/configure.sh b/configure.sh --- a/configure.sh +++ b/configure.sh @@ -4239,7 +4239,8 @@ for ac_header in sys/fcntl.h sys/select. netdb.h malloc.h float.h limits.h iconv.h libcharset.h langinfo.h mcheck.h \ sys/acl.h acl/libacl.h attr/xattr.h sys/xattr.h sys/extattr.h dl.h \ popt.h popt/popt.h linux/falloc.h netinet/in_systm.h netgroup.h \ - zlib.h xxhash.h openssl/md4.h openssl/md5.h zstd.h lz4.h sys/file.h + zlib.h xxhash.h openssl/md4.h openssl/md5.h zstd.h lz4.h sys/file.h \ + bsd/string.h do : as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh` ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default" @@ -4326,7 +4327,7 @@ ac_config_headers="$ac_config_headers co -PACKAGE_VERSION=`sed 's/.*"\(.*\)".*/\1/' <$srcdir/version.h` +PACKAGE_VERSION=`sed -n 's/.*RSYNC_VERSION.*"\(.*\)".*/\1/p' <$srcdir/version.h` { $as_echo "$as_me:${as_lineno-$LINENO}: Configuring rsync $PACKAGE_VERSION" >&5 $as_echo "$as_me: Configuring rsync $PACKAGE_VERSION" >&6;} @@ -7995,7 +7996,8 @@ _ACEOF fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking type of array argument to getgroups" >&5 +if test "$cross_compiling" = no; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking type of array argument to getgroups" >&5 $as_echo_n "checking type of array argument to getgroups... " >&6; } if ${ac_cv_type_getgroups+:} false; then : $as_echo_n "(cached) " >&6 @@ -8062,6 +8064,11 @@ cat >>confdefs.h <<_ACEOF _ACEOF +else + +$as_echo "#define GETGROUPS_T gid_t" >>confdefs.h + +fi ac_fn_c_check_member "$LINENO" "struct stat" "st_rdev" "ac_cv_member_struct_stat_st_rdev" " #ifdef HAVE_SYS_TYPES_H #include @@ -10089,7 +10096,7 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ int main () { -signed char *s = "" +signed char *s = (signed char *)"" ; return 0; } diff -upN a/rrsync.1 b/rrsync.1 --- a/rrsync.1 +++ b/rrsync.1 @@ -1,4 +1,4 @@ -.TH "rrsync" "1" "11 Apr 2022" "rrsync from rsync 3.2.4pre4" "User Commands" +.TH "rrsync" "1" "13 Aug 2022" "rrsync from rsync 3.2.5pre2" "User Commands" .\" prefix=/usr .P .SH "NAME" @@ -14,14 +14,18 @@ rrsync [-ro|-rw] [-munge] [-no-del] [-no The single non-option argument specifies the restricted \fIDIR\fP to use. It can be relative to the user's home directory or an absolute path. .P -The online version of this man page (that includes cross-linking of topics) +The online version of this manpage (that includes cross-linking of topics) is available at https://download.samba.org/pub/rsync/rrsync.1. .P .SH "DESCRIPTION" .P A user's ssh login can be restricted to only allow the running of an rsync -transfer in one of two easy ways: forcing the running of the rrsync script -or forcing the running of an rsync daemon-over-ssh command. +transfer in one of two easy ways: +.P +.IP o +forcing the running of the rrsync script +.IP o +forcing the running of an rsync daemon-over-ssh command. .P To use the rrsync script, edit the user's \fB~/.ssh/authorized_keys\fP file and add a prefix like one of the following (followed by a space) in front of each @@ -54,13 +58,14 @@ command="rsync --server --daemon --confi Then, ensure that the rsyncd.conf file is created with one or more module names with the appropriate path and option restrictions. If rsync's \fB\-\-config\fP option is omitted, it defaults to \fB~/rsyncd.conf\fP. -See the \fBrsyncd.conf\fP man page for details of how to configure an rsync daemon. +See the \fBrsyncd.conf\fP(5) manpage for details of how to +configure an rsync daemon. .P When using rrsync, there can be just one restricted dir per authorized key. A daemon setup, on the other hand, allows multiple module names inside the config file, each one with its own path setting. .P -The remainder of this man page is dedicated to using the rrsync script. +The remainder of this manpage is dedicated to using the rrsync script. .P .SH "OPTIONS" .P @@ -117,11 +122,11 @@ command="rrsync -ro results" ssh-rsa AAA .P .SH "SEE ALSO" .P -\fBrsync\fP(1) +\fBrsync\fP(1), \fBrsyncd.conf\fP(5) .P .SH "VERSION" .P -This man page is current for version 3.2.4pre4 of rsync. +This manpage is current for version 3.2.5pre2 of rsync. .P .SH "CREDITS" .P diff -upN a/rrsync.1.html b/rrsync.1.html --- a/rrsync.1.html +++ b/rrsync.1.html @@ -38,12 +38,15 @@ dd p:first-of-type {

The single non-option argument specifies the restricted DIR to use. It can be relative to the user's home directory or an absolute path.

-

The online version of this man page (that includes cross-linking of topics) +

The online version of this manpage (that includes cross-linking of topics) is available at https://download.samba.org/pub/rsync/rrsync.1.

DESCRIPTION

A user's ssh login can be restricted to only allow the running of an rsync -transfer in one of two easy ways: forcing the running of the rrsync script -or forcing the running of an rsync daemon-over-ssh command.

+transfer in one of two easy ways:

+
    +
  • forcing the running of the rrsync script
    • +
  • forcing the running of an rsync daemon-over-ssh command.
    • +

To use the rrsync script, edit the user's ~/.ssh/authorized_keys file and add a prefix like one of the following (followed by a space) in front of each ssh-key line that should be restricted:

@@ -68,11 +71,12 @@ command="rsync --server --daemon --

Then, ensure that the rsyncd.conf file is created with one or more module names with the appropriate path and option restrictions. If rsync's --config option is omitted, it defaults to ~/rsyncd.conf. -See the rsyncd.conf man page for details of how to configure an rsync daemon.

+See the rsyncd.conf(5) manpage for details of how to +configure an rsync daemon.

When using rrsync, there can be just one restricted dir per authorized key. A daemon setup, on the other hand, allows multiple module names inside the config file, each one with its own path setting.

-

The remainder of this man page is dedicated to using the rrsync script.

+

The remainder of this manpage is dedicated to using the rrsync script.

OPTIONS

@@ -127,9 +131,9 @@ command="rrsync -ro results" s

FILES

~/.ssh/authorized_keys

SEE ALSO

-

rsync(1)

+

rsync(1), rsyncd.conf(5)

VERSION

-

This man page is current for version 3.2.4pre4 of rsync.

+

This manpage is current for version 3.2.5pre2 of rsync.

CREDITS

rsync is distributed under the GNU General Public License. See the file COPYING for details.

@@ -138,5 +142,5 @@ project is AUTHOR

The original rrsync perl script was written by Joe Smith. Many people have later contributed to it. The python version was created by Wayne Davison.

-

11 Apr 2022

+

13 Aug 2022

diff -upN a/rsync-ssl.1 b/rsync-ssl.1 --- a/rsync-ssl.1 +++ b/rsync-ssl.1 @@ -1,4 +1,4 @@ -.TH "rsync-ssl" "1" "15 Apr 2022" "rsync-ssl from rsync 3.2.4" "User Commands" +.TH "rsync-ssl" "1" "14 Aug 2022" "rsync-ssl from rsync 3.2.5" "User Commands" .\" prefix=/usr .P .SH "NAME" @@ -98,6 +98,11 @@ rsync-ssl -aiv rsync://example.com:9874/ .fi .RE .P +.SH "THE SERVER SIDE" +.P +For help setting up an SSL/TLS supporting rsync, see the instructions in +rsyncd.conf. +.P .SH "SEE ALSO" .P \fBrsync\fP(1), \fBrsyncd.conf\fP(5) @@ -121,7 +126,7 @@ Please report bugs! See the web site at .P .SH "VERSION" .P -This manpage is current for version 3.2.4 of rsync. +This manpage is current for version 3.2.5 of rsync. .P .SH "CREDITS" .P diff -upN a/rsync-ssl.1.html b/rsync-ssl.1.html --- a/rsync-ssl.1.html +++ b/rsync-ssl.1.html @@ -122,6 +122,9 @@ then for "stunnel".

rsync-ssl -aiv rsync://example.com:9874/mod/ dest
+

THE SERVER SIDE

+

For help setting up an SSL/TLS supporting rsync, see the instructions in +rsyncd.conf.

SEE ALSO

rsync(1), rsyncd.conf(5)

CAVEATS

@@ -137,7 +140,7 @@ exported RSYNC_SSL_TYPE environment vari

BUGS

Please report bugs! See the web site at https://rsync.samba.org/.

VERSION

-

This manpage is current for version 3.2.4 of rsync.

+

This manpage is current for version 3.2.5 of rsync.

CREDITS

Rsync is distributed under the GNU General Public License. See the file COPYING for details.

@@ -147,5 +150,5 @@ FAQ-O-Matic which may cover questions un

This manpage was written by Wayne Davison.

Mailing lists for support and development are available at https://lists.samba.org/.

-

15 Apr 2022

+

14 Aug 2022

diff -upN a/rsync.1 b/rsync.1 --- a/rsync.1 +++ b/rsync.1 @@ -1,4 +1,4 @@ -.TH "rsync" "1" "15 Apr 2022" "rsync 3.2.4" "User Commands" +.TH "rsync" "1" "14 Aug 2022" "rsync 3.2.5" "User Commands" .\" prefix=/usr .P .SH "NAME" @@ -188,6 +188,55 @@ rsync somehost.mydomain.com:: .P See the following section for more details. .P +.SH "SORTED TRANSFER ORDER" +.P +Rsync always sorts the specified filenames into its internal transfer list. +This handles the merging together of the contents of identically named +directories, makes it easy to remove duplicate filenames. It can, however, +confuse someone when the files are transferred in a different order than what +was given on the command-line. +.P +If you need a particular file to be transferred prior to another, either +separate the files into different rsync calls, or consider using +\fB\-\-delay-updates\fP (which doesn't affect the sorted transfer order, but +does make the final file-updating phase happen much more rapidly). +.P +.SH "MULTI-HOST SECURITY" +.P +Rsync takes steps to ensure that the file requests that are shared in a +transfer are protected against various security issues. Most of the potential +problems arise on the receiving side where rsync takes steps to ensure that the +list of files being transferred remains within the bounds of what was +requested. +.P +Toward this end, rsync 3.1.2 and later have aborted when a file list contains +an absolute or relative path that tries to escape out of the top of the +transfer. Also, beginning with version 3.2.5, rsync does two more safety +checks of the file list to (1) ensure that no extra source arguments were added +into the transfer other than those that the client requested and (2) ensure +that the file list obeys the exclude rules that were sent to the sender. +.P +For those that don't yet have a 3.2.5 client rsync (or those that want to be +extra careful), it is safest to do a copy into a dedicated destination +directory for the remote files when you don't trust the remote host. For +example, instead of doing an rsync copy into your home directory: +.RS 4 +.P +.nf +rsync -aiv host1:dir1 ~ +.fi +.RE +.P +Dedicate a "host1-files" dir to the remote content: +.RS 4 +.P +.nf +rsync -aiv host1:dir1 ~/host1-files +.fi +.RE +.P +See the \fB\-\-trust-sender\fP option for additional details. +.P .SH "ADVANCED USAGE" .P The syntax for requesting multiple files from a remote host is done by @@ -202,8 +251,15 @@ rsync -aiv host::modname/first ::modname .fi .RE .P -In a modern rsync, you only need to quote or backslash-escape things like -spaces from the local shell but not also from the remote shell: +Really old versions of rsync (2.6.9 and before) only allowed specifying one +remote-source arg, so some people have instead relied on the remote-shell +performing space splitting to break up an arg into multiple paths. Such +unintuitive behavior is no longer supported by default (though you can request +it, as described below). +.P +Starting in 3.2.4, filenames are passed to a remote shell in such a way as to +preserve the characters you give it. Thus, if you ask for a file with spaces +in the name, that's what the remote rsync looks for: .RS 4 .P .nf @@ -211,19 +267,12 @@ rsync -aiv host:'a simple file.pdf' /des .fi .RE .P -Really old versions of rsync only allowed specifying one remote-source arg, so -it required the remote side to split the args at a space. You can still get -this old-style arg splitting by using the \fB\-\-old-args\fP option: -.RS 4 -.P -.nf -rsync -ai --old-args host:'dir1/file1 dir2/file2' /dest -rsync -ai --old-args host::'modname/dir1/file1 modname/dir2/file2' /dest -.fi -.RE -.P -See that option's section for an environment variable that -can be exported to help old scripts. +If you use scripts that have been written to manually apply extra quoting to +the remote rsync args (or to require remote arg splitting), you can ask rsync +to let your script handle the extra escaping. This is done by either adding +the \fB\-\-old-args\fP option to the rsync runs in the script (which requires +a new rsync) or exporting RSYNC_OLD_ARGS=1 and RSYNC_PROTECT_ARGS=0 +(which works with old or new rsync versions). .P .SH "CONNECTING TO AN RSYNC DAEMON" .P @@ -338,6 +387,10 @@ rsync -av -e "ssh -l ssh-user" rsync-use The "ssh-user" will be used at the ssh level; the "rsync-user" will be used to log-in to the "module". .P +In this setup, the daemon is started by the ssh command that is accessing the +system (which can be forced via the \fB~/.ssh/authorized_keys\fP file, if desired). +However, when accessing a daemon directly, it needs to be started beforehand. +.P .SH "STARTING AN RSYNC DAEMON TO ACCEPT CONNECTIONS" .P In order to connect to an rsync daemon, the remote system needs to have a @@ -351,61 +404,27 @@ the daemon (including stand-alone and in If you're using one of the remote-shell transports for the transfer, there is no need to manually start an rsync daemon. .P -.SH "SORTED TRANSFER ORDER" -.P -Rsync always sorts the specified filenames into its internal transfer list. -This handles the merging together of the contents of identically named -directories, makes it easy to remove duplicate filenames, and may confuse -someone when the files are transferred in a different order than what was given -on the command-line. -.P -If you need a particular file to be transferred prior to another, either -separate the files into different rsync calls, or consider using -\fB\-\-delay-updates\fP (which doesn't affect the sorted transfer order, but -does make the final file-updating phase happen much more rapidly). -.P .SH "EXAMPLES" .P -Here are some examples of how I use rsync. -.P -To backup my wife's home directory, which consists of large MS Word files and -mail folders, I use a cron job that runs -.RS 4 -.P -.nf -rsync -Cavz . arvidsjaur:backup -.fi -.RE -.P -each night over a PPP connection to a duplicate directory on my machine -"arvidsjaur". +Here are some examples of how rsync can be used. .P -To synchronize my samba source trees I use the following Makefile targets: +To backup a home directory, which consists of large MS Word files and mail +folders, a per-user cron job can be used that runs this each day: .RS 4 .P .nf -get: - rsync -avuzb --exclude '*~' samba:samba/ . -put: - rsync -Cavuzb . samba:samba/ -sync: get put +rsync -aiz . bkhost:backup/joe/ .fi .RE .P -This allows me to sync with a CVS directory at the other end of the connection. -I then do CVS operations on the remote machine, which saves a lot of time as -the remote CVS protocol isn't very efficient. -.P -I mirror a directory between my "old" and "new" ftp sites with the command: +To move some files from a remote host to the local host, you could run: .RS 4 .P .nf -rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge" +rsync -aiv --remove-source-files rhost:/tmp/{file1,file2}.c ~/src/ .fi .RE .P -This is launched from cron every few hours. -.P .SH "OPTION SUMMARY" .P Here is a short summary of the options available in rsync. Each option also @@ -523,6 +542,7 @@ has its own detailed description later i --from0, -0 all *-from/filter files are delimited by 0s --old-args disable the modern arg-protection idiom --protect-args, -s no space-splitting; wildcard chars only +--trust-sender trust the remote sender's file list --copy-as=USER[:GROUP] specify user & optional group for the copy --address=ADDRESS bind address for outgoing socket to daemon --port=PORT specify double-colon alternate port number @@ -583,14 +603,19 @@ accepted: Rsync accepts both long (double-dash + word) and short (single-dash + letter) options. The full list of the available options are described below. If an option can be specified in more than one way, the choices are comma-separated. -Some options only have a long variant, not a short. If the option takes a -parameter, the parameter is only listed after the long variant, even though it -must also be specified for the short. When specifying a parameter, you can -either use the form \fB\-\-option=param\fP or replace the '=' with whitespace. The -parameter may need to be quoted in some manner for it to survive the shell's -command-line parsing. Keep in mind that a leading tilde (\fB~\fP) in a filename is -substituted by your shell, so \fB\-\-option=~/foo\fP will not change the tilde into -your home directory (remove the '=' for that). +Some options only have a long variant, not a short. +.P +If the option takes a parameter, the parameter is only listed after the long +variant, even though it must also be specified for the short. When specifying +a parameter, you can either use the form \fB\-\-option=param\fP, \fB\-\-option\ param\fP, +\fB\-o=param\fP, \fB\-o\ param\fP, or \fB\-oparam\fP (the latter choices assume that your +option has a short variant). +.P +The parameter may need to be quoted in some manner for it to survive the +shell's command-line parsing. Also keep in mind that a leading tilde (\fB~\fP) in +a pathname is substituted by your shell, so make sure that you separate the +option name from the pathname using a space if you want the local shell to +expand it. .P .IP "\fB\-\-help\fP" Print a short help page describing the options available in rsync and exit. @@ -861,7 +886,7 @@ disable the incremental recursion mode. .IP In order to make \fB\-\-delete\fP compatible with incremental recursion, rsync 3.0.0 made \fB\-\-delete-during\fP the default delete mode (which -was first first added in 2.6.4). +was first added in 2.6.4). .IP One side-effect of incremental recursion is that any missing sub-directories inside a recursively-scanned directory are (by default) @@ -1026,9 +1051,8 @@ matter what date is on the objects. In directory where the destination has a file, the transfer would occur regardless of the timestamps. .IP -This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred. +This option is a TRANSFER RULE, so don't expect any +exclude side effects. .IP A caution for those that choose to combine \fB\-\-inplace\fP with \fB\-\-update\fP: an interrupted transfer will leave behind a partial file on the @@ -1242,7 +1266,7 @@ to the server when it has/needs the mung transfer, the client is the sender, so specifying the option directly unmunges symlinks while specifying it as a remote option munges symlinks. .IP -This option has no affect when sent to a daemon via \fB\-\-remote-option\fP +This option has no effect when sent to a daemon via \fB\-\-remote-option\fP because the daemon configures whether it wants munged symlinks via its "\fBmunge\ symlinks\fP" parameter. .IP @@ -1567,7 +1591,7 @@ The \fB\-D\fP option is equivalent to "\ .IP "\fB\-\-copy-devices\fP" This tells rsync to treat a device on the sending side as a regular file, allowing it to be copied to a normal destination file (or another device -if \fB\-\-write-devices\fP was also specifed). +if \fB\-\-write-devices\fP was also specified). .IP This option is refused by default by an rsync daemon. .IP "\fB\-\-write-devices\fP" @@ -1801,17 +1825,15 @@ exist yet on the destination. If this o \fB\-\-ignore-existing\fP option, no files will be updated (which can be useful if all you want to do is delete extraneous files). .IP -This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred. +This option is a TRANSFER RULE, so don't expect any +exclude side effects. .IP "\fB\-\-ignore-existing\fP" This tells rsync to skip updating files that already exist on the destination (this does \fInot\fP ignore existing directories, or nothing would get done). See also \fB\-\-ignore-non-existing\fP. .IP -This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred. +This option is a TRANSFER RULE, so don't expect any +exclude side effects. .IP This option can be useful for those doing backups using the \fB\-\-link-dest\fP option when they need to continue a backup run that @@ -1926,13 +1948,24 @@ more details on file-deletion. See also the \fB\-\-delete-delay\fP option that might be a faster choice for those that just want the deletions to occur at the end of the transfer. .IP "\fB\-\-delete-excluded\fP" -In addition to deleting the files on the receiving side that are not on the -sending side, this tells rsync to also delete any files on the receiving -side that are excluded (see \fB\-\-exclude\fP). See the FILTER -RULES section for a way to make individual exclusions behave this way -on the receiver, and for a way to protect files from \fB\-\-delete-excluded\fP. -See \fB\-\-delete\fP (which is implied) for more details on -file-deletion. +This option turns any unqualified exclude/include rules into server-side +rules that do not affect the receiver's deletions. +.IP +By default, an exclude or include has both a server-side effect (to "hide" +and "show" files when building the server's file list) and a receiver-side +effect (to "protect" and "risk" files when deletions are occuring). Any +rule that has no modifier to specify what sides it is executed on will be +instead treated as if it were a server-side rule only, avoiding any +"protect" effects of the rules. +.IP +A rule can still apply to both sides even with this option specified if the +rule is given both the sender & receiver modifer letters (e.g., \fB\-f'\-sr\ foo'\fP). Receiver-side protect/risk rules can also be explicitly specified +to limit the deletions. This saves you from having to edit a bunch of +\fB\-f'\-\ foo'\fP rules into \fB\-f'\-s\ foo'\fP (aka \fB\-f'H\ foo'\fP) rules (not to mention +the corresponding includes). +.IP +See the FILTER RULES section for more information. See +\fB\-\-delete\fP (which is implied) for more details on deletion. .IP "\fB\-\-ignore-missing-args\fP" When rsync is first processing the explicitly requested source files (e.g. command-line arguments or \fB\-\-files-from\fP entries), it is normally @@ -1981,9 +2014,8 @@ specified SIZE. A numeric value can be the numeric units or left unqualified to specify bytes. Feel free to use a fractional value along with the units, such as \fB\-\-max-size=1.5m\fP. .IP -This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred. +This option is a TRANSFER RULE, so don't expect any +exclude side effects. .IP The first letter of a units string can be \fBB\fP (bytes), \fBK\fP (kilo), \fBM\fP (mega), \fBG\fP (giga), \fBT\fP (tera), or \fBP\fP (peta). If the string is a single @@ -2244,8 +2276,8 @@ See the FILTER RULES section for detaile options work. .IP "\fB\-\-exclude=PATTERN\fP" This option is a simplified form of the \fB\-\-filter\fP option that -defaults to an exclude rule and does not allow the full rule-parsing syntax -of normal filter rules. +specifies an exclude rule and does not allow the full rule-parsing syntax +of normal filter rules. This is equivalent to specifying \fB\-f'\-\ PATTERN'\fP. .IP See the FILTER RULES section for detailed information on this option. .IP "\fB\-\-exclude-from=FILE\fP" @@ -2254,11 +2286,18 @@ a FILE that contains exclude patterns (o file are ignored, as are whole-line comments that start with '\fB;\fP' or '\fB#\fP' (filename rules that contain those characters are unaffected). .IP +If a line begins with "\fB\-\ \fP" (dash, space) or "\fB+\ \fP" (plus, space), then +the type of rule is being explicitly specified as an exclude or an include +(respectively). Any rules without such a prefix are taken to be an exclude. +.IP +If a line consists of just "\fB!\fP", then the current filter rules are cleared +before adding any further rules. +.IP If \fIFILE\fP is '\fB\-\fP', the list will be read from standard input. .IP "\fB\-\-include=PATTERN\fP" This option is a simplified form of the \fB\-\-filter\fP option that -defaults to an include rule and does not allow the full rule-parsing syntax -of normal filter rules. +specifies an include rule and does not allow the full rule-parsing syntax +of normal filter rules. This is equivalent to specifying \fB\-f'+\ PATTERN'\fP. .IP See the FILTER RULES section for detailed information on this option. .IP "\fB\-\-include-from=FILE\fP" @@ -2267,6 +2306,13 @@ a FILE that contains include patterns (o file are ignored, as are whole-line comments that start with '\fB;\fP' or '\fB#\fP' (filename rules that contain those characters are unaffected). .IP +If a line begins with "\fB\-\ \fP" (dash, space) or "\fB+\ \fP" (plus, space), then +the type of rule is being explicitly specified as an exclude or an include +(respectively). Any rules without such a prefix are taken to be an include. +.IP +If a line consists of just "\fB!\fP", then the current filter rules are cleared +before adding any further rules. +.IP If \fIFILE\fP is '\fB\-\fP', the list will be read from standard input. .IP "\fB\-\-files-from=FILE\fP" Using this option allows you to specify the exact list of files to transfer @@ -2369,6 +2415,12 @@ repeated-option setting. If it is "0", behavior. The environment is always overridden by manually specified positive or negative options (the negative is \fB\-\-no-old-args\fP). .IP +Note that this option also disables the extra safety check added in 3.2.5 +that ensures that a remote sender isn't including extra top-level items in +the file-list that you didn't request. This side-effect is necessary +because we can't know for sure what names to expect when the remote shell +is interpreting the args. +.IP This option conflicts with the \fB\-\-protect-args\fP option. .IP "\fB\-\-protect-args\fP, \fB\-s\fP" This option sends all filenames and most options to the remote rsync @@ -2399,6 +2451,50 @@ This option conflicts with the \fB\-\-ol .IP Note that this option is incompatible with the use of the restricted rsync script (\fBrrsync\fP) since it hides options from the script's inspection. +.IP "\fB\-\-trust-sender\fP" +This option disables two extra validation checks that a local client +performs on the file list generated by a remote sender. This option should +only be used if you trust the sender to not put something malicious in the +file list (something that could possibly be done via a modified rsync, a +modified shell, or some other similar manipulation). +.IP +Normally, the rsync client (as of version 3.2.5) runs two extra validation +checks when pulling files from a remote rsync: +.IP +.RS +.IP o +It verifies that additional arg items didn't get added at the top of the +transfer. +.IP o +It verifies that none of the items in the file list are names that should +have been excluded (if filter rules were specified). +.RE +.IP +Note that various options can turn off one or both of these checks if the +option interferes with the validation. For instance: +.IP +.RS +.IP o +Using a per-directory filter file reads filter rules that only the server +knows about, so the filter checking is disabled. +.IP o +Using the \fB\-\-old-args\fP option allows the sender to manipulate the +requested args, so the arg checking is disabled. +.IP o +Reading the files-from list from the server side means that the client +doesn't know the arg list, so the arg checking is disabled. +.IP o +Using \fB\-\-read-batch\fP disables both checks since the batch file's +contents will have been verified when it was created. +.RE +.IP +This option may help an under-powered client server if the extra pattern +matching is slowing things down on a huge transfer. It can also be used to +work around a currently-unknown bug in the verification logic for a transfer +from a trusted sender. +.IP +When using this option it is a good idea to specify a dedicated destination +directory, as discussed in the MULTI-HOST SECURITY section. .IP "\fB\-\-copy-as=USER[:GROUP]\fP" This option instructs rsync to use the USER and (if specified after a colon) the GROUP for the copy operations. This only works if the user that @@ -3082,7 +3178,7 @@ as a client option, transfer logging wil of "%i %n%L". See the \fB\-\-log-file-format\fP option if you wish to override this. .IP -Here's a example command that requests the remote side to log what is +Here's an example command that requests the remote side to log what is happening: .RS 4 .IP @@ -3340,10 +3436,8 @@ children. This is useful for avoiding t directories when the sending rsync is recursively scanning a hierarchy of files using include/exclude/filter rules. .IP -Note that the use of transfer rules, such as the \fB\-\-min-size\fP -option, does not affect what goes into the file list, and thus does not -leave directories empty, even if none of the files in a directory match the -transfer rule. +This option can still leave empty directories on the receiving side if you +make use of TRANSFER_RULES. .IP Because the file-list is actually being pruned, this option also affects what directories get deleted when a delete is active. However, keep in @@ -3489,8 +3583,8 @@ include the destination. .RE .IP CAUTION: keep in mind that a source arg with a wild-card is expanded by the -shell into multiple args, so it is never safe to try to list such an arg -without using this option. For example: +shell into multiple args, so it is never safe to try to specify a single +wild-card arg to try to infer this option. A safe example is: .RS 4 .IP .nf @@ -3777,27 +3871,173 @@ options available for starting an rsync .P .SH "FILTER RULES" .P -The filter rules allow for flexible selection of which files to transfer -(include) and which files to skip (exclude). The rules either directly specify -include/exclude patterns or they specify a way to acquire more include/exclude -patterns (e.g. to read them from a file). -.P -As the list of files/directories to transfer is built, rsync checks each name -to be transferred against the list of include/exclude patterns in turn, and the -first matching pattern is acted on: if it is an exclude pattern, then that file -is skipped; if it is an include pattern then that filename is not skipped; if -no matching pattern is found, then the filename is not skipped. -.P -Aside: because the interactions of filter rules can be complex, it is useful to -use the \fB\-\-debug=FILTER\fP option if things aren't working the way you expect. -The level-1 output (the default if no level number is specified) mentions the -filter rule that is first matched by each file in the transfer. It also warns -if a filter rule has trailing whitespace. The level-2 output mentions a lot -more filter events, including the definition of each rule and the handling of -per-directory filter files. +The filter rules allow for custom control of several aspects of how files are +handled: +.P +.IP o +Control which files the sending side puts into the file list that describes +the transfer hierarchy +.IP o +Control which files the receiving side protects from deletion when the file +is not in the sender's file list +.IP o +Control which extended attribute names are skipped when copying xattrs +.P +The rules are either directly specified via option arguments or they can be +read in from one or more files. The filter-rule files can even be a part of +the hierarchy of files being copied, affecting different parts of the tree in +different ways. +.P +.SS "SIMPLE INCLUDE/EXCLUDE RULES" +.P +We will first cover the basics of how include & exclude rules affect what files +are transferred, ignoring any deletion side-effects. Filter rules mainly +affect the contents of directories that rsync is "recursing" into, but they can +also affect a top-level item in the transfer that was specified as a argument. +.P +The default for any unmatched file/dir is for it to be included in the +transfer, which puts the file/dir into the sender's file list. The use of an +exclude rule causes one or more matching files/dirs to be left out of the +sender's file list. An include rule can be used to limit the effect of an +exclude rule that is matching too many files. +.P +The order of the rules is important because the first rule that matches is the +one that takes effect. Thus, if an early rule excludes a file, no include rule +that comes after it can have any effect. This means that you must place any +include overrides somewhere prior to the exclude that it is intended to limit. +.P +When a directory is excluded, all its contents and sub-contents are also +excluded. The sender doesn't scan through any of it at all, which can save a +lot of time when skipping large unneeded sub-trees. +.P +It is also important to understand that the include/exclude rules are applied +to every file and directory that the sender is recursing into. Thus, if you +want a particular deep file to be included, you have to make sure that none of +the directories that must be traversed on the way down to that file are +excluded or else the file will never be discovered to be included. As an +example, if the directory "\fBa/path\fP" was given as a transfer argument and you +want to ensure that the file "\fBa/path/down/deep/wanted.txt\fP" is a part of the +transfer, then the sender must not exclude the directories "\fBa/path\fP", +"\fBa/path/down\fP", or "\fBa/path/down/deep\fP" as it makes it way scanning through +the file tree. +.P +When you are working on the rules, it can be helpful to ask rsync to tell you +what is being excluded/included and why. Specifying \fB\-\-debug=FILTER\fP or (when +pulling files) \fB\-M\-\-debug=FILTER\fP turns on level 1 of the FILTER debug +information that will output a message any time that a file or directory is +included or excluded and which rule it matched. Beginning in 3.2.4 it will +also warn if a filter rule has trailing whitespace, since an exclude of "foo\ " +(with a trailing space) will not exclude a file named "foo". +.P +Exclude and include rules can specify wildcard PATTERN MATCHING RULES +(similar to shell wildcards) that allow you to match things like a file suffix +or a portion of a filename. +.P +A rule can be limited to only affecting a directory by putting a trailing slash +onto the filename. +.P +.SS "SIMPLE INCLUDE/EXCLUDE EXAMPLE" +.P +With the following file tree created on the sending side: +.RS 4 +.P +.nf +mkdir x/ +touch x/file.txt +mkdir x/y/ +touch x/y/file.txt +touch x/y/zzz.txt +mkdir x/z/ +touch x/z/file.txt +.fi +.RE +.P +Then the following rsync command will transfer the file "\fBx/y/file.txt\fP" and +the directories needed to hold it, resulting in the path "\fB/tmp/x/y/file.txt\fP" +existing on the remote host: +.RS 4 +.P +.nf +rsync -ai -f'+ x/' -f'+ x/y/' -f'+ x/y/file.txt' -f'- *' x host:/tmp/ +.fi +.RE +.P +Aside: this copy could also have been accomplished using the \fB\-R\fP +option (though the 2 commands behave differently if deletions are enabled): +.RS 4 +.P +.nf +rsync -aiR x/y/file.txt host:/tmp/ +.fi +.RE +.P +The following command does not need an include of the "x" directory because it +is not a part of the transfer (note the traililng slash). Running this command +would copy just "\fB/tmp/x/file.txt\fP" because the "y" and "z" dirs get excluded: +.RS 4 +.P +.nf +rsync -ai -f'+ file.txt' -f'- *' x/ host:/tmp/x/ +.fi +.RE +.P +This command would omit the zzz.txt file while copying "x" and everything else +it contains: +.RS 4 +.P +.nf +rsync -ai -f'- zzz.txt' x host:/tmp/ +.fi +.RE +.P +.SS "FILTER RULES WHEN DELETING" +.P +By default the include & exclude filter rules affect both the sender +(as it creates its file list) +and the receiver (as it creates its file lists for calculating deletions). If +no delete option is in effect, the receiver skips creating the delete-related +file lists. This two-sided default can be manually overridden so that you are +only specifying sender rules or receiver rules, as described in the FILTER +RULES IN DEPTH section. +.P +When deleting, an exclude protects a file from being removed on the receiving +side while an include overrides that protection (putting the file at risk of +deletion). The default is for a file to be at risk\ \-\- its safety depends on it +matching a corresponding file from the sender. +.P +An example of the two-sided exclude effect can be illustrated by the copying of +a C development directory between 2 systems. When doing a touch-up copy, you +might want to skip copying the built executable and the \fB.o\fP files (sender +hide) so that the receiving side can build their own and not lose any object +files that are already correct (receiver protect). For instance: +.RS 4 +.P +.nf +rsync -ai --del -f'- *.o' -f'- cmd' src host:/dest/ +.fi +.RE +.P +Note that using \fB\-f'\-p\ *.o'\fP is even better than \fB\-f'\-\ *.o'\fP if there is a +chance that the directory structure may have changed. The "p" modifier is +discussed in FILTER RULE MODIFIERS. +.P +One final note, if your shell doesn't mind unexpanded wildcards, you could +simplify the typing of the filter options by using an underscore in place of +the space and leaving off the quotes. For instance, \fB\-f\ \-_*.o\ \-f\ \-_cmd\fP (and +similar) could be used instead of the filter options above. +.P +.SS "FILTER RULES IN DEPTH" +.P +Rsync supports old-style include/exclude rules and new-style filter rules. The +older rules are specified using \fB\-\-include\fP and \fB\-\-exclude\fP as +well as the \fB\-\-include-from\fP and \fB\-\-exclude-from\fP. These are +limited in behavior but they don't require a "\-" or "+" prefix. An old-style +exclude rule is turned into a "\fB\-\ name\fP" filter rule (with no modifiers) and an +old-style include rule is turned into a "\fB+\ name\fP" filter rule (with no +modifiers). .P -Rsync builds an ordered list of filter rules as specified on the command-line. -Filter rules have the following syntax: +Rsync builds an ordered list of filter rules as specified on the command-line +and/or read-in from files. New style filter rules have the following syntax: .RS 4 .P .nf @@ -3809,41 +4049,43 @@ RULE,MODIFIERS [PATTERN_OR_FILENAME] You have your choice of using either short or long RULE names, as described below. If you use a short-named rule, the ',' separating the RULE from the MODIFIERS is optional. The PATTERN or FILENAME that follows (when present) -must come after either a single space or an underscore (_). Here are the -available rule prefixes: +must come after either a single space or an underscore (_). Any additional +spaces and/or underscores are considered to be a part of the pattern name. +Here are the available rule prefixes: .P .IP "\fBexclude,\ '\-'\fP" -specifies an exclude pattern. +specifies an exclude pattern that (by default) is both a +\fBhide\fP and a \fBprotect\fP. .IP "\fBinclude,\ '+'\fP" -specifies an include pattern. +specifies an include pattern that (by default) is both a +\fBshow\fP and a \fBrisk\fP. .IP "\fBmerge,\ '.'\fP" -specifies a merge-file to read for more rules. +specifies a merge-file on the client side to read for more +rules. .IP "\fBdir-merge,\ ':'\fP" -specifies a per-directory merge-file. +specifies a per-directory merge-file. Using this kind of +filter rule requires that you trust the sending side's filter checking, so +it has the side-effect mentioned under the \fB\-\-trust-sender\fP option. .IP "\fBhide,\ 'H'\fP" specifies a pattern for hiding files from the transfer. +Equivalent to a sender-only exclude, so \fB\-f'H\ foo'\fP could also be specified +as \fB\-f'\-s\ foo'\fP. .IP "\fBshow,\ 'S'\fP" -files that match the pattern are not hidden. +files that match the pattern are not hidden. Equivalent to a +sender-only include, so \fB\-f'S\ foo'\fP could also be specified as \fB\-f'+s\ foo'\fP. .IP "\fBprotect,\ 'P'\fP" specifies a pattern for protecting files from deletion. +Equivalent to a receiver-only exclude, so \fB\-f'P\ foo'\fP could also be +specified as \fB\-f'\-r\ foo'\fP. .IP "\fBrisk,\ 'R'\fP" -files that match the pattern are not protected. +files that match the pattern are not protected. Equivalent to a +receiver-only include, so \fB\-f'R\ foo'\fP could also be specified as \fB\-f'+r\ foo'\fP. .IP "\fBclear,\ '!'\fP" clears the current include/exclude list (takes no arg) .P -When rules are being read from a file, empty lines are ignored, as are -whole-line comments that start with a '\fB#\fP' (filename rules that contain a hash -are unaffected). -.P -Note that the \fB\-\-include\fP & \fB\-\-exclude\fP command-line options do -not allow the full range of rule parsing as described above\ \-\- they only allow -the specification of include / exclude patterns plus a "\fB!\fP" token to clear the -list (and the normal comment parsing when rules are read from a file). If a -pattern does not begin with "\fB\-\ \fP" (dash, space) or "\fB+\ \fP" (plus, space), then -the rule will be interpreted as if "\fB+\ \fP" (for an include option) or "\fB\-\ \fP" -(for an exclude option) were prefixed to the string. A \fB\-\-filter\fP -option, on the other hand, must always contain either a short or long rule name -at the start of the rule. +When rules are being read from a file (using merge or dir-merge), empty lines +are ignored, as are whole-line comments that start with a '\fB#\fP' (filename rules +that contain a hash character are unaffected). .P Note also that the \fB\-\-filter\fP, \fB\-\-include\fP, and \fB\-\-exclude\fP options take one rule/pattern each. To add multiple ones, @@ -3851,150 +4093,110 @@ you can repeat the options on the comman the \fB\-\-filter\fP option, or the \fB\-\-include-from\fP / \fB\-\-exclude-from\fP options. .P -.SH "INCLUDE/EXCLUDE PATTERN RULES" +.SS "PATTERN MATCHING RULES" .P -You can include and exclude files by specifying patterns using the "+", "\-", -etc. filter rules (as introduced in the FILTER RULES section above). The -include/exclude rules each specify a pattern that is matched against the names -of the files that are going to be transferred. These patterns can take several -forms: -.P -.IP o -if the pattern starts with a \fB/\fP then it is anchored to a particular spot in -the hierarchy of files, otherwise it is matched against the end of the -pathname. This is similar to a leading \fB^\fP in regular expressions. Thus -\fB/foo\fP would match a name of "foo" at either the "root of the transfer" (for -a global rule) or in the merge-file's directory (for a per-directory rule). -An unqualified \fBfoo\fP would match a name of "foo" anywhere in the tree because -the algorithm is applied recursively from the top down; it behaves as if each -path component gets a turn at being the end of the filename. Even the -unanchored "sub/foo" would match at any point in the hierarchy where a "foo" -was found within a directory named "sub". See the section on ANCHORING -INCLUDE/EXCLUDE PATTERNS for a full discussion of how to specify a pattern -that matches at the root of the transfer. -.IP o -if the pattern ends with a \fB/\fP then it will only match a directory, not a -regular file, symlink, or device. -.IP o -rsync chooses between doing a simple string match and wildcard matching by -checking if the pattern contains one of these three wildcard characters: -\&'\fB*\fP', '\fB?\fP', and '\fB[\fP' . -.IP o -a '\fB*\fP' matches any path component, but it stops at slashes. -.IP o -use '\fB**\fP' to match anything, including slashes. -.IP o -a '\fB?\fP' matches any character except a slash (\fB/\fP). -.IP o -a '\fB[\fP' introduces a character class, such as \fB[a-z]\fP or \fB[[:alpha:]]\fP. -.IP o -in a wildcard pattern, a backslash can be used to escape a wildcard -character, but it is matched literally when no wildcards are present. This -means that there is an extra level of backslash removal when a pattern -contains wildcard characters compared to a pattern that has none. e.g. if -you add a wildcard to "\fBfoo\\bar\fP" (which matches the backslash) you would -need to use "\fBfoo\\\\bar*\fP" to avoid the "\fB\\b\fP" becoming just "b". -.IP o -if the pattern contains a \fB/\fP (not counting a trailing /) or a "\fB**\fP", then it -is matched against the full pathname, including any leading directories. If -the pattern doesn't contain a \fB/\fP or a "\fB**\fP", then it is matched only against -the final component of the filename. (Remember that the algorithm is applied -recursively so "full filename" can actually be any portion of a path from the -starting directory on down.) -.IP o -a trailing "\fBdir_name/***\fP" will match both the directory (as if "dir_name/" -had been specified) and everything in the directory (as if "\fBdir_name/**\fP" -had been specified). This behavior was added in version 2.6.7. -.P -Note that, when using the \fB\-\-recursive\fP (\fB\-r\fP) option (which is implied -by \fB\-a\fP), every subdir component of every path is visited left to -right, with each directory having a chance for exclusion before its content. -In this way include/exclude patterns are applied recursively to the pathname of -each node in the filesystem's tree (those inside the transfer). The exclude -patterns short-circuit the directory traversal stage as rsync finds the files -to send. +Most of the rules mentioned above take an argument that specifies what the rule +should match. If rsync is recursing through a directory hierarchy, keep in +mind that each pattern is matched against the name of every directory in the +descent path as rsync finds the filenames to send. .P -For instance, to include "\fB/foo/bar/baz\fP", the directories "\fB/foo\fP" and "\fB/foo/bar\fP" -must not be excluded. Excluding one of those parent directories prevents the -examination of its content, cutting off rsync's recursion into those paths and -rendering the include for "\fB/foo/bar/baz\fP" ineffectual (since rsync can't match -something it never sees in the cut-off section of the directory hierarchy). -.P -The concept path exclusion is particularly important when using a trailing '\fB*\fP' -rule. For instance, this won't work: -.RS 4 +The matching rules for the pattern argument take several forms: .P -.nf -+ /some/path/this-file-will-not-be-found -+ /file-is-included -- * -.fi -.RE +.IP o +If a pattern contains a \fB/\fP (not counting a trailing slash) or a "\fB**\fP" +(which can match a slash), then the pattern is matched against the full +pathname, including any leading directories within the transfer. If the +pattern doesn't contain a (non-trailing) \fB/\fP or a "\fB**\fP", then it is matched +only against the final component of the filename or pathname. For example, +\fBfoo\fP means that the final path component must be "foo" while \fBfoo/bar\fP would +match the last 2 elements of the path (as long as both elements are within +the transfer). +.IP o +A pattern that ends with a \fB/\fP only matches a directory, not a regular file, +symlink, or device. +.IP o +A pattern that starts with a \fB/\fP is anchored to the start of the transfer +path instead of the end. For example, \fB/foo/**\fP or \fB/foo/bar/**\fP match only +leading elements in the path. If the rule is read from a per-directory +filter file, the transfer path being matched will begin at the level of the +filter file instead of the top of the transfer. See the section on +ANCHORING INCLUDE/EXCLUDE PATTERNS for a full discussion of how to +specify a pattern that matches at the root of the transfer. .P -This fails because the parent directory "some" is excluded by the '\fB*\fP' rule, so -rsync never visits any of the files in the "some" or "some/path" directories. -One solution is to ask for all directories in the hierarchy to be included by -using a single rule: "\fB+\ */\fP" (put it somewhere before the "\fB\-\ *\fP" rule), and -perhaps use the \fB\-\-prune-empty-dirs\fP option. Another solution is to add -specific include rules for all the parent dirs that need to be visited. For -instance, this set of rules works fine: -.RS 4 +Rsync chooses between doing a simple string match and wildcard matching by +checking if the pattern contains one of these three wildcard characters: '\fB*\fP', +\&'\fB?\fP', and '\fB[\fP' : .P -.nf -+ /some/ -+ /some/path/ -+ /some/path/this-file-is-found -+ /file-also-included -- * -.fi -.RE +.IP o +a '\fB?\fP' matches any single character except a slash (\fB/\fP). +.IP o +a '\fB*\fP' matches zero or more non-slash characters. +.IP o +a '\fB**\fP' matches zero or more characters, including slashes. +.IP o +a '\fB[\fP' introduces a character class, such as \fB[a-z]\fP or \fB[[:alpha:]]\fP, that +must match one character. +.IP o +a trailing \fB***\fP in the pattern is a shorthand that allows you to match a +directory and all its contents using a single rule. For example, specifying +"\fBdir_name/***\fP" will match both the "dir_name" directory (as if "\fBdir_name/\fP" +had been specified) and everything in the directory (as if "\fBdir_name/**\fP" +had been specified). +.IP o +a backslash can be used to escape a wildcard character, but it is only +interpreted as an escape character if at least one wildcard character is +present in the match pattern. For instance, the pattern "\fBfoo\\bar\fP" matches +that single backslash literally, while the pattern "\fBfoo\\bar*\fP" would need to +be changed to "\fBfoo\\\\bar*\fP" to avoid the "\fB\\b\fP" becoming just "b". .P Here are some examples of exclude/include matching: .P .IP o -"\fB\-\ *.o\fP" would exclude all names matching \fB*.o\fP +Option \fB\-f'\-\ *.o'\fP would exclude all filenames ending with \fB.o\fP .IP o -"\fB\-\ /foo\fP" would exclude a file (or directory) named foo in the transfer-root -directory +Option \fB\-f'\-\ /foo'\fP would exclude a file (or directory) named foo in the +transfer-root directory .IP o -"\fB\-\ foo/\fP" would exclude any directory named foo +Option \fB\-f'\-\ foo/'\fP would exclude any directory named foo .IP o -"\fB\-\ /foo/*/bar\fP" would exclude any file named bar which is at two levels -below a directory named foo in the transfer-root directory +Option \fB\-f'\-\ foo/*/bar'\fP would exclude any file/dir named bar which is at two +levels below a directory named foo (if foo is in the transfer) .IP o -"\fB\-\ /foo/**/bar\fP" would exclude any file named bar two or more levels below a -directory named foo in the transfer-root directory +Option \fB\-f'\-\ /foo/**/bar'\fP would exclude any file/dir named bar that was two +or more levels below a top-level directory named foo (note that /foo/bar is +\fBnot\fP excluded by this) .IP o -The combination of "\fB+\ */\fP", "\fB+\ *.c\fP", and "\fB\-\ *\fP" would include all -directories and C source files but nothing else (see also the -\fB\-\-prune-empty-dirs\fP option) +Options \fB\-f'+\ */'\ \-f'+\ *.c'\ \-f'\-\ *'\fP would include all directories and .c +source files but nothing else .IP o -The combination of "\fB+\ foo/\fP", "\fB+\ foo/bar.c\fP", and "\fB\-\ *\fP" would include -only the foo directory and foo/bar.c (the foo directory must be explicitly -included or it would be excluded by the "\fB*\fP") +Options \fB\-f'+\ foo/'\ \-f'+\ foo/bar.c'\ \-f'\-\ *'\fP would include only the foo +directory and foo/bar.c (the foo directory must be explicitly included or it +would be excluded by the "\fB\-\ *\fP") +.P +.SS "FILTER RULE MODIFIERS" .P -The following modifiers are accepted after a "\fB+\fP" or "\fB\-\fP": +The following modifiers are accepted after an include (+) or exclude (\-) rule: .P .IP o A \fB/\fP specifies that the include/exclude rule should be matched against the -absolute pathname of the current item. For example, "\fB\-/\ /etc/passwd\fP" would -exclude the passwd file any time the transfer was sending files from the -"/etc" directory, and "\-/ subdir/foo" would always exclude "foo" when it is -in a dir named "subdir", even if "foo" is at the root of the current +absolute pathname of the current item. For example, \fB\-f'\-/\ /etc/passwd'\fP +would exclude the passwd file any time the transfer was sending files from +the "/etc" directory, and "\-/ subdir/foo" would always exclude "foo" when it +is in a dir named "subdir", even if "foo" is at the root of the current transfer. .IP o A \fB!\fP specifies that the include/exclude should take effect if the pattern -fails to match. For instance, "\fB\-!\ */\fP" would exclude all non-directories. +fails to match. For instance, \fB\-f'\-!\ */'\fP would exclude all non-directories. .IP o A \fBC\fP is used to indicate that all the global CVS-exclude rules should be inserted as excludes in place of the "\-C". No arg should follow. .IP o An \fBs\fP is used to indicate that the rule applies to the sending side. When a -rule affects the sending side, it prevents files from being transferred. The -default is for a rule to affect both sides unless \fB\-\-delete-excluded\fP -was specified, in which case default rules become sender-side only. See also -the hide (H) and show (S) rules, which are an alternate way to specify -sending-side includes/excludes. +rule affects the sending side, it affects what files are put into the +sender's file list. The default is for a rule to affect both sides unless +\fB\-\-delete-excluded\fP was specified, in which case default rules become +sender-side only. See also the hide (H) and show (S) rules, which are an +alternate way to specify sending-side includes/excludes. .IP o An \fBr\fP is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See @@ -4013,7 +4215,7 @@ operations (and is thus ignored when mat xattr-matching rules are specified, a default xattr filtering rule is used (see the \fB\-\-xattrs\fP option). .P -.SH "MERGE-FILE FILTER RULES" +.SS "MERGE-FILE FILTER RULES" .P You can merge whole files into your filter rules by specifying either a merge (.) or a dir-merge (:) filter rule (as introduced in the FILTER RULES @@ -4172,7 +4374,7 @@ $HOME/.cvsignore, and the value of $CVSI command-line option and instead insert a "\-C" rule into your filter rules; e.g. "\fB\-\-filter=\-C\fP". .P -.SH "LIST-CLEARING FILTER RULE" +.SS "LIST-CLEARING FILTER RULE" .P You can clear the current include/exclude list by using the "!" filter rule (as introduced in the FILTER RULES section above). The "current" list is either @@ -4180,7 +4382,7 @@ the global list of rules (if the rule is options) or a set of per-directory rules (which are inherited in their own sub-list, so a subdirectory can use this to clear out the parent's rules). .P -.SH "ANCHORING INCLUDE/EXCLUDE PATTERNS" +.SS "ANCHORING INCLUDE/EXCLUDE PATTERNS" .P As mentioned earlier, global include/exclude patterns are anchored at the "root of the transfer" (as opposed to per-directory patterns, which are anchored at @@ -4243,7 +4445,7 @@ The easiest way to see what name you sho output when using \fB\-\-verbose\fP and put a / in front of the name (use the \fB\-\-dry-run\fP option if you're not yet ready to copy any files). .P -.SH "PER-DIRECTORY RULES AND DELETE" +.SS "PER-DIRECTORY RULES AND DELETE" .P Without a delete option, per-directory rules are only relevant on the sending side, so you can feel free to exclude the merge files themselves without @@ -4303,6 +4505,38 @@ rsync -avFF --delete host:src/dir /dest .fi .RE .P +.SH "TRANSFER RULES" +.P +In addition to the FILTER RULES that affect the recursive file scans that +generate the file list on the sending and (when deleting) receiving sides, +there are transfer rules. These rules affect which files the generator decides +need to be transferred without the side effects of an exclude filter rule. +Transfer rules affect only files and never directories. +.P +Because a transfer rule does not affect what goes into the sender's (and +receiver's) file list, it cannot have any effect on which files get deleted on +the receiving side. For example, if the file "foo" is present in the sender's +list but its size is such that it is omitted due to a transfer rule, the +receiving side does not request the file. However, its presence in the file +list means that a delete pass will not remove a matching file named "foo" on +the receiving side. On the other hand, a server-side exclude (hide) of the +file "foo" leaves the file out of the server's file list, and absent a +receiver-side exclude (protect) the receiver will remove a matching file named +"foo" if deletions are requested. +.P +Given that the files are still in the sender's file list, the +\fB\-\-prune-empty-dirs\fP option will not judge a directory as being empty +even if it contains only files that the transfer rules omitted. +.P +Similarly, a transfer rule does not have any extra effect on which files are +deleted on the receiving side, so setting a maximum file size for the transfer +does not prevent big files from being deleted. +.P +Examples of transfer rules include the default "quick check" algorithm (which +compares size & modify time), the \fB\-\-update\fP option, the +\fB\-\-max-size\fP option, the \fB\-\-ignore-non-existing\fP option, and a +few others. +.P .SH "BATCH MODE" .P Batch mode can be used to apply the same set of updates to many identical @@ -4627,7 +4861,7 @@ to use when making a daemon connection. DAEMON for full details. .IP "\fBRSYNC_SHELL\fP" This environment variable is mainly used in debug setups to set the program -to use to run the program specified by [\fBRSYNC_CONNECT_PROG\fP]. See +to use to run the program specified by \fBRSYNC_CONNECT_PROG\fP. See CONNECTING TO AN RSYNC DAEMON for full details. .P .SH "FILES" @@ -4654,7 +4888,7 @@ Please report bugs! See the web site at .P .SH "VERSION" .P -This manpage is current for version 3.2.4 of rsync. +This manpage is current for version 3.2.5 of rsync. .P .SH "INTERNAL OPTIONS" .P diff -upN a/rsync.1.html b/rsync.1.html --- a/rsync.1.html +++ b/rsync.1.html @@ -168,6 +168,42 @@ rsync daemon by leaving off the module n

See the following section for more details.

+

SORTED TRANSFER ORDER

+

Rsync always sorts the specified filenames into its internal transfer list. +This handles the merging together of the contents of identically named +directories, makes it easy to remove duplicate filenames. It can, however, +confuse someone when the files are transferred in a different order than what +was given on the command-line.

+

If you need a particular file to be transferred prior to another, either +separate the files into different rsync calls, or consider using +--delay-updates (which doesn't affect the sorted transfer order, but +does make the final file-updating phase happen much more rapidly).

+

MULTI-HOST SECURITY

+

Rsync takes steps to ensure that the file requests that are shared in a +transfer are protected against various security issues. Most of the potential +problems arise on the receiving side where rsync takes steps to ensure that the +list of files being transferred remains within the bounds of what was +requested.

+

Toward this end, rsync 3.1.2 and later have aborted when a file list contains +an absolute or relative path that tries to escape out of the top of the +transfer. Also, beginning with version 3.2.5, rsync does two more safety +checks of the file list to (1) ensure that no extra source arguments were added +into the transfer other than those that the client requested and (2) ensure +that the file list obeys the exclude rules that were sent to the sender.

+

For those that don't yet have a 3.2.5 client rsync (or those that want to be +extra careful), it is safest to do a copy into a dedicated destination +directory for the remote files when you don't trust the remote host. For +example, instead of doing an rsync copy into your home directory:

+
+
rsync -aiv host1:dir1 ~
+
+
+

Dedicate a "host1-files" dir to the remote content:

+
+
rsync -aiv host1:dir1 ~/host1-files
+
+
+

See the --trust-sender option for additional details.

ADVANCED USAGE

The syntax for requesting multiple files from a remote host is done by specifying additional remote-host args in the same style as the first, or with @@ -178,22 +214,24 @@ rsync -aiv host::modname/file{1,2} host: rsync -aiv host::modname/first ::modname/extra{1,2} /dest/ -

In a modern rsync, you only need to quote or backslash-escape things like -spaces from the local shell but not also from the remote shell:

+

Really old versions of rsync (2.6.9 and before) only allowed specifying one +remote-source arg, so some people have instead relied on the remote-shell +performing space splitting to break up an arg into multiple paths. Such +unintuitive behavior is no longer supported by default (though you can request +it, as described below).

+

Starting in 3.2.4, filenames are passed to a remote shell in such a way as to +preserve the characters you give it. Thus, if you ask for a file with spaces +in the name, that's what the remote rsync looks for:

rsync -aiv host:'a simple file.pdf' /dest/
-

Really old versions of rsync only allowed specifying one remote-source arg, so -it required the remote side to split the args at a space. You can still get -this old-style arg splitting by using the --old-args option:

-
-
rsync -ai --old-args host:'dir1/file1 dir2/file2' /dest
-rsync -ai --old-args host::'modname/dir1/file1 modname/dir2/file2' /dest
-
-
-

See that option's section for an environment variable that -can be exported to help old scripts.

+

If you use scripts that have been written to manually apply extra quoting to +the remote rsync args (or to require remote arg splitting), you can ask rsync +to let your script handle the extra escaping. This is done by either adding +the --old-args option to the rsync runs in the script (which requires +a new rsync) or exporting RSYNC_OLD_ARGS=1 and RSYNC_PROTECT_ARGS=0 +(which works with old or new rsync versions).

CONNECTING TO AN RSYNC DAEMON

It is also possible to use rsync without a remote shell as the transport. In this case you will directly connect to a remote rsync daemon, typically using @@ -279,6 +317,9 @@ example that uses the short version of t

The "ssh-user" will be used at the ssh level; the "rsync-user" will be used to log-in to the "module".

+

In this setup, the daemon is started by the ssh command that is accessing the +system (which can be forced via the ~/.ssh/authorized_keys file, if desired). +However, when accessing a daemon directly, it needs to be started beforehand.

STARTING AN RSYNC DAEMON TO ACCEPT CONNECTIONS

In order to connect to an rsync daemon, the remote system needs to have a daemon already running (or it needs to have configured something like inetd to @@ -289,44 +330,19 @@ the config file for the daemon, and it c the daemon (including stand-alone and inetd configurations).

If you're using one of the remote-shell transports for the transfer, there is no need to manually start an rsync daemon.

-

SORTED TRANSFER ORDER

-

Rsync always sorts the specified filenames into its internal transfer list. -This handles the merging together of the contents of identically named -directories, makes it easy to remove duplicate filenames, and may confuse -someone when the files are transferred in a different order than what was given -on the command-line.

-

If you need a particular file to be transferred prior to another, either -separate the files into different rsync calls, or consider using ---delay-updates (which doesn't affect the sorted transfer order, but -does make the final file-updating phase happen much more rapidly).

EXAMPLES

-

Here are some examples of how I use rsync.

-

To backup my wife's home directory, which consists of large MS Word files and -mail folders, I use a cron job that runs

-
-
rsync -Cavz . arvidsjaur:backup
-
-
-

each night over a PPP connection to a duplicate directory on my machine -"arvidsjaur".

-

To synchronize my samba source trees I use the following Makefile targets:

+

Here are some examples of how rsync can be used.

+

To backup a home directory, which consists of large MS Word files and mail +folders, a per-user cron job can be used that runs this each day:

-
get:
-    rsync -avuzb --exclude '*~' samba:samba/ .
-put:
-    rsync -Cavuzb . samba:samba/
-sync: get put
+
rsync -aiz . bkhost:backup/joe/
-

This allows me to sync with a CVS directory at the other end of the connection. -I then do CVS operations on the remote machine, which saves a lot of time as -the remote CVS protocol isn't very efficient.

-

I mirror a directory between my "old" and "new" ftp sites with the command:

+

To move some files from a remote host to the local host, you could run:

-
rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"
+
rsync -aiv --remove-source-files rhost:/tmp/{file1,file2}.c ~/src/
-

This is launched from cron every few hours.

OPTION SUMMARY

Here is a short summary of the options available in rsync. Each option also has its own detailed description later in this manpage.

@@ -441,6 +457,7 @@ has its own detailed description later i --from0, -0 all *-from/filter files are delimited by 0s --old-args disable the modern arg-protection idiom --protect-args, -s no space-splitting; wildcard chars only +--trust-sender trust the remote sender's file list --copy-as=USER[:GROUP] specify user & optional group for the copy --address=ADDRESS bind address for outgoing socket to daemon --port=PORT specify double-colon alternate port number @@ -496,14 +513,17 @@ accepted:

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options. The full list of the available options are described below. If an option can be specified in more than one way, the choices are comma-separated. -Some options only have a long variant, not a short. If the option takes a -parameter, the parameter is only listed after the long variant, even though it -must also be specified for the short. When specifying a parameter, you can -either use the form --option=param or replace the '=' with whitespace. The -parameter may need to be quoted in some manner for it to survive the shell's -command-line parsing. Keep in mind that a leading tilde (~) in a filename is -substituted by your shell, so --option=~/foo will not change the tilde into -your home directory (remove the '=' for that).

+Some options only have a long variant, not a short.

+

If the option takes a parameter, the parameter is only listed after the long +variant, even though it must also be specified for the short. When specifying +a parameter, you can either use the form --option=param, --option param, +-o=param, -o param, or -oparam (the latter choices assume that your +option has a short variant).

+

The parameter may need to be quoted in some manner for it to survive the +shell's command-line parsing. Also keep in mind that a leading tilde (~) in +a pathname is substituted by your shell, so make sure that you separate the +option name from the pathname using a space if you want the local shell to +expand it.

--help
@@ -772,7 +792,7 @@ disable the incremental recursion mode.

In order to make --delete compatible with incremental recursion, rsync 3.0.0 made --delete-during the default delete mode (which -was first first added in 2.6.4).

+was first added in 2.6.4).

One side-effect of incremental recursion is that any missing sub-directories inside a recursively-scanned directory are (by default) created prior to recursing into the sub-dirs. This earlier creation point @@ -923,9 +943,8 @@ receiver is always considered to be impo matter what date is on the objects. In other words, if the source has a directory where the destination has a file, the transfer would occur regardless of the timestamps.

-

This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred.

+

This option is a TRANSFER RULE, so don't expect any +exclude side effects.

A caution for those that choose to combine --inplace with --update: an interrupted transfer will leave behind a partial file on the receiving side that has a very recent modified time, so re-running the @@ -1122,7 +1141,7 @@ has/needs the munged symlinks, or use -

This option has no affect when sent to a daemon via --remote-option +

This option has no effect when sent to a daemon via --remote-option because the daemon configures whether it wants munged symlinks via its "munge symlinks" parameter.

The symlink value is munged/unmunged once it is in the transfer, so any @@ -1415,7 +1434,7 @@ by specifying --copy-devices

This tells rsync to treat a device on the sending side as a regular file, allowing it to be copied to a normal destination file (or another device -if --write-devices was also specifed).

+if --write-devices was also specified).

This option is refused by default by an rsync daemon.

@@ -1646,18 +1665,16 @@ by this option.

exist yet on the destination. If this option is combined with the --ignore-existing option, no files will be updated (which can be useful if all you want to do is delete extraneous files).

-

This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred.

+

This option is a TRANSFER RULE, so don't expect any +exclude side effects.

--ignore-existing

This tells rsync to skip updating files that already exist on the destination (this does not ignore existing directories, or nothing would get done). See also --ignore-non-existing.

-

This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred.

+

This option is a TRANSFER RULE, so don't expect any +exclude side effects.

This option can be useful for those doing backups using the --link-dest option when they need to continue a backup run that got interrupted. Since a --link-dest run is copied into a new @@ -1776,13 +1793,21 @@ for those that just want the deletions t

--delete-excluded
-

In addition to deleting the files on the receiving side that are not on the -sending side, this tells rsync to also delete any files on the receiving -side that are excluded (see --exclude). See the FILTER -RULES section for a way to make individual exclusions behave this way -on the receiver, and for a way to protect files from --delete-excluded. -See --delete (which is implied) for more details on -file-deletion.

+

This option turns any unqualified exclude/include rules into server-side +rules that do not affect the receiver's deletions.

+

By default, an exclude or include has both a server-side effect (to "hide" +and "show" files when building the server's file list) and a receiver-side +effect (to "protect" and "risk" files when deletions are occuring). Any +rule that has no modifier to specify what sides it is executed on will be +instead treated as if it were a server-side rule only, avoiding any +"protect" effects of the rules.

+

A rule can still apply to both sides even with this option specified if the +rule is given both the sender & receiver modifer letters (e.g., -f'-sr foo'). Receiver-side protect/risk rules can also be explicitly specified +to limit the deletions. This saves you from having to edit a bunch of +-f'- foo' rules into -f'-s foo' (aka -f'H foo') rules (not to mention +the corresponding includes).

+

See the FILTER RULES section for more information. See +--delete (which is implied) for more details on deletion.

--ignore-missing-args
@@ -1839,9 +1864,8 @@ really old versions didn't warn when the specified SIZE. A numeric value can be suffixed with a string to indicate the numeric units or left unqualified to specify bytes. Feel free to use a fractional value along with the units, such as --max-size=1.5m.

-

This option is a transfer rule, not an exclude, so it doesn't affect the -data that goes into the file-lists, and thus it doesn't affect deletions. -It just limits the files that the receiver requests to be transferred.

+

This option is a TRANSFER RULE, so don't expect any +exclude side effects.

The first letter of a units string can be B (bytes), K (kilo), M (mega), G (giga), T (tera), or P (peta). If the string is a single char or has "ib" added to it (e.g. "G" or "GiB") then the units are @@ -2078,8 +2102,8 @@ options work.

--exclude=PATTERN

This option is a simplified form of the --filter option that -defaults to an exclude rule and does not allow the full rule-parsing syntax -of normal filter rules.

+specifies an exclude rule and does not allow the full rule-parsing syntax +of normal filter rules. This is equivalent to specifying -f'- PATTERN'.

See the FILTER RULES section for detailed information on this option.

@@ -2088,13 +2112,18 @@ of normal filter rules.

a FILE that contains exclude patterns (one per line). Blank lines in the file are ignored, as are whole-line comments that start with ';' or '#' (filename rules that contain those characters are unaffected).

+

If a line begins with "- " (dash, space) or "+ " (plus, space), then +the type of rule is being explicitly specified as an exclude or an include +(respectively). Any rules without such a prefix are taken to be an exclude.

+

If a line consists of just "!", then the current filter rules are cleared +before adding any further rules.

If FILE is '-', the list will be read from standard input.

--include=PATTERN

This option is a simplified form of the --filter option that -defaults to an include rule and does not allow the full rule-parsing syntax -of normal filter rules.

+specifies an include rule and does not allow the full rule-parsing syntax +of normal filter rules. This is equivalent to specifying -f'+ PATTERN'.

See the FILTER RULES section for detailed information on this option.

@@ -2103,6 +2132,11 @@ of normal filter rules.

a FILE that contains include patterns (one per line). Blank lines in the file are ignored, as are whole-line comments that start with ';' or '#' (filename rules that contain those characters are unaffected).

+

If a line begins with "- " (dash, space) or "+ " (plus, space), then +the type of rule is being explicitly specified as an exclude or an include +(respectively). Any rules without such a prefix are taken to be an include.

+

If a line consists of just "!", then the current filter rules are cleared +before adding any further rules.

If FILE is '-', the list will be read from standard input.

@@ -2192,6 +2226,11 @@ setting. If it has the value "2&qu repeated-option setting. If it is "0", you'll get the default escaping behavior. The environment is always overridden by manually specified positive or negative options (the negative is --no-old-args).

+

Note that this option also disables the extra safety check added in 3.2.5 +that ensures that a remote sender isn't including extra top-level items in +the file-list that you didn't request. This side-effect is necessary +because we can't know for sure what names to expect when the remote shell +is interpreting the args.

This option conflicts with the --protect-args option.

@@ -2220,6 +2259,40 @@ versions). This environment variable is script (rrsync) since it hides options from the script's inspection.

+
--trust-sender
+

This option disables two extra validation checks that a local client +performs on the file list generated by a remote sender. This option should +only be used if you trust the sender to not put something malicious in the +file list (something that could possibly be done via a modified rsync, a +modified shell, or some other similar manipulation).

+

Normally, the rsync client (as of version 3.2.5) runs two extra validation +checks when pulling files from a remote rsync:

+
    +
  • It verifies that additional arg items didn't get added at the top of the +transfer.
    • +
  • It verifies that none of the items in the file list are names that should +have been excluded (if filter rules were specified).
    • +
+

Note that various options can turn off one or both of these checks if the +option interferes with the validation. For instance:

+
    +
  • Using a per-directory filter file reads filter rules that only the server +knows about, so the filter checking is disabled.
    • +
  • Using the --old-args option allows the sender to manipulate the +requested args, so the arg checking is disabled.
    • +
  • Reading the files-from list from the server side means that the client +doesn't know the arg list, so the arg checking is disabled.
    • +
  • Using --read-batch disables both checks since the batch file's +contents will have been verified when it was created.
    • +
+

This option may help an under-powered client server if the extra pattern +matching is slowing things down on a huge transfer. It can also be used to +work around a currently-unknown bug in the verification logic for a transfer +from a trusted sender.

+

When using this option it is a good idea to specify a dedicated destination +directory, as discussed in the MULTI-HOST SECURITY section.

+
+
--copy-as=USER[:GROUP]

This option instructs rsync to use the USER and (if specified after a colon) the GROUP for the copy operations. This only works if the user that @@ -2833,7 +2906,7 @@ client side and/or the server side of a as a client option, transfer logging will be enabled with a default format of "%i %n%L". See the --log-file-format option if you wish to override this.

-

Here's a example command that requests the remote side to log what is +

Here's an example command that requests the remote side to log what is happening:

rsync -av --remote-option=--log-file=/tmp/rlog src/ dest/
@@ -3056,10 +3129,8 @@ the file-list, including nested director
 children.  This is useful for avoiding the creation of a bunch of useless
 directories when the sending rsync is recursively scanning a hierarchy of
 files using include/exclude/filter rules.

-
Note that the use of transfer rules, such as the --min-size
-option, does not affect what goes into the file list, and thus does not
-leave directories empty, even if none of the files in a directory match the
-transfer rule.

+
This option can still leave empty directories on the receiving side if you
+make use of TRANSFER_RULES.

 
Because the file-list is actually being pruned, this option also affects
 what directories get deleted when a delete is active.  However, keep in
 mind that excluded files and directories can prevent existing items from
@@ -3188,8 +3259,8 @@ file-listing command, or
 include the destination.
 
 
CAUTION: keep in mind that a source arg with a wild-card is expanded by the
-shell into multiple args, so it is never safe to try to list such an arg
-without using this option. For example:

+shell into multiple args, so it is never safe to try to specify a single
+wild-card arg to try to infer this option. A safe example is:

 

 
rsync -av --list-only foo* dest/
 

@@ -3488,24 +3559,134 @@ options available for starting an rsync

FILTER RULES

-

The filter rules allow for flexible selection of which files to transfer -(include) and which files to skip (exclude). The rules either directly specify -include/exclude patterns or they specify a way to acquire more include/exclude -patterns (e.g. to read them from a file).

-

As the list of files/directories to transfer is built, rsync checks each name -to be transferred against the list of include/exclude patterns in turn, and the -first matching pattern is acted on: if it is an exclude pattern, then that file -is skipped; if it is an include pattern then that filename is not skipped; if -no matching pattern is found, then the filename is not skipped.

-

Aside: because the interactions of filter rules can be complex, it is useful to -use the --debug=FILTER option if things aren't working the way you expect. -The level-1 output (the default if no level number is specified) mentions the -filter rule that is first matched by each file in the transfer. It also warns -if a filter rule has trailing whitespace. The level-2 output mentions a lot -more filter events, including the definition of each rule and the handling of -per-directory filter files.

-

Rsync builds an ordered list of filter rules as specified on the command-line. -Filter rules have the following syntax:

+

The filter rules allow for custom control of several aspects of how files are +handled:

+
    +
  • Control which files the sending side puts into the file list that describes +the transfer hierarchy
    • +
  • Control which files the receiving side protects from deletion when the file +is not in the sender's file list
    • +
  • Control which extended attribute names are skipped when copying xattrs
    • +
+

The rules are either directly specified via option arguments or they can be +read in from one or more files. The filter-rule files can even be a part of +the hierarchy of files being copied, affecting different parts of the tree in +different ways.

+

SIMPLE INCLUDE/EXCLUDE RULES

+

We will first cover the basics of how include & exclude rules affect what files +are transferred, ignoring any deletion side-effects. Filter rules mainly +affect the contents of directories that rsync is "recursing" into, but they can +also affect a top-level item in the transfer that was specified as a argument.

+

The default for any unmatched file/dir is for it to be included in the +transfer, which puts the file/dir into the sender's file list. The use of an +exclude rule causes one or more matching files/dirs to be left out of the +sender's file list. An include rule can be used to limit the effect of an +exclude rule that is matching too many files.

+

The order of the rules is important because the first rule that matches is the +one that takes effect. Thus, if an early rule excludes a file, no include rule +that comes after it can have any effect. This means that you must place any +include overrides somewhere prior to the exclude that it is intended to limit.

+

When a directory is excluded, all its contents and sub-contents are also +excluded. The sender doesn't scan through any of it at all, which can save a +lot of time when skipping large unneeded sub-trees.

+

It is also important to understand that the include/exclude rules are applied +to every file and directory that the sender is recursing into. Thus, if you +want a particular deep file to be included, you have to make sure that none of +the directories that must be traversed on the way down to that file are +excluded or else the file will never be discovered to be included. As an +example, if the directory "a/path" was given as a transfer argument and you +want to ensure that the file "a/path/down/deep/wanted.txt" is a part of the +transfer, then the sender must not exclude the directories "a/path", +"a/path/down", or "a/path/down/deep" as it makes it way scanning through +the file tree.

+

When you are working on the rules, it can be helpful to ask rsync to tell you +what is being excluded/included and why. Specifying --debug=FILTER or (when +pulling files) -M--debug=FILTER turns on level 1 of the FILTER debug +information that will output a message any time that a file or directory is +included or excluded and which rule it matched. Beginning in 3.2.4 it will +also warn if a filter rule has trailing whitespace, since an exclude of "foo " +(with a trailing space) will not exclude a file named "foo".

+

Exclude and include rules can specify wildcard PATTERN MATCHING RULES +(similar to shell wildcards) that allow you to match things like a file suffix +or a portion of a filename.

+

A rule can be limited to only affecting a directory by putting a trailing slash +onto the filename.

+

SIMPLE INCLUDE/EXCLUDE EXAMPLE

+

With the following file tree created on the sending side:

+
+
mkdir x/
+touch x/file.txt
+mkdir x/y/
+touch x/y/file.txt
+touch x/y/zzz.txt
+mkdir x/z/
+touch x/z/file.txt
+
+
+

Then the following rsync command will transfer the file "x/y/file.txt" and +the directories needed to hold it, resulting in the path "/tmp/x/y/file.txt" +existing on the remote host:

+
+
rsync -ai -f'+ x/' -f'+ x/y/' -f'+ x/y/file.txt' -f'- *' x host:/tmp/
+
+
+

Aside: this copy could also have been accomplished using the -R +option (though the 2 commands behave differently if deletions are enabled):

+
+
rsync -aiR x/y/file.txt host:/tmp/
+
+
+

The following command does not need an include of the "x" directory because it +is not a part of the transfer (note the traililng slash). Running this command +would copy just "/tmp/x/file.txt" because the "y" and "z" dirs get excluded:

+
+
rsync -ai -f'+ file.txt' -f'- *' x/ host:/tmp/x/
+
+
+

This command would omit the zzz.txt file while copying "x" and everything else +it contains:

+
+
rsync -ai -f'- zzz.txt' x host:/tmp/
+
+
+

FILTER RULES WHEN DELETING

+

By default the include & exclude filter rules affect both the sender +(as it creates its file list) +and the receiver (as it creates its file lists for calculating deletions). If +no delete option is in effect, the receiver skips creating the delete-related +file lists. This two-sided default can be manually overridden so that you are +only specifying sender rules or receiver rules, as described in the FILTER +RULES IN DEPTH section.

+

When deleting, an exclude protects a file from being removed on the receiving +side while an include overrides that protection (putting the file at risk of +deletion). The default is for a file to be at risk -⁠-⁠ its safety depends on it +matching a corresponding file from the sender.

+

An example of the two-sided exclude effect can be illustrated by the copying of +a C development directory between 2 systems. When doing a touch-up copy, you +might want to skip copying the built executable and the .o files (sender +hide) so that the receiving side can build their own and not lose any object +files that are already correct (receiver protect). For instance:

+
+
rsync -ai --del -f'- *.o' -f'- cmd' src host:/dest/
+
+
+

Note that using -f'-p *.o' is even better than -f'- *.o' if there is a +chance that the directory structure may have changed. The "p" modifier is +discussed in FILTER RULE MODIFIERS.

+

One final note, if your shell doesn't mind unexpanded wildcards, you could +simplify the typing of the filter options by using an underscore in place of +the space and leaving off the quotes. For instance, -f -_*.o -f -_cmd (and +similar) could be used instead of the filter options above.

+

FILTER RULES IN DEPTH

+

Rsync supports old-style include/exclude rules and new-style filter rules. The +older rules are specified using --include and --exclude as +well as the --include-from and --exclude-from. These are +limited in behavior but they don't require a "-⁠" or "+" prefix. An old-style +exclude rule is turned into a "- name" filter rule (with no modifiers) and an +old-style include rule is turned into a "+ name" filter rule (with no +modifiers).

+

Rsync builds an ordered list of filter rules as specified on the command-line +and/or read-in from files. New style filter rules have the following syntax:

RULE [PATTERN_OR_FILENAME]
 RULE,MODIFIERS [PATTERN_OR_FILENAME]
@@ -3514,150 +3695,120 @@ RULE,MODIFIERS [PATTERN_OR_FILENAME]
 
You have your choice of using either short or long RULE names, as described
 below.  If you use a short-named rule, the ',' separating the RULE from the
 MODIFIERS is optional.  The PATTERN or FILENAME that follows (when present)
-must come after either a single space or an underscore (_).  Here are the
-available rule prefixes:

+must come after either a single space or an underscore (_). Any additional
+spaces and/or underscores are considered to be a part of the pattern name.
+Here are the available rule prefixes:

 

-
exclude, '-'
 specifies an exclude pattern.

-
include, '+'
 specifies an include pattern.

-
merge, '.'
 specifies a merge-file to read for more rules.

-
dir-merge, ':'
 specifies a per-directory merge-file.

-
hide, 'H'
 specifies a pattern for hiding files from the transfer.

-
show, 'S'
 files that match the pattern are not hidden.

-
protect, 'P'
 specifies a pattern for protecting files from deletion.

-
risk, 'R'
 files that match the pattern are not protected.

+
exclude, '-'
 specifies an exclude pattern that (by default) is both a
+hide and a protect.

+
include, '+'
 specifies an include pattern that (by default) is both a
+show and a risk.

+
merge, '.'
 specifies a merge-file on the client side to read for more
+rules.

+
dir-merge, ':'
 specifies a per-directory merge-file.  Using this kind of
+filter rule requires that you trust the sending side's filter checking, so
+it has the side-effect mentioned under the --trust-sender option.

+
hide, 'H'
 specifies a pattern for hiding files from the transfer.
+Equivalent to a sender-only exclude, so -f'H foo' could also be specified
+as -f'-s foo'.

+
show, 'S'
 files that match the pattern are not hidden. Equivalent to a
+sender-only include, so -f'S foo' could also be specified as -f'+s foo'.

+
protect, 'P'
 specifies a pattern for protecting files from deletion.
+Equivalent to a receiver-only exclude, so -f'P foo' could also be
+specified as -f'-r foo'.

+
risk, 'R'
 files that match the pattern are not protected. Equivalent to a
+receiver-only include, so -f'R foo' could also be specified as -f'+r foo'.

 
clear, '!'
 clears the current include/exclude list (takes no arg)

 

-
When rules are being read from a file, empty lines are ignored, as are
-whole-line comments that start with a '#' (filename rules that contain a hash
-are unaffected).

-
Note that the --include & --exclude command-line options do
-not allow the full range of rule parsing as described above -⁠-⁠ they only allow
-the specification of include / exclude patterns plus a "!" token to clear the
-list (and the normal comment parsing when rules are read from a file).  If a
-pattern does not begin with "- " (dash, space) or "+ " (plus, space), then
-the rule will be interpreted as if "+ " (for an include option) or "- "
-(for an exclude option) were prefixed to the string.  A --filter
-option, on the other hand, must always contain either a short or long rule name
-at the start of the rule.

+
When rules are being read from a file (using merge or dir-merge), empty lines
+are ignored, as are whole-line comments that start with a '#' (filename rules
+that contain a hash character are unaffected).

 
Note also that the --filter, --include, and
 --exclude options take one rule/pattern each.  To add multiple ones,
 you can repeat the options on the command-line, use the merge-file syntax of
 the --filter option, or the --include-from /
 --exclude-from options.

-
INCLUDE/EXCLUDE PATTERN RULES

-
You can include and exclude files by specifying patterns using the "+", "-⁠",
-etc. filter rules (as introduced in the FILTER RULES section above).  The
-include/exclude rules each specify a pattern that is matched against the names
-of the files that are going to be transferred.  These patterns can take several
-forms:

-
    
-
  • if the pattern starts with a / then it is anchored to a particular spot in
-the hierarchy of files, otherwise it is matched against the end of the
-pathname.  This is similar to a leading ^ in regular expressions.  Thus
-/foo would match a name of "foo" at either the "root of the transfer" (for
-a global rule) or in the merge-file's directory (for a per-directory rule).
-An unqualified foo would match a name of "foo" anywhere in the tree because
-the algorithm is applied recursively from the top down; it behaves as if each
-path component gets a turn at being the end of the filename.  Even the
-unanchored "sub/foo" would match at any point in the hierarchy where a "foo"
-was found within a directory named "sub".  See the section on ANCHORING
-INCLUDE/EXCLUDE PATTERNS for a full discussion of how to specify a pattern
-that matches at the root of the transfer.
    • 
-
  • if the pattern ends with a / then it will only match a directory, not a
-regular file, symlink, or device.
    • 
-
  • rsync chooses between doing a simple string match and wildcard matching by
-checking if the pattern contains one of these three wildcard characters:
-'*', '?', and '[' .
    • 
-
  • a '*' matches any path component, but it stops at slashes.
    • 
-
  • use '**' to match anything, including slashes.
    • 
-
  • a '?' matches any character except a slash (/).
    • 
-
  • a '[' introduces a character class, such as [a-z] or [[:alpha:]].
    • 
-
  • in a wildcard pattern, a backslash can be used to escape a wildcard
-character, but it is matched literally when no wildcards are present.  This
-means that there is an extra level of backslash removal when a pattern
-contains wildcard characters compared to a pattern that has none.  e.g. if
-you add a wildcard to "foo\bar" (which matches the backslash) you would
-need to use "foo\\bar*" to avoid the "\b" becoming just "b".
    • 
-
  • if the pattern contains a / (not counting a trailing /) or a "**", then it
-is matched against the full pathname, including any leading directories.  If
-the pattern doesn't contain a / or a "**", then it is matched only against
-the final component of the filename. (Remember that the algorithm is applied
-recursively so "full filename" can actually be any portion of a path from the
-starting directory on down.)
    • 
-
  • a trailing "dir_name/***" will match both the directory (as if "dir_name/"
+
    PATTERN MATCHING RULES
    
+
    Most of the rules mentioned above take an argument that specifies what the rule
+should match.  If rsync is recursing through a directory hierarchy, keep in
+mind that each pattern is matched against the name of every directory in the
+descent path as rsync finds the filenames to send.
    
+
    The matching rules for the pattern argument take several forms:
    
+
      
+
    • If a pattern contains a / (not counting a trailing slash) or a "**"
+(which can match a slash), then the pattern is matched against the full
+pathname, including any leading directories within the transfer.  If the
+pattern doesn't contain a (non-trailing) / or a "**", then it is matched
+only against the final component of the filename or pathname. For example,
+foo means that the final path component must be "foo" while foo/bar would
+match the last 2 elements of the path (as long as both elements are within
+the transfer).
      • 
+
    • A pattern that ends with a / only matches a directory, not a regular file,
+symlink, or device.
      • 
+
    • A pattern that starts with a / is anchored to the start of the transfer
+path instead of the end.  For example, /foo/** or /foo/bar/** match only
+leading elements in the path.  If the rule is read from a per-directory
+filter file, the transfer path being matched will begin at the level of the
+filter file instead of the top of the transfer.  See the section on
+ANCHORING INCLUDE/EXCLUDE PATTERNS for a full discussion of how to
+specify a pattern that matches at the root of the transfer.
      • 
+
    
+
    Rsync chooses between doing a simple string match and wildcard matching by
+checking if the pattern contains one of these three wildcard characters: '*',
+'?', and '[' :
    
+
      
+
    • a '?' matches any single character except a slash (/).
      • 
+
    • a '*' matches zero or more non-slash characters.
      • 
+
    • a '**' matches zero or more characters, including slashes.
      • 
+
    • a '[' introduces a character class, such as [a-z] or [[:alpha:]], that
+must match one character.
      • 
+
    • a trailing *** in the pattern is a shorthand that allows you to match a
+directory and all its contents using a single rule.  For example, specifying
+"dir_name/***" will match both the "dir_name" directory (as if "dir_name/"
 had been specified) and everything in the directory (as if "dir_name/**"
-had been specified).  This behavior was added in version 2.6.7.
      • 
+had been specified).
+
    • a backslash can be used to escape a wildcard character, but it is only
+interpreted as an escape character if at least one wildcard character is
+present in the match pattern. For instance, the pattern "foo\bar" matches
+that single backslash literally, while the pattern "foo\bar*" would need to
+be changed to "foo\\bar*" to avoid the "\b" becoming just "b".
      • 
 
    
-
    Note that, when using the --recursive (-r) option (which is implied
-by -a), every subdir component of every path is visited left to
-right, with each directory having a chance for exclusion before its content.
-In this way include/exclude patterns are applied recursively to the pathname of
-each node in the filesystem's tree (those inside the transfer).  The exclude
-patterns short-circuit the directory traversal stage as rsync finds the files
-to send.
    
-
    For instance, to include "/foo/bar/baz", the directories "/foo" and "/foo/bar"
-must not be excluded.  Excluding one of those parent directories prevents the
-examination of its content, cutting off rsync's recursion into those paths and
-rendering the include for "/foo/bar/baz" ineffectual (since rsync can't match
-something it never sees in the cut-off section of the directory hierarchy).
    
-
    The concept path exclusion is particularly important when using a trailing '*'
-rule.  For instance, this won't work:
    
-
    
-
    + /some/path/this-file-will-not-be-found
-+ /file-is-included
-- *
-
    
-
    
-
    This fails because the parent directory "some" is excluded by the '*' rule, so
-rsync never visits any of the files in the "some" or "some/path" directories.
-One solution is to ask for all directories in the hierarchy to be included by
-using a single rule: "+ */" (put it somewhere before the "- *" rule), and
-perhaps use the --prune-empty-dirs option.  Another solution is to add
-specific include rules for all the parent dirs that need to be visited.  For
-instance, this set of rules works fine:
    
-
    
-
    + /some/
-+ /some/path/
-+ /some/path/this-file-is-found
-+ /file-also-included
-- *
-
    
-
    
 
    Here are some examples of exclude/include matching:
    
 
      
-
    • "- *.o" would exclude all names matching *.o
      • 
-
    • "- /foo" would exclude a file (or directory) named foo in the transfer-root
-directory
      • 
-
    • "- foo/" would exclude any directory named foo
      • 
-
    • "- /foo/*/bar" would exclude any file named bar which is at two levels
-below a directory named foo in the transfer-root directory
      • 
-
    • "- /foo/**/bar" would exclude any file named bar two or more levels below a
-directory named foo in the transfer-root directory
      • 
-
    • The combination of "+ */", "+ *.c", and "- *" would include all
-directories and C source files but nothing else (see also the
---prune-empty-dirs option)
      • 
-
    • The combination of "+ foo/", "+ foo/bar.c", and "- *" would include
-only the foo directory and foo/bar.c (the foo directory must be explicitly
-included or it would be excluded by the "*")
      • 
+
    • Option -f'- *.o' would exclude all filenames ending with .o
      • 
+
    • Option -f'- /foo' would exclude a file (or directory) named foo in the
+transfer-root directory
      • 
+
    • Option -f'- foo/' would exclude any directory named foo
      • 
+
    • Option -f'- foo/*/bar' would exclude any file/dir named bar which is at two
+levels below a directory named foo (if foo is in the transfer)
      • 
+
    • Option -f'- /foo/**/bar' would exclude any file/dir named bar that was two
+or more levels below a top-level directory named foo (note that /foo/bar is
+not excluded by this)
      • 
+
    • Options -f'+ */' -f'+ *.c' -f'- *' would include all directories and .c
+source files but nothing else
      • 
+
    • Options -f'+ foo/' -f'+ foo/bar.c' -f'- *' would include only the foo
+directory and foo/bar.c (the foo directory must be explicitly included or it
+would be excluded by the "- *")
      • 
 
    
-
    The following modifiers are accepted after a "+" or "-":
    
+
    FILTER RULE MODIFIERS
    
+
    The following modifiers are accepted after an include (+) or exclude (-⁠) rule:
    
 
      
 
    • A / specifies that the include/exclude rule should be matched against the
-absolute pathname of the current item.  For example, "-/ /etc/passwd" would
-exclude the passwd file any time the transfer was sending files from the
-"/etc" directory, and "-⁠/ subdir/foo" would always exclude "foo" when it is
-in a dir named "subdir", even if "foo" is at the root of the current
+absolute pathname of the current item.  For example, -f'-/ /etc/passwd'
+would exclude the passwd file any time the transfer was sending files from
+the "/etc" directory, and "-⁠/ subdir/foo" would always exclude "foo" when it
+is in a dir named "subdir", even if "foo" is at the root of the current
 transfer.
      • 
 
    • A ! specifies that the include/exclude should take effect if the pattern
-fails to match.  For instance, "-! */" would exclude all non-directories.
      • 
+fails to match.  For instance, -f'-! */' would exclude all non-directories.
 
    • A C is used to indicate that all the global CVS-exclude rules should be
 inserted as excludes in place of the "-⁠C".  No arg should follow.
      • 
 
    • An s is used to indicate that the rule applies to the sending side.  When a
-rule affects the sending side, it prevents files from being transferred.  The
-default is for a rule to affect both sides unless --delete-excluded
-was specified, in which case default rules become sender-side only.  See also
-the hide (H) and show (S) rules, which are an alternate way to specify
-sending-side includes/excludes.
      • 
+rule affects the sending side, it affects what files are put into the
+sender's file list.  The default is for a rule to affect both sides unless
+--delete-excluded was specified, in which case default rules become
+sender-side only.  See also the hide (H) and show (S) rules, which are an
+alternate way to specify sending-side includes/excludes.
 
    • An r is used to indicate that the rule applies to the receiving side.  When
 a rule affects the receiving side, it prevents files from being deleted.  See
 the s modifier for more info.  See also the protect (P) and risk (R) rules,
@@ -3673,7 +3824,7 @@ operations (and is thus ignored when mat
 xattr-matching rules are specified, a default xattr filtering rule is used
 (see the --xattrs option).
      • 
 
    
-
    MERGE-FILE FILTER RULES
    
+
    MERGE-FILE FILTER RULES
    
 
    You can merge whole files into your filter rules by specifying either a merge
 (.) or a dir-merge (:) filter rule (as introduced in the FILTER RULES
 section above).
    
@@ -3801,13 +3952,13 @@ exclude rules (i.e. the default list of
 $HOME/.cvsignore, and the value of $CVSIGNORE) you should omit the -C
 command-line option and instead insert a "-⁠C" rule into your filter rules; e.g.
 "--filter=-C".
    
-
    LIST-CLEARING FILTER RULE
    
+
    LIST-CLEARING FILTER RULE
    
 
    You can clear the current include/exclude list by using the "!" filter rule (as
 introduced in the FILTER RULES section above).  The "current" list is either
 the global list of rules (if the rule is encountered while parsing the filter
 options) or a set of per-directory rules (which are inherited in their own
 sub-list, so a subdirectory can use this to clear out the parent's rules).
    
-
    ANCHORING INCLUDE/EXCLUDE PATTERNS
    
+
    ANCHORING INCLUDE/EXCLUDE PATTERNS
    
 
    As mentioned earlier, global include/exclude patterns are anchored at the "root
 of the transfer" (as opposed to per-directory patterns, which are anchored at
 the merge-file's directory).  If you think of the transfer as a subtree of
@@ -3857,7 +4008,7 @@ Target file: /dest/you/bar/baz
 
    The easiest way to see what name you should filter is to just look at the
 output when using --verbose and put a / in front of the name (use the
 --dry-run option if you're not yet ready to copy any files).
    
-
    PER-DIRECTORY RULES AND DELETE
    
+
    PER-DIRECTORY RULES AND DELETE
    
 
    Without a delete option, per-directory rules are only relevant on the sending
 side, so you can feel free to exclude the merge files themselves without
 affecting the transfer.  To make this easy, the 'e' modifier adds this exclude
@@ -3903,6 +4054,32 @@ one of these commands:
    
 rsync -avFF --delete host:src/dir /dest
+

TRANSFER RULES

+

In addition to the FILTER RULES that affect the recursive file scans that +generate the file list on the sending and (when deleting) receiving sides, +there are transfer rules. These rules affect which files the generator decides +need to be transferred without the side effects of an exclude filter rule. +Transfer rules affect only files and never directories.

+

Because a transfer rule does not affect what goes into the sender's (and +receiver's) file list, it cannot have any effect on which files get deleted on +the receiving side. For example, if the file "foo" is present in the sender's +list but its size is such that it is omitted due to a transfer rule, the +receiving side does not request the file. However, its presence in the file +list means that a delete pass will not remove a matching file named "foo" on +the receiving side. On the other hand, a server-side exclude (hide) of the +file "foo" leaves the file out of the server's file list, and absent a +receiver-side exclude (protect) the receiver will remove a matching file named +"foo" if deletions are requested.

+

Given that the files are still in the sender's file list, the +--prune-empty-dirs option will not judge a directory as being empty +even if it contains only files that the transfer rules omitted.

+

Similarly, a transfer rule does not have any extra effect on which files are +deleted on the receiving side, so setting a maximum file size for the transfer +does not prevent big files from being deleted.

+

Examples of transfer rules include the default "quick check" algorithm (which +compares size & modify time), the --update option, the +--max-size option, the --ignore-non-existing option, and a +few others.

BATCH MODE

Batch mode can be used to apply the same set of updates to many identical systems. Suppose one has a tree which is replicated on a number of hosts. Now @@ -4190,7 +4367,7 @@ DAEMON for full details.

RSYNC_SHELL

This environment variable is mainly used in debug setups to set the program -to use to run the program specified by [RSYNC_CONNECT_PROG]. See +to use to run the program specified by RSYNC_CONNECT_PROG. See CONNECTING TO AN RSYNC DAEMON for full details.

@@ -4208,7 +4385,7 @@ the comments on the https://rsync.samba.org/.

VERSION

-

This manpage is current for version 3.2.4 of rsync.

+

This manpage is current for version 3.2.5 of rsync.

INTERNAL OPTIONS

The options --server and --sender are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these @@ -4239,5 +4416,5 @@ people have later contributed to it. It Davison.

Mailing lists for support and development are available at https://lists.samba.org/.

-

15 Apr 2022

+

14 Aug 2022

diff -upN a/rsyncd.conf.5 b/rsyncd.conf.5 --- a/rsyncd.conf.5 +++ b/rsyncd.conf.5 @@ -1,4 +1,4 @@ -.TH "rsyncd.conf" "5" "15 Apr 2022" "rsyncd.conf from rsync 3.2.4" "User Commands" +.TH "rsyncd.conf" "5" "14 Aug 2022" "rsyncd.conf from rsync 3.2.5" "User Commands" .\" prefix=/usr .P .SH "NAME" @@ -504,7 +504,7 @@ triple-star pattern like "\fB/secret/*** .IP The "filter" parameter takes a space-separated list of daemon filter rules, though it is smart enough to know not to split a token at an internal space -in a rule (e.g. "\fB\-\ /foo\ \-\ /bar\fP" is parsed as two rules). You may specify +in a rule (e.g. "\fB\-\ /foo\ \ \-\ /bar\fP" is parsed as two rules). You may specify one or more merge-file rules using the normal syntax. Only one "filter" parameter can apply to a given module in the config file, so put all the rules you want in a single parameter. Note that per-directory merge-file @@ -514,8 +514,8 @@ the per-dir merge files are included in requests that they be used. .IP "\fBexclude\fP" This parameter takes a space-separated list of daemon exclude patterns. As -with the client \fB\-\-exclude\fP option, patterns can be qualified with "\fB\-\fP" or -"\fB+\fP" to explicitly indicate exclude/include. Only one "exclude" parameter +with the client \fB\-\-exclude\fP option, patterns can be qualified with "\fB\-\ \fP" or +"\fB+\ \fP" to explicitly indicate exclude/include. Only one "exclude" parameter can apply to a given module. See the "filter" parameter for a description of how excluded files affect the daemon. .IP "\fBinclude\fP" @@ -759,7 +759,7 @@ made more human-readable. The 3 support human-readability to be off. Each added apostrophe increases the level (e.g. "\fB%''l\ %'b\ %f\fP"). .IP -The default log format is "\fB%o\ %h\ [%a]\ %m\ (%u)\ %f\ %l\fP", and a "\fB%t\ [%p]\fP" +The default log format is "\fB%o\ %h\ [%a]\ %m\ (%u)\ %f\ %l\fP", and a "\fB%t\ [%p]\ \fP" is always prefixed when using the "log file" parameter. (A perl script that will summarize this default log format is included in the rsync source code distribution in the "support" subdirectory: rsyncstats.) @@ -794,7 +794,7 @@ See the \fB\-\-checksum-choice\fP option .IP o %l the length of the file in bytes .IP o -%L the string "\fB\->\ SYMLINK\fP", "\fB=>\ HARDLINK\fP", or "" (where \fBSYMLINK\fP +%L the string "\fB\ \->\ SYMLINK\fP", "\fB\ =>\ HARDLINK\fP", or "" (where \fBSYMLINK\fP or \fBHARDLINK\fP is a filename) .IP o %m the module name @@ -1136,17 +1136,17 @@ SSL proxy. .SH "SSL/TLS Daemon Setup" .P When setting up an rsync daemon for access via SSL/TLS, you will need to -configure a proxy (such as haproxy or nginx) as the front-end that handles the -encryption. +configure a TCP proxy (such as haproxy or nginx) as the front-end that handles +the encryption. .P .IP o You should limit the access to the backend-rsyncd port to only allow the proxy to connect. If it is on the same host as the proxy, then configuring it to only listen on localhost is a good idea. .IP o -You should consider turning on the \fBproxy\ protocol\fP parameter if your proxy -supports sending that information. The examples below assume that this is -enabled. +You should consider turning on the \fBproxy\ protocol\fP rsync-daemon parameter if +your proxy supports sending that information. The examples below assume that +this is enabled. .P An example haproxy setup is as follows: .RS 4 @@ -1176,7 +1176,7 @@ stream { ssl_certificate_key /etc/letsencrypt/example.com/privkey.pem; proxy_pass localhost:873; - proxy_protocol on; # Requires "proxy protocol = true" + proxy_protocol on; # Requires rsyncd.conf "proxy protocol = true" proxy_timeout 1m; proxy_connect_timeout 5s; } @@ -1184,7 +1184,7 @@ stream { .fi .RE .P -.SH "EXAMPLES" +.SH "DAEMON CONFIG EXAMPLES" .P A simple rsyncd.conf file that allow anonymous rsync to a ftp area at \fB/home/ftp\fP would be: @@ -1256,7 +1256,7 @@ https://rsync.samba.org/. .P .SH "VERSION" .P -This manpage is current for version 3.2.4 of rsync. +This manpage is current for version 3.2.5 of rsync. .P .SH "CREDITS" .P @@ -1273,8 +1273,9 @@ Thanks to Karsten Thygesen for his many .P .SH "AUTHOR" .P -Rsync was written by Andrew Tridgell and Paul Mackerras. Many people have -later contributed to it. +Rsync was originally written by Andrew Tridgell and Paul Mackerras. Many +people have later contributed to it. It is currently maintained by Wayne +Davison. .P Mailing lists for support and development are available at https://lists.samba.org/. diff -upN a/rsyncd.conf.5.html b/rsyncd.conf.5.html --- a/rsyncd.conf.5.html +++ b/rsyncd.conf.5.html @@ -526,7 +526,7 @@ exclude everything in the subtree; the e triple-star pattern like "/secret/***".

The "filter" parameter takes a space-separated list of daemon filter rules, though it is smart enough to know not to split a token at an internal space -in a rule (e.g. "- /foo - /bar" is parsed as two rules). You may specify +in a rule (e.g. "- /foo - /bar" is parsed as two rules). You may specify one or more merge-file rules using the normal syntax. Only one "filter" parameter can apply to a given module in the config file, so put all the rules you want in a single parameter. Note that per-directory merge-file @@ -538,8 +538,8 @@ requests that they be used.

exclude

This parameter takes a space-separated list of daemon exclude patterns. As -with the client --exclude option, patterns can be qualified with "-" or -"+" to explicitly indicate exclude/include. Only one "exclude" parameter +with the client --exclude option, patterns can be qualified with "- " or +"+ " to explicitly indicate exclude/include. Only one "exclude" parameter can apply to a given module. See the "filter" parameter for a description of how excluded files affect the daemon.

@@ -784,7 +784,7 @@ made more human-readable. The 3 support --human-readable command-line option, though the default is for human-readability to be off. Each added apostrophe increases the level (e.g. "%''l %'b %f").

-

The default log format is "%o %h [%a] %m (%u) %f %l", and a "%t [%p]" +

The default log format is "%o %h [%a] %m (%u) %f %l", and a "%t [%p] " is always prefixed when using the "log file" parameter. (A perl script that will summarize this default log format is included in the rsync source code distribution in the "support" subdirectory: rsyncstats.)

@@ -806,7 +806,7 @@ See the --checksum-choice o
  • %h the remote host name (only available for a daemon)
  • %i an itemized list of what is being updated
  • %l the length of the file in bytes
    • -
  • %L the string "-> SYMLINK", "=> HARDLINK", or "" (where SYMLINK +
  • %L the string " -> SYMLINK", " => HARDLINK", or "" (where SYMLINK or HARDLINK is a filename)
  • %m the module name
  • %M the last-modified time of the file
    • @@ -1067,15 +1067,15 @@ authentication is provided. Use ssh as t SSL proxy.

    SSL/TLS Daemon Setup

    When setting up an rsync daemon for access via SSL/TLS, you will need to -configure a proxy (such as haproxy or nginx) as the front-end that handles the -encryption.

    +configure a TCP proxy (such as haproxy or nginx) as the front-end that handles +the encryption.

    • You should limit the access to the backend-rsyncd port to only allow the proxy to connect. If it is on the same host as the proxy, then configuring it to only listen on localhost is a good idea.
      • -
    • You should consider turning on the proxy protocol parameter if your proxy -supports sending that information. The examples below assume that this is -enabled.
      • +
    • You should consider turning on the proxy protocol rsync-daemon parameter if +your proxy supports sending that information. The examples below assume that +this is enabled.

    An example haproxy setup is as follows:

    @@ -1100,14 +1100,14 @@ backend be_rsync ssl_certificate_key /etc/letsencrypt/example.com/privkey.pem; proxy_pass localhost:873; - proxy_protocol on; # Requires "proxy protocol = true" + proxy_protocol on; # Requires rsyncd.conf "proxy protocol = true" proxy_timeout 1m; proxy_connect_timeout 5s; } }
    -

    EXAMPLES

    +

    DAEMON CONFIG EXAMPLES

    A simple rsyncd.conf file that allow anonymous rsync to a ftp area at /home/ftp would be:

    @@ -1162,7 +1162,7 @@ susan:herpass

    Please report bugs! The rsync bug tracking system is online at https://rsync.samba.org/.

    VERSION

    -

    This manpage is current for version 3.2.4 of rsync.

    +

    This manpage is current for version 3.2.5 of rsync.

    CREDITS

    Rsync is distributed under the GNU General Public License. See the file COPYING for details.

    @@ -1172,9 +1172,10 @@ project is AUTHOR -

    Rsync was written by Andrew Tridgell and Paul Mackerras. Many people have -later contributed to it.

    +

    Rsync was originally written by Andrew Tridgell and Paul Mackerras. Many +people have later contributed to it. It is currently maintained by Wayne +Davison.

    Mailing lists for support and development are available at https://lists.samba.org/.

    -

    15 Apr 2022

    +

    14 Aug 2022