The --apparent-size option to 'du' was added in
coreutils-4.5.8 (2003). FreeBSD 8.0 (2009) added the same functionality
under the short option -A. This long option previously had no short
option, so this patch adds -A to be compatible with FreeBSD.
* NEWS: Mention the new short option.
* doc/coreutils.texi: Document the short option.
* src/du.c (usage): Likewise.
(APPARENT_SIZE_OPTION): Remove definition.
(EXCLUDE_OPTION): Define to CHAR_MAX + 1.
(long_options): Use the -A short option for --apparent-size.
(main): Likewise.
* tests/du/apparent.sh: Test that '-b', '-A -B 1', and
'--apparent-size --block-size 1' function the same.
* tests/du/bigtime.sh: At least on Linux, the ext4 filesystem
doesn't support such large timestamp, while tmpfs does. Try a bit
harder to look for a filesystem with large timestamp support.
* tests/du/move-dir-while-traversing.sh: Expand the work to avoid
a false failure where du completes before the directory is moved.
Also expand the timeout to our more standard 10s to avoid the
"directory mover" being killed before du processes the directory.
This doesn't perceptibly impact the run time of the test.
Reported by Bruno Haible on a CentOS 7 system.
* cfg.mk (sc_standard_outputs): Add a grep command for source files.
* src/du.c (main): Use standard input instead of stdin, standard output
instead of stdout, and standard error instead of stderr in messages.
* src/nohup.c (main): Likewise.
* src/sort.c (main): Likewise.
* src/split.c (main): Likewise.
* src/stdbuf.c (main): Likewise.
* src/wc.c (main): Likewise.
* tests/du/files0-from.pl (@Tests): Adjust test case to new messages.
* tests/sort/sort-files0-from.pl: Likewise.
* tests/wc/wc-files0-from.pl: Likewise.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
* tests/du/threshold.sh: Directories are assumed to be
of size 0 with --apparent since commit v9.1-187-g110bcd283
so remove --apparent cases from this test.
Problem reported by Christoph Anton Mitterer (Bug#61884).
* src/du.c (process_file): When counting apparent sizes, count
only usable st_size members.
* tests/du/apparent.sh: New file.
* tests/local.mk (all_root_tests): Add it.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
In documentation and comments, don’t assume that secondary storage
devices are disk devices. Similarly, don’t assume that main memory
uses magnetic cores, which became obsolete in the 1970s.
* src/du.c (usage):
* src/ls.c (usage):
* src/shred.c (usage): Reword to avoid “disk” in usage messages.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* tests/cp/cp-a-selinux.sh: Use 'skip_' rather than the probably
undefined 'skip'.
* tests/du/2g.sh: Likewise.
* tests/install/install-Z-selinux.sh: Likewise.
* tests/misc/chcon.sh: Likewise.
* tests/misc/selinux.sh: Likewise.
* tests/mkdir/restorecon.sh: Likewise.
* cfg.mk (sc_prohibit-skip): A new syntax check to catch the issue.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* tests/rm/dangling-symlink.sh: The effective delay was only
1.5s. So delay before the operation, and increase the iteration
count by 1 to increase the delay to 6.3s. This failure was
noticed once on a FreeBSD 10.2 x86_64 virtual machine.
* tests/dd/stats.sh: Remove quotes on sleep argument
to ensure arguments are accumulated appropriately.
* tests/du/move-dir-while-traversing.sh: sleep before operation,
to increase the effective delay from 1.6s to 3.2s.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Sleep first so that the effective delay
matches the commented value.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
programs may not be built due to missing system dependencies,
or any program can be excluded at configure time with
--enable-no-install-program. So ensure we're not testing the
system versions in these cases.
* init.cfg (print_ver_): Call require_built_ first.
* tests/misc/tty-eof.pl: Skip programs not built.
* tests/Coreutils.pm (run_tests): Likewise.
* tests/misc/ls-misc.pl: Use 'env test' rather than abs path.
* tests/misc/test-diag.pl: Likewise.
* tests/local.mk: Adjust include order for dependencies.
* tests/misc/arch.sh: Remove redundant calls to require_built_.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/groups-dash.sh: Likewise.
* tests/misc/groups-version.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/cp/acl.sh: Remove problematic call to print_ver_ [gs]etfacl.
* tests/mv/acl.sh: Likewise.
* cfg.mk (sc_env_test_dependencies): A new syntax check to enforce
specifying dependencies with print_ver_ for programs
specified through the env command.
* du/bigtime.sh: Add new print_ver_ dependencies.
* du/max-depth.sh: Likewise.
* dd/ascii.sh: Likewise.
* tests/ls/capability.sh: Likewise.
* tests/ls/root-rel-symlink-color.sh: Likewise.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/readlink-fp-loop.sh: Likewise.
* tests/misc/sort-debug-keys.sh: Likewise.
* tests/readlink/can-e.sh: Likewise.
* tests/readlink/can-f.sh: Likewise.
* tests/readlink/can-m.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/inotify-race2.sh: Likewise.
* tests/touch/no-create-missing.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
* tests/misc/printenv.sh: Tweak to avoid syntax check trigger.
* tests/misc/help-version.sh: Likewise.
* tests/misc/yes.sh: Likewise.
* tests/misc/printf-quote.sh: Use previously unused $prog.
* configure.ac (EXTRA_MANS): Add $gl_no_install_prog to the list
so that check-x-vs-1 syntax check is satisfied.
A side effect of this cleanup is we no longer
depend on our own kill command being built.
* init.cfg (require_trap_signame_): A new function to verify
that the shell supports specifying traps by signal name.
(require_kill_group_): A new function to ensure the shell
supports sending a signal to a group.
* tests/du/move-dir-while-traversing.sh: Ensure trap supports
signal names.
* tests/misc/stty-invalid.sh: Likewise.
* tests/misc/stty-pairs.sh: Likewise.
* tests/misc/stty-row-col.sh: Likewise.
* tests/misc/stty.sh: Likewise.
* tests/misc/sort-compress.sh: Likewise. Also simplify trap call.
* tests/install/trap.sh: Likewise.
* tests/misc/timeout.sh: Likewise.
* tests/dd/stats.sh: Likewise. Also use default kill command.
* tests/misc/timeout-group.sh: Likewise.
Quote file names using the "shell-escape" or "shell-escape-always"
methods, which quote as appropriate for most shells,
and better support copy and paste of presented names.
The "always" variant is used when the file name is
embedded in an error message with surrounding spaces.
* cfg.mk (sc_error_shell_quotes): A new syntax check rule
to suggest quotef() where appropriate.
(sc_error_shell_always_quotes): Likewise for quoteaf().
* src/system.h (quotef): A new define to apply shell quoting
when needed. I.E. when shell character or ':' is present.
(quoteaf): Likewise, but always quote.
* src/*.c: Use quotef() and quoteaf() rather than quote()
where appropriate.
* tests/: Adjust accordingly.
Related to commit v8.24-61-g6796698 this provides
more consistent quoting, as quotearg_colon() defaults
to "literal" quoting by default, while quote()
provides appropriate quoting for diagnostics by default.
* gl/modules/randread: Depend on quote module rather than quotearg.
* gl/lib/randread.c: Used quote() not quotearg_colon().
* src/: Likewise.
* src/shred.c: Likewise. Also avoid unnecessary quoting
introducing overhead when wiping names.
* cfg.mk: Relax the matching expression to allow
"qname" variables as used in shred.c to satisfy the check.
* tests/: Adjust accordingly.
These strings are often file names or other user specified
parameters, which can give confusing errors in
the presence of unexpected characters for example.
* cfg.mk (sc_error_quotes): A new syntax check rule.
* src/*.c: Wrap error() string arguments with quote().
* tests/: Adjust accordingly.
* NEWS: Mention the improvement.
* tests/dd/sparse.sh: Sync files before checking allocations,
which may be done asynchronously on NFS and BTRFS at least.
Also mark this test as very expensive on remote file systems.
* tests/du/2g.sh: Likewise, also use fallocate if available
to efficiently allocate the large file, otherwise skip
on remote file systems.
* tests/tail-2/inotify-rotate-resources.sh: Use the more
standard is_local_dir_() to check remoteness.
* tests/cp/fiemap-empty.sh: Comment on the sync issue
for this currerntly unused test.
Fixes http://bugs.gnu.org/20570
Many tests use `program ... && fail=1` to ensure expected
error situations are indicated. However that would mask
an unexpected exit (like a crash). Therefore explicitly
check the expected exit code.
Note where error messages are also verified, the extra
protection is not added.
* tests/init.sh (returns_): A new helper function to
check the return code of a command, and used
throughout the tests.
* cfg.mk (sc_prohibit_and_fail_1): Add a syntax check
to avoid new instances of this issue.
Run "make update-copyright" and then...
* tests/sample-test: Adjust to use the single most recent year.
* tests/du/bind-mount-dir-cycle-v2.sh: Fix case in copyright message,
so that year is updated automatically in future.
This patch fixes the handling of sub-bind-mount cycles which are
incorrectly detected as the file system errors. If you bind mount the
directory 'a' to its subdirectory 'a/b/c' and then run 'du a/b' you
will get the circular dependency warning even though nothing is wrong
with the file system. This happens because the first directory that is
traversed twice in this case is not a bind mount but a child of bind
mount. The solution is to traverse all the directories in the cycle
that fts detected and check whether they are not a (bind) mount.
* src/du.c (mount_point_in_fts_cycle): New function that checks whether
any of the directories in the cycle that fts detected is a mount point.
* src/du.c (process_file): Update the function to use the new function
that looks up all the directories in the fts cycle instead of only the
last one.
* tests/du/bind-mount-dir-cycle-v2.sh: New test case that exhibits the
described behavior.
* tests/local.mk: Reference the new root test.
* NEWS: Mention the bug fix.
On XFS, when creating the ~2G test file 'big' in a for-loop by
appending 20M each time, the file ends up using ~4G - visible in
'st_blocks'. The unused space would be reclaimed later.
This feature is called "speculative preallocation" which aims at
avoiding fragmentation.
According to the XFS FAQ [1], there are two particular aspects of
XFS speculative preallocation that are triggering this:
1. "Applications that repeatedly trigger preallocation and reclaim
cycles [after file close] can cause fragmentation.
Therefore, this pattern is detected and causes the preallocation
to persist beyond the lifecycle of the file descriptor."
2. "Preallocation sizes grow as files grow larger."
[1] http://xfs.org/index.php/XFS_FAQ
Avoid one of the above by only doing a single close (reclaim cycle).
* tests/du/2g.sh: Similar to the fix for a dd test (see commit
v8.22-65-g7c03fe2), avoid speculative preallocation by creating
the 'big' file in one go instead of appending to it in the loop.
Remove debugging statements as the output with 'set -x' is
sufficient nowadays.
Avoid complicated and error-prone parsing of df's output via
sed(1), cut(1), etc., and instead use df's more modern --output
option.
* src/ioblksize.h (in a comment): Simplify the extraction of the
device name of the mounted file system from df's output.
* tests/dd/skip-seek-past-dev.sh: Likewise.
* tests/du/2g.sh: Likewise for the 'avail' column here.
Also avoid the deprecated use of "tail -NUM".
* tests/misc/stat-mount.sh: While at it, remove the determination
of the mount point of "." via df(1) plus sed(1) as it is unused
since commit v8.5-159-gf57cb37 anyway. Instead, improve this test
by verifying that the output of "stat -c%m ." at least starts with
a slash '/'.
* src/du.c (process_file): Treat cycles due to bind mounts
like cycles due to following symlinks.
* tests/du/bind-mount-dir-cycle.sh: Adjust accordingly.
* NEWS: Mention the change in behavior.
Reported at http://bugzilla.redhat.com/836557
Run "make update-copyright", but then also run this,
perl -pi -e 's/2\d\d\d-//' tests/sample-test
to make that one script use the single most recent year number.
Prompted by the continuous integration build failure at:
http://hydra.nixos.org/build/5584121
du(1) uses the first file object of the two test files linked to the
same inode, 'd/f' and 'd/h', whatever the system returns first.
Use 'd/f' in both the expected and the actual output.
* test/du/inodes.sh: Change the expected output as described above
when returning the --all directory entries (without -l). Also replace
the name of the hardlink 'd/h' by 'd/f' in the actual output.
* test/du/inodes.sh: In the cases where compare() fails, that function
would show the unified diff automatically. Therefore, remove the
excess "cat out".
In the cases where expecting an empty file, use compare() again
rather than the simpler "test -s" because possible error reports
will then include the file's content for the same reason as above.
Prompted by the continuous integration build failure at:
http://hydra.nixos.org/build/5582213
* test/du/inodes.sh: Due to undefined order in returned directory
entries, the expected output might not match, so sort both expected
and actual output when returning --all directory entries.
Also use a simpler test for ensuring no errors are output.
