This was detected with ASAN, but can also be seen without ASAN with:
$ tac - - <&-
tac: standard input: read error: Bad file descriptor
*** Error in `tac': malloc(): memory corruption: 0x...
* src/tac.c (copy_to_temp): Don't close our output stream on
(possibly transient) output error, or on input error.
(temp_stream): clearerr() on the stream about to be reused,
to ensure future stream use is not impacted by transient errors.
* tests/misc/tac-2-nonseekable.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/25041
Reproduced under UBSAN with `tail -f <&-` giving:
tail.c:2220:18: runtime error: load of value 190,
which is not a valid value for type ‘_Bool'
* src/tail.c (tail_file): Ensure f->ignore is initialized
in all cases where we can't tail the specified file.
* tests/tail-2/follow-stdin.sh: Add a test case which
checks stderr has no UBSAN warnings.
Fixes http://bugs.gnu.org/25041
* NEWS: Mention in improvements about the workaround for
the glibc issue with closed stdin, and the new supported file systems.
* tests/misc/b2sum.sh: Spelling fix.
* tests/ls/quote-align.sh: Remove "total" line
which can vary per file system depending on allocation.
Reported by Assaf Gordon on OpenSolaris (5.11/5.10).
This was detected with:
echo a > a; pr "-S$(printf "\t\t\t")" a -m a > /dev/null
Resulting in ASAN triggering:
====================================================
ERROR: AddressSanitizer: global-buffer-overflow
READ of size 1 at 0x00000041b622 thread T0
#0 0x40506a in print_sep_string ../src/pr.c:2241
#1 0x407ec4 in read_line ../src/pr.c:2493
#2 0x40985c in print_page ../src/pr.c:1802
#3 0x40985c in print_files ../src/pr.c:1618
#4 0x4036e0 in main ../src/pr.c:1136
* src/pr.c (init_parameters): Ensure we only override the
specified separator when it's a single tab, thus matching
the calculated separator length.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
* src/ptx.c (fix_output_parameters): Ensure line_width doesn't
go negative, which can happen when the --width is less
than the --gap-size.
* tests/misc/ptx-overrun.sh: Add a test case that triggers
with ASAN. (Note the longer filename is needed to trigger).
Fixes http://bugs.gnu.org/25011
* tests/rm/rm-readdir-fail.sh: ASAN correctly indicated
that fts was writing to freed memory. This was because
we reused a single dirent in our readdir() test wrapper.
Since fts was deallocating those dirents, we now get
a new dirent for each call to our readdir wrapper.
ASAN reported this error for: split -n2/3 /dev/null
ERROR: AddressSanitizer: negative-size-param: (size=-1)
#0 0x7f0d4c36951d in __asan_memmove (/lib64/libasan.so.2+0x8d51d)
#1 0x404e06 in memmove /usr/include/bits/string3.h:59
#2 0x404e06 in bytes_chunk_extract src/split.c:988
#3 0x404e06 in main src/split.c:1626
Specifically there would be invalid memory access
and subsequent processing if the chunk to be extracted
was beyond the initial amount read from file (which is
currently capped at 128KiB). This issue is not in a
released version, only being introduced in commit v8.25-4-g62e7af0
* src/split.c (bytes_chunk_extract): The initial_read != SIZE_MAX
should have been combined with && rather than ||, but also this
condition is always true in this function so remove entirely.
* tests/split/b-chunk.sh: Add a test case.
Fixes http://bugs.gnu.org/25003
* src/comm.c (total_option): Add bool variable for the new option.
(TOTAL_OPTION): Add enum value.
(long_options): Add array element for the new option.
(usage): Document the new option here.
(compare_files): Count the lines in total[3], and output the summary at
the end.
(main): Accept the new option.
* doc/coreutils.texi (comm invocation): Document it.
* tests/misc/comm.pl: Test it. While at it, improve the test data
to have 1 unique line in the first file, 2 unique lines in the second
file, and 3 common lines.
* NEWS (New Features): Mention the new option.
Fixes http://bugs.gnu.org/24929
* tests/misc/printf-quote.sh: FreeBSD 11 was seen to treat
\u0378 as a printable character. Therefore change to
using the \u0081 C1 control character. We use the UTF-8
representation because our printf implementation explicitly
disallows \u0081 as input.
Reported by Assaf Gordon
On BSD /bin/sh it was seen that unexported env vars passed to
returns_() would not be propagated to the wrapped command.
* cfg.mk (sc_prohibit_env_returns): Add a syntax check to disallow.
* tests/misc/csplit-io-err.sh: Rearrange to export vars in a subshell.
* tests/rm/rm-readdir-fail.sh: Likewise.
* tests/misc/nohup.sh: Export and unset vars around returns_.
* tests/misc/printenv.sh: Likewise.
Reported by Assaf Gordon
This provides better alignment when some names are quoted,
which also provides better indication that quotes are not
part of the name.
* src/ls.c (align_variable_outer_quotes): A new variable
set when ls is aligning columns (not using -m, non-zero -w),
and has a variable quoting style (shell, shell-escape, c-maybe).
(quote_name_buf): Writes to buffer rather than FILE,
taking care to avoid data copying if possible. Refactored from...
(quote_name): ...here. This now manages the buffer passed
to quote_name_buf() and outputs the padding, colors and name
in the appropriate order, while managing the --dired offsets.
(get_color_indicator): A new function to return the color sequence,
refactored from...
(print_color_indicator): ...here. This now simply outputs.
(print_dir): Refactor common parts to quote_name().
(clear_files): Reset the flag indicating at least one
file is quoted in the current directory.
(needs_quoting): A new function to indicate at the scan stage
whether a name needs quoting. Called from...
(gobble_file): ...here, until we find the first quoted file.
(print_name_with_quoting): Mostly refactored to quote_name().
* tests/ls/quote-align.sh: A new test for various output formats.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Note we don't support the --algorithm option of the
b2sum command in the external BLAKE2 project, as that
was deemed too confusing for users.
"BLAKE2b" was chosen as the default algorithm to use,
which is single threaded but performs well on 64 bit.
* src/blake2: CC0 source copied from external project.
* cfg.mk[VC_LIST_ALWAYS_EXCLUDE_REGEX]: Exclude blake2/
from syntax checks, make update-copyright, etc.
* src/local.mk: Reference the sources for b2sum,
and set the compilation flags.
* doc/coreutils.texi (b2sum invocation): Reference the
md5sum invocation node, and add descriptions of -l.
* tests/misc/b2sum.sh: Add new test.
* tests/local.mk: Reference new test.
* AUTHORS: Add new binary.
* README: Likewise.
* build-aux/gen-lists-of-programs.sh: Likewise.
* man/.gitignore: Likewise.
* scripts/git-hooks/commit-msg: Likewise.
* man/b2sum.x: New man page template.
* man/local.mk: Reference new template.
* src/.gitignore: Ignore new binaries.
* src/blake2/.gitignore: Ignore new build atrifacts.
* src/md5sum.c (usage): Describe the new -l option.
* NEWS: Mention the new program.
* src/copy.h (cp_options): Add a new flag for install(1).
* src/copy.c (copy_internal): For cp, also output remediation
advice which also indicates why directories aren't copied by default.
The message is unchanged for install(1).
* src/cp.c (cp_option_init): Init install_mode to false.
* src/mv.c (cp_option_init): Likewise.
* src/install.c (cp_option_init): Init install_mode to true.
* tests/install/basic-1.sh: Add a test case.
* tests/cp/link-deref.sh: Adjust test case.
Fixes http://bugs.gnu.org/24958
* tests/misc/sort-h-thousands-sep.sh: The sv_SE locale
may be available while sv_SE.utf8 is not. Therefore
test and use the same locale, now definfed in a variable.
Reported by Jim Meyering.
The previous "returns_"-using change failed to convert many
uses of "$?". Convert all but two of the remaining ones.
* tests/ls/stat-vs-dirent.sh: Likewise.
* tests/misc/head-write-error.sh: Likewise.
* tests/misc/nice.sh: Likewise.
* tests/misc/nohup.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/sync.sh: Likewise.
* tests/tail-2/pid.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
Thanks to Bernhard Volker for spotting this.
* src/tail.c (tail_file): On failure to open a file,
set ignore=true when --retry is not specified.
* tests/tail-2/assert-2.sh: Adjust to the new behavior.
* tests/tail-2/retry.sh: Add a test case. Also change
from `tail ... && fail=1` to the more robust `returns_ 1 ...`
construct which detects segfaults etc.
* NEWS: Document the fix.
* src/tail.c (tail_forever): The BLOCKING optimization is only
enabled for non regular files (which can't be truncated), so ensure
we don't enable that unless we've a valid file descriptor.
* tests/tail-2/retry.sh: Add a test case.
* NEWS: Mention the bug fix.
* src/tail.c (ignore_pipe_or_fifo): Mark the descriptor as -1
for pipes so that any_live_files() detects correctly that
the entry is no longer live.
* tests/tail-2/pipe-f.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/24903 which was detected
using Symbolic Execution techniques developed in
the course of the SYMBIOSYS research project at
COMSYS, RWTH Aachen University.
* doc/coreutils.texi(cut invocation): Give a more accurate description
of cut field handling limitations.
* tests/misc/stat-fmt.sh: Fix the test header to be more general.
* tests/tail-2/retry.sh: Spellings.
Incorporating these coreutils specific changes:
backupfile: initialize default suffix within the implementation
strftime,strptime: support %q to represent the quarter
Also sync these copies with gnulib:
* bootstrap: Now supports AIX.
* gnulib: Update to the latest.
* tests/init.sh: dash(1) is now disabled.
* doc/coreutils.texi (stat invocation): Describe the
QUOTING_STYLE values now supported.
* src/stat.c (getenv_quoting_style): A new function called
from main, that sets the default quoting style for quotearg.
(main): Call getenv_quoting_style() when %N specified.
* tests/misc/stat-fmt.sh: Add a test case.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/23422
* src/dd.c (parse_integer): Suggest to use "00x" instead of "0x",
which is significant for the "count", "seek", and "skip" operands.
* tests/dd/misc.sh: Add a test case.
Fixes http://bugs.gnu.org/24874
* NEWS: Mention the fix.
* src/md5sum.c (digest_file): Add a new MISSING parameter to
return whether the file was missing, separately from the digest.
* tests/misc/md5sum.pl: Add a test case.
Fixes http://bugs.gnu.org/24795
* src/rm.c (main): Ensure the full --no-preserve-root
option is specified, rather than allowing --n etc.
* tests/rm/r-root.sh: Add a test case.
* NEWS: Mention the change in behavior.
Improved by Jim Meyering.
Fixes http://bugs.gnu.org/24604
which was not the case when inotify was not available.
* src/tail.c (any_live_files): Simplify, since the IGNORE
flag is now only set when a file should be ignored indefinitely.
(recheck): Only output the "giving up on name" message
when that's actually the case. Only set the IGNORE flag
when ignoring a file indefinitely.
(tail_file): Likewise.
* tests/tail-2/retry.sh: Add a test case. Also run
all existing test cases with and without inotify.
NEWS: Mention the fix.
THANKS.in: Add the reporter.
Fixes http://bugs.gnu.org/24495 which was detected
using Symbolic Execution techniques developed in
the course of the SYMBIOSYS research project at
COMSYS, RWTH Aachen University.
The old code mishandled --time-spec='+%%b', as it misinterpreted
the '%b' as being the month abbreviation. Also, it mishandled
the extremely unlikely case of a month abbreviation containing '%'.
The performance part of this patch sped up 'ls' by about 1% on my
little benchmark of 'ls -lR' on the source directory in the
en_US.UTF-8 locale (Fedora 24 x86-64).
* NEWS: Document the bug fix.
* src/ls.c (first_percent_b, abformat_init): New static functions.
(ABFORMAT_SIZE): New constant.
(use_abformat): New static var.
(abmon, required_mon_width): Remove these static vars.
(abmon_init): Now accepts a pointer to abmon, and returns a boolean.
All callers changed. Reject month abbrs containing '%', as these
would mess up strftime. Simplify mbsalign result checking,
since (size_t) -1 exceeds ABFORMAT_SIZE.
(abformat_init, align_nstrftime): Precompute all 24 formats at
startup, rather than computing a format for each time stamp.
(decode_switches): Call abformat_init instead of abmon_init.
(align_nstrftime): Accept recentness bool instead of format.
All callers changed.
* tests/misc/time-style.sh: Test for format with '%%b'.
* src/cp.c (make_dir_parents_private): Use default permissions for
created directories when --no-preserve=mode is specified.
* tests/cp/cp-parents.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/24251
* src/expand.c, src/unexpand.c: Move global variables from here...
* src/expand-common.h, src/expand-common.c: ... to here.
* src/expand.c, src/unexpand.c: (parse_tab_stops, validate_tab_stops,
next_file): Move identical functions to new module.
(add_tab_stop): Move to new module, including additional code from
'unexpand' (keeping max_column_width) which will have no effect in when
used in 'expand'. Refactor common next-column calculation code
into a new function 'get_next_tab_column'.
* src/local.mk: (src_expand_SOURCES, src_unexpand_SOURCES): Add
'expand-common.c'; (noinst_HEADERS): Add 'expand-common.h'.
* po/POTFILES.in: Add 'expand-common.c'.
* tests/misc/expand.pl: Add more tests.
* tests/misc/unexpand.pl: Likewise.
* TODO: Move conclusions to above test after investigation.
* src/sort.c (traverse_raw_number): Accept thousands separator only
if it is immediately followed by a digit.
* tests/misc/sort-h-thousands-sep.sh: Cover the fix for this bug.
Suggested by Pádraig Brady in http://bugs.gnu.org/24015
* src/sort.c (traverse_raw_number): Allow to skip only one occurrence
of thousands_sep to avoid finding the unit in the next column in case
thousands_sep matches as blank and is used as column delimiter.
* tests/misc/sort-h-thousands-sep.sh: Add regression test for this bug.
* tests/local.mk: Reference the test.
* NEWS: Mention the bug fix.
Reported at https://bugzilla.redhat.com/1355780
Fixes http://bugs.gnu.org/24015
* doc/coreutils.texi (install invocation): Update -Z documentation.
* src/install.c (make_ancestor): Set default security context before
calling mkdir() if the -Z option is given.
(process_dir): Call restorecon() on the destination directory if the
-Z option is given.
(usage): Update -Z documentation.
* tests/install/install-Z-selinux.sh: A new test for 'install -Z -D'
and 'install -Z -d' based on tests/mkdir/restorecon.sh.
* tests/local.mk: Reference the test.
* NEWS: Mention the improvement.
Reported at https://bugzilla.redhat.com/1339135
Fixes http://bugs.gnu.org/23868
* tests/rm/rm-readdir-fail.sh: Since we use the returned dirent
from the readdir wrapper it must be the correct type and not
just cast. Therefore setup so that we only have to define a
wrapper for readdir(), which works appropriately on 32 and 64 bit.
Issue reported by Bernhard Voelker, where rm was seen to invoke
rmdir() on invalid file names.
Follow-up to commit c92585b1: improve epipe test script, with
suggestions from Bernhard Voelker and Pádraig Brady.
see: http://lists.gnu.org/archive/html/coreutils/2016-04/msg00067.html .
* tests/misc/seq-epipe.sh: Avoid too many subshells,
stricter validation of errors and output from seq,
skip if trapping SIGPIPE is not supported.
* tests/rm/rm-readdir-fail.sh: A new test to simulate readdir()
failing immediately or after returning a few entries, and verifying
that rm does the appropriate thing.
This was initially reported at:
http://bugzilla.opensuse.org/show_bug.cgi?id=984910
where it was mentioned that readdir() may fail
when an NFS server has a poor readdir cookie implementation.
* gnulib: Update to latest.
* NEWS: Specifically mention the fts readdir() fix
and reindent to standard indentation.
* tests/init.sh: Update from gnulib.
Print warning regardless of locale, avoid warning if key is zero width.
Problem reported by Karl Berry in http://bugs.gnu.org/23665 .
* src/sort.c: (key_warnings): change conditions for 'leading spaces'
warning.
* tests/misc/sort-debug-warn.sh: adjust tests accordingly.
Easier troubleshooting of individual 'sort --debug' messages.
Discussed in http://bugs.gnu.org/23665 .
* tests/misc/sort-debug-warn.sh: add progress number before each sort
invocation.
Ensure I/O errors are detected (and terminate seq), preventing seq
from infloop (or running for long time with a large
range) upon write errors or ignored SIGPIPE. Examples:
seq 1 inf > /dev/full (seq_fast)
seq 1.1 0.1 inf >/dev/full (print_numbers)
* src/seq.c (io_error): A new function to diagnose appropriate
stdio errors and exit the program with failure status.
(seq_fast, print_numbers): Explicitly check for write errors
and terminate the program with diagnostic.
* tests/misc/seq-io-errors.sh: Test error detection with /dev/full.
* tests/misc/seq-epipe.sh: Test error detection with broken pipes.
* tests/local.mk: Add new tests.
* NEWS: Mention the fix.
