* gnulib: Update to the latest gnulib to pull in the fix
(v1.0-2406-g89f63027de) to allocate the $PATH processing memory
before the vfork call, which is required on ppc and sparc.
* tests/install/basic-1.sh: Ensure posix_spawnp() suports a large $PATH,
which needs careful handling with vfork() as detailed in gnulib.
It was seen that gnulib's read_file_system_list may use fopen instead
of open. Adjust the df(1) tests to replace both library functions.
* tests/df/no-mtab-status.sh: Change the shared library code invoked
via LD_PRELOAD to override both fopen and open. While at it, perform
varargs processing only when path is not "/proc/self/mountinfo".
* tests/df/skip-duplicates.sh: Likewise.
Two df(1) tests were skipped (since commit ee367bd38d), because
the build of the shared library in those tests failed.
+ gcc -Wall -shared --std=gnu99 -fPIC -O2 k.c -o k.so -ldl
k.c: In function 'open':
k.c:37:7: error: implicit declaration of function 'streq'; did you \
mean 'strsep'? [-Wimplicit-function-declaration]
37 | if (streq (path, "/proc/self/mountinfo"))
| ^~~~~
| strsep
Gnulib streq is not available in the tests.
* tests/df/no-mtab-status.sh: Replace "streq" by "0==strcmp" in the
shared library source.
* tests/df/skip-duplicates.sh: Likewise.
This improvement is due to changes to Gnulib's fprintftime module.
* NEWS: Mention the improvement.
* src/getlimits.c (OFF64_T_MAX, OFF64_T_MIN): New macros.
(main): Print them.
* tests/misc/write-errors.sh: Call getlimits_. Add a date invocation.
* src/copy-file-data.c (infer_scantype): Fall back to a plain copy
if SEEK_HOLE indicates non-sparse, as zero copy avoids copy offload.
This was seen with transparently compressed files on OpenZFS.
* tests/cp/sparse-perf.sh: Add a test case even though it might
only trigger on compressed file systems that don't support reflink.
* NEWS: Mention the bug fix.
Addresses https://github.com/coreutils/coreutils/issues/122
* src/sort.c (main): Ignore SIGPIPE so we've more control over
how we handle for stdout and compression programs.
(sort_die): Handle EPIPE from stdout and mimic a standard SIGPIPE,
otherwise reverting to a standard exit(SORT_FAILURE);
* tests/sort/sort-compress-proc.sh: Add a test case.
* NEWS: Mention the bug fix.
* tests/date/resolution.sh: Fix comparison on systems with less than
nano second reslution, where we use sed to discard the redundant
trailing zeros output by date --resolution.
Reported by Bruno Haible on macOS.
* NEWS: Mention the improvement. Mention that 'sort' will continue
without compressing temporary files if the program specified by
--compress-program cannot be executed.
* doc/coreutils.texi (sort invocation): Document the behavior when the
program specified by --compress-program cannot be executed.
* src/sort.c: Include spawn.h.
(MAX_FORK_TRIES_COMPRESS, MAX_FORK_TRIES_DECOMPRESS): Remove definition.
(MAX_TRIES_COMPRESS, MAX_TRIES_DECOMPRESS): New definitions based on
MAX_FORK_TRIES_COMPRESS and MAX_FORK_TRIES_DECOMPRESS.
(async_safe_die): Remove function.
(posix_spawn_file_actions_move_fd): New function.
(pipe_fork): Remove function.
(pipe_child): New function based on pipe_fork. Return an error number
instead of a pid. Use posix_spawnp instead of calling fork and expecting
the caller to exec.
(maybe_create_temp): Call pipe_child instead of pipe_fork. Print a
warning to standard error if --compress-program cannot be executed and
the error is different than the previous call. Remove code for the child
process.
(open_temp): Remove code for the child process. Improve error message.
* tests/sort/sort-compress.sh: Add a test case for when the program
specified by --compress-program does not exist.
* NEWS: Mention the improvement.
* src/fmt.c (put_line): Exit if any error writing line.
(flush_paragraph): Exit if any error writing buffer.
* tests/misc/write-errors.sh: Enable the (flush_paragraph) test case,
and add another to check the put_line() case.
* src/numfmt.c (process line): Inspect the stdio error state when
outputting each line so that we don't have to check each output function
but do eventually exit upon write error, while also remaining buffered.
(main): Also check when outputting a header for the edge case
of very long headers.
* tests/misc/write-errors.sh: Enable the numfmt test case.
* NEWS: Mention the improvement, and reorganize all numfmt improvements.
* tests/numfmt/mb-non-utf8.sh: Test GB18030 delimiter search.
* tests/local.mk: Reference the new test, and move
the existing numfmt.pl test from tests/misc to tests/numfmt.
* bootstrap.conf: Depend on mbsstr() to robustly search for a
multi-byte delimiter character (string) within a multi-byte string.
* src/numfmt.c (main): Accept a valid multi-byte delimiter character.
(next_field): Adjust delimiter search from single byte
to multi-byte aware. Use mbsstr to find the first match.
* tests/misc/numfmt.pl: Add test case.
* NEWS: Mention the improvement.
* src/numfmt.c (process_line): Restore byte overwritten with NUL,
as it may be part of a multi-byte blank.
(process_suffixed_number): Skip multi-byte blanks,
and correctly determine width with mbswidth().
(parse_format_string): Use c_isblank() to explicitly
indicate that's all the format spec supports.
* tests/misc/numfmt.pl: Add test cases.
* NEWS: Mention the bug fix.
Output, accept, or disallow a string between the number and unit
as recommended in <https://physics.nist.gov/cuu/Units/checklist.html>
I.e. support outputting numbers of the form: "1234 M"
* src/numfmt.c (simple_strtod_human): Skip unit separator if present,
or disallow a unit separator if empty.
(double_to_human): Output unit separator if specified.
(main): Accept --unit-separator.
* tests/misc/numfmt.pl: Add test cases.
* doc/coreutils.texi: Describe the new option,
giving examples of interaction with --delimiter.
* NEWS: Mention the new feature.
* THANKS.in: Add Johannes Schauer Marin Rodrigues,
who provided a preliminary patch.
This does not validate grouping character placement,
and currently just ignores grouping characters.
* src/numfmt.c (simple_strtod_int): Skip grouping chars
that are part of a number.
* tests/misc/numfmt.pl: Add test cases.
* NEWS: Mention the improvement.
* src/numfmt.c (simple_strtod_human): Accept (multi-byte)
non-breaking space character between number and unit.
Note we restrict this to a single character between number
and unit, to allow less ambiguous parsing if multiple blanks
are used to delimit fields.
* tests/misc/numfmt.pl: Add test cases.
* doc/coreutils.texi (numfmt invocation): Fix stale description
--delimiter skipping whitespace.
* NEWS: Mention the improvement.
* tests/du/bigtime.sh: At least on Linux, the ext4 filesystem
doesn't support such large timestamp, while tmpfs does. Try a bit
harder to look for a filesystem with large timestamp support.
Rename to less redundant names, now that we use
a separate test directory per util.
* tests/basenc/basenc-bounded-memory.sh -> .../bounded-memory.sh
* tests/basenc/basenc-large.sh -> .../large-input.sh
* tests/local.mk: Reference new names.
* src/digest.c (split_3): Fallback to untagged matching in the
case where -a is specified and we have matched a TAG in
the possibly base64 data. This might happen in 1 in every 64K files.
Note we remove the modification of string S (and redundant streq) in
the tag matching, as that was not needed since v8.32-223-g217cd278e.
* tests/cksum/cksum-c.sh: Add a test case.
* NEWS: Mention the bug fix.
* src/digest.c (sha2_sum_stream): Change from unreachable()
to affirm() so that we have defined behavior unless
we configure with --disable-assert.
(sha3_sum_stream): Likewise.
(split_3): Validate SHA2-lengths before passing on.
* tests/cksum/cksum-c.sh: Add a test case.
* NEWS: Mention the bug fix.
* src/digest.c (split_3): Look up the provided tag with -a sha2
because there is not a 1:1 mapping between them.
* tests/cksum/cksum-c.sh: Add a test case.
* NEWS: Mention the bug fix.
* NEWS: Mention the bug.
* src/digest.c (split_3): Check that the base64 digest matches the
length supported by the algorithm.
(digest_check): Check that the read digest matches the base64 length of
the algorithm's digest. The previous condition would not work for
'cksum -a blake2b -l 8 ...'.
* tests/cksum/cksum-base64-untagged.sh: New file.
* tests/local.mk (all_tests): Add the new test.
This avoids CWE-122: Heap-based Buffer Overflow
where we could write blank characters beyond
the allocated heap buffer.
* src/expand-common.c (set_max_column_width): Refactor function from ...
(add_tab_stop): ... here.
(set_extend_size): Call new function.
(set_increment_size): Likewise.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/79555
* configure.ac: Add detection of AVX512 intrinsics for wc.
* src/local.mk: Build AVX512 wc libraries.
* src/wc.c: Add runtime detection of AVX512 intrinsics and call
appropriate function when detected.
* src/wc.h (wc_lines_avx512): Declare function.
* tests/wc/wc-cpu.sh: Add a test that disables AVX512 intrinsics.
* src/wc_avx512.c: New file containing the wc -l implementation using
AVX512. The logic and code is reused from the AVX2 implementation with
slight adaptations. Replaced __builtin_popcount by __builtin_popcountll
and the combination of _mm256_cmpeq_epi8 and _mm256_movemask_epi8 by a
single call to _mm512_cmpeq_epi8_mask.
* NEWS: Mention the improvement.
A coverage report indicated these weren't tested
(as generally the test shell builtin is used).
* tests/test/test-file.sh: Add a new test.
* tests/local.mk: Reference the new test.
* src/tail.c (file_lines): Seek to the previous block instead of the
beginning (or a little before) of the block that was just scanned.
Otherwise, the same block is read and scanned (at least partially)
again. This bug was introduced by commit v9.7-219-g976f8abc1.
* tests/tail/basic-seek.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: mention the bug fix.
base58_length() operated naively on an int
which resulted in an overflow to a negative number
for any input > 2^31-1/138, i.e. 15,561,475 bytes.
* src/basenc.c (base_length): Change input and output
parameter types from int to idx_t since this needs to
cater for the full input size in the base58 case.
(base58_length): Likewise. Also reorder the calculation
to be less exact, but doing the division first
to minimize the chance of overflow (which now on 64 bit
would only happen for inputs > around 6 Exa bytes).
* tests/basenc/basenc-large.sh: Add a new test,
that triggers with valgrind or ASAN.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
* tests/ls/block-size.sh: Skip the case where there are
non-printable characters in ls' output, which is the case
with NBSP thousands separators on FreeBSD 11 and 12.
We may drop the MBSW_REJECT_UNPRINTABLE in future from
ls and numfmt, but for now avoid these characters in the test.
Reported by Bruno Haible.
* tests/du/move-dir-while-traversing.sh: Expand the work to avoid
a false failure where du completes before the directory is moved.
Also expand the timeout to our more standard 10s to avoid the
"directory mover" being killed before du processes the directory.
This doesn't perceptibly impact the run time of the test.
Reported by Bruno Haible on a CentOS 7 system.
