sort -u could omit one or more lines of expected output.
This bug arose because sort recorded the most recently printed line via
reference, and if you were unlucky, the storage for that line would be
reused (overwritten) as additional input was read into memory. If you
were doubly unlucky, the new value of the "saved" line would not only
match the very next line, but if that next line were also the first in
a series of identical, not-yet-printed lines, then the corrupted "saved"
line value would result in the omission of all matching lines.
* src/sort.c (saved_line): New static/global, renamed and moved from...
(write_unique): ...here. Old name was "saved", which was too generic
for its new role as file-scoped global.
(fillbuf): With --unique, when we're about to read into a buffer that
overlaps the saved "preceding" line (saved_line), copy the line's .text
member to a realloc'd-as-needed temporary buffer and adjust the line's
key-defining members if they're set.
(overlap): New function.
* tests/misc/sort: New tests.
* NEWS (Bug fixes): Mention it.
* THANKS.in: Update.
Bug introduced via commit v8.5-89-g9face83.
Reported by Rasmus Borup Hansen in
http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/23173/focus=24647
* src/sort.c (check_inputs): A new function to verify all inputs
are accessible before further processing.
(check_output): A new function to open or create a specified
output file, before futher processing.
(stream_open): Adjust to truncating the previously opened
output file rather than opening directly.
(avoid_trashing_input): Optimize to stat the output file
descriptor, rather than the file name.
(main): Call the new functions to check accessibility of
inputs and output, before processing starts.
* tests/misc/sort: Adjust to the changed error message.
* tests/misc/sort-merge-fdlimit: Account for the earlier opened
file descriptor of the specified output file.
* tests/misc/sort-exit-early: A new test to exercise the improvements.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the improvement.
Suggested-by: Bernhard Voelker
Relax initial regexp to match more instances, but add a
filter to avoid some invalid conversions. Run this:
git grep -l "\`[^']*'" tests | xargs perl -pi -e '$q=q"'\''";' \
-e '$q="$q\\$q$q"; /(= ?\`|\`expr|\`echo|\Q$q\E)/ and next;' \
-e ' s/\`([^'\''"]*?'\'')/'\''$1/g'
The last disjunct in the above (...) filter is to exempt
any line that contains this string: '\''
With quoting like that, converting a ` to ' is likely to cause trouble,
so we'll handle those manually. Here are three examples where
the exemption is required:
*': `link-to-dir/'\'': hard link not allowed for directory'*) ;;
printf 'creating file `%s'\''\n' $f
'mv: inter-device move failed: `%s'\'' to `%s'\'';'\
Exempt lines with '$' or '=', since those are prone to improper
conversion. Run this:
git grep -l "\`[^']*'" tests \
|xargs perl -pi -e '/[=\$]/ and next;s/\`([^'\''"]*?'\'')/'\''$1/g'
* NEWS: Document this.
* src/sort.c (getmonth): Omit LEN arg, as MONTH is now null-terminated.
(compare_random): Don't null-terminate keys, as caller now does that.
(compare_version): Remove.
(debug_key): Null-terminate string for getmonth.
(keycompare): Support combining -R with any of -d, -f, -i, -V.
Also, support combining -V with any of -d, -i.
(check_ordering_compatibility): Allow newly-supported combinations.
* tests/misc/sort (02q, 02r, 02s): New tests, for new combinations.
(incompat2): Now test -nR, since -fR are now compatible.
* NEWS: Document changes to sort -h, which are now minor with
respect to the pre-July-30th version.
* doc/coreutils.texi (sort invocation): Likewise. The
documentation now describes how -h comparison is done rather than
being vague with border cases.
* src/sort.c (long_double, strtold): Move back to general_numcompare.
(LD, compute_human): Remove.
(find_unit_order): Remove THOU_SEP parameter, since thousands
separators are now allowed by all callers. Revert to previous
behavior of sorting by suffix, and returning the order rather than
2 * order + binary, since we no longer care whether binary powers
are being used. However, treat all zeros the same, instead of
sorting 0M before 0G; this is more consistent with the desired
behavior of sorting -1G before -1M.
* tests/misc/sort (h1, h3, h6): Adjust to match mostly-reverted
behavior. However, check that all zeros sort together.
* tests/misc/sort-debug-keys: Omit a "_", since the trailing "i"
in "1234Gi" is no longer part of the key.
* NEWS: Document changes to sort -h.
* doc/coreutils.texi (sort invocation): Likewise.
* src/sort.c (long_double, strtold): Move to prelude, since they're
now used by multiple functions.
(LD): New macro.
(struct keyfield.iec_present): Remove this member. All uses removed.
(check_mixed_SI_IEC): Remove. This code was busted in the presence
of multiple threads, as it had a race condition.
(find_unit_order): Remove arg KEY; add arg THOU_SEP; arg ENDPTR is
now char ** rather than char const **. Return an integer that
distinguishes decimal from binary powers. Parse the number
consistently with the intersection of strtold and strnumcmp.
Set *ENDPTR unconditionally.
(compute_human): New static function.
(human_numcompare): Remove arg KEY. Remove 'const' from other args.
Use strnumcmp if possible, but fall back on floating point if not.
(numcompare, general_numcompare): Arg EA is now char ** rather
than char const **.
(numcompare): Adjust to new find_unit_order signature and behavior.
(keycompare): Adjus to new human_numcompare signature.
* tests/misc/sort (h1, h3, h4, h6): Adjust to new behavior.
* tests/misc/sort-debug-keys: Likewise.
* src/sort.c (fillbuf): Don't append eol unless the line is nonempty.
This fixes a bug that was partly but not completely fixed by
the aadc67dfdb commit (dated July 15).
* tests/misc/sort (realloc-buf-2): New test, which catches this
bug on 64-bit hosts.
This regression was introduced in commit 224a69b5, 2009-02-24,
"sort: Fix two bugs with determining the end of field".
The specific regression being that we include 1 field too many when
an end field is specified using obsolescent key syntax (+POS -POS).
* src/sort.c (struct keyfield): Clarify the description of the eword
member, as suggested by Alan Curry.
(main): When processing obsolescent format key specifications,
normalize eword to a zero based count when no specific end char is given
for an end field. This matches what's done when keys are specified with -k.
* tests/misc/sort: Add a few more tests for the obsolescent key formats,
with test 07i being the particular failure addressed by this change.
* THANKS: Add Alan Curry who precisely identified the issue.
* NEWS: Mention the fix.
Reported by Santiago Rodríguez
* src/sort.c: Store the si_present state per key rather than globally
* tests/misc/sort: Add a check that would have previously failed.
Also add a test to demonstrate that invalid IEC/SI mixtures are not
always noticed when they're not significant to the sort.
* NEWS: Document the new option
* doc/coreutils.texi (sort invocation): ditto
* src/sort.c (main): handle the new --human-numeric-sort option (-h).
(human_numcompare): A new function to compare SI and IEC suffixes
before falling back to the standard --numeric comparison.
(find_unit_order): A new helper function to find the order
of magnitude of a number string as determined by its suffix.
(check_mixed_SI_IEC): A new helper function to exit with error
if both SI and IEC suffixes are presented.
* tests/misc/sort: Add 8 tests to test the new functionality.
* THANKS: Update
* src/sort.c (avoid_trashing_input): Fix an off-by-one error and
guard the use of memmove.
* NEWS (Bug fixes): Mention it.
* tests/misc/sort: Add tests to exercise the offending code.
* THANKS: Update.
Reported by Otavio Salvador in http://bugs.debian.org/525048.
* src/sort.c: When no specific number of chars to skip
is specified for the end field, always skip the whole field.
Also never include leading spaces from next field.
* tests/misc/sort: Add 2 new tests for these cases.
* NEWS: Mention this bug fix.
* THANKS: Add bug reporter.
Reported by Davide Canova.
* tests/Coreutils.pm: Add function to make limits available
* tests/test-lib.sh: ditto
* tests/misc/join: Check for both SIZE_OFLOW and UINTMAX_OFLOW
rather than using arbitrary 2^128
* tests/misc/sort: ditto
* tests/misc/uniq: ditto
* tests/misc/printf: Check for both INT_OFLOW and INT_UFLOW
rather than using arbitrary -2^31
* tests/misc/seq-long-double: Check for INTMAX_OFLOW
rather than using arbitrary 2^63
* tests/misc/split-fail: Check --lines --bytes and --line-bytes
options limits on all platforms. Note getlimits obviates the
need to use expr to check if 32 bit integers are supported,
which I think was invalid anyway as expr now supports bignum?
* tests/misc/test: Check for UINTMAX_OFLOW rather than
using arbitrary 2^64 and 2^128. Check for INTMAX_UFLOW
rather than using arbitrary -2^64
* tests/misc/timeout-parameters: Check for UINT_OFLOW
rather than using arbitrary 2^32
* tests/misc/truncate-overflow: Don't depend on truncate
to determine if we're on a 32 or 64 bit platform and
instead use the various OFF_T limits
* tests/misc/sort-merge: Check for UINTMAX_OFLOW
rather than using arbitrary 2^64+1
* tests/misc/unexpand: ditto
* tests/check.mk (TESTS_ENVIRONMENT): Save and restore TMPDIR around
envvar-check, so that the few scripts that require $TMPDIR don't fail.
This is also good to let a user's default TMPDIR setting be used e.g.,
in the search for an 'other-partition'.
FIXME: this is pretty ugly. maybe undo it and find a better way.
(TESTS_ENVIRONMENT): Invoke perl scripts with $(PERL), and use -T
if the script requires that. Otherwise, use $(SHELL).
* tests/misc/md5sum-newline: Create a file whose name contains
a newline in Perl (resort to using "system", since open refuses).
Fix old brokenness exposed by this change:
* tests/du/files0-from: Correct test not to rely on stdin
being attached to a non-tty.
* tests/misc/sort (3g, 3h, 3i): Likewise: add explicit empty input file.
Avoid warnings about using qw()-around-commas.
* tests/rm/fail-eperm: Now that this test is run from a temporary
subdirectory, adjust the full name of the "rm" program we're going
to run.
Change #!/bin/sh to #!/usr/bin/perl, and factor out the few lines
of boilerplate code to invoke perl. Do not "require 5.00x";
a configure-time Perl test handles that
* tests/dd/skip-seek:
* tests/misc/base64:
* tests/misc/basename:
* tests/misc/cut:
* tests/misc/date:
* tests/misc/dircolors:
* tests/misc/dirname:
* tests/misc/expand:
* tests/misc/expr:
* tests/misc/factor:
* tests/misc/fmt:
* tests/misc/fold:
* tests/misc/head:
* tests/misc/head-elide-tail:
* tests/misc/join:
* tests/misc/ls-misc:
* tests/misc/md5sum:
* tests/misc/md5sum-newline:
* tests/misc/mktemp:
* tests/misc/od:
* tests/misc/paste:
* tests/misc/pr:
* tests/misc/printf-cov:
* tests/misc/seq:
* tests/misc/sha1sum:
* tests/misc/sha1sum-vec:
* tests/misc/sha224sum:
* tests/misc/sha256sum:
* tests/misc/sha384sum:
* tests/misc/sha512sum:
* tests/misc/sort-merge:
* tests/misc/stat-printf:
* tests/misc/sum:
* tests/misc/tac:
* tests/misc/tail:
* tests/misc/test:
* tests/misc/test-diag:
* tests/misc/tr:
* tests/misc/tsort:
* tests/misc/tty-eof:
* tests/misc/unexpand:
* tests/misc/uniq:
* tests/misc/wc:
* tests/misc/wc-files0-from:
* tests/misc/xstrtol:
* tests/mv/i-1:
* tests/pr/pr-tests:
* tests/rm/empty-name:
* tests/rm/fail-eperm:
* tests/rm/unreadable:
