* src/sort.c (uintptr): New type.
(enum procstate, struct procnode, update_proc): Remove.
(proctab_hasher, proctab_comparator, register_proc, wait_proc):
(reap_some): The proctab is now simply a hash of process-IDs
rather than of pointers to objects with reference counts and
states; this is smaller and faster and easier to understand.
(nprocs): Now pid_t, not size_t, since one cannot have more than
PID_MAX children.
(reap): If the argument is -1, wait; if 0 (a new value), do not.
Delete pid from proctab as needed. Ignore children that are not
in proctab, as they are from the program that exec'ed us and are
irrelevant to our success or failure.
(delete_proc, reap_all): New functions.
(open_temp): Register the child.
(sort): Clean up all children afterwards; without this patch,
'sort' sometimes missed failures in children due to race conditions.
* tests/Makefile.am (TESTS): Add misc/sort-compress-proc.
* tests/misc/sort-compress-proc: New file, to test for the
bugs fixed above.
Running a command like this on a multi-core system
sort < big-file | less
would peg all processors at near 100% utilization.
* src/sort.c: (struct merge_node) Change member lock to mutex.
All uses changed.
* tests/Makefile.am (XFAIL_TESTS): Remove definition, now that
this test passes once again. I.e., the sort-spinlock-abuse test
no longer fails.
* NEWS (Bug reports): Mention this.
Reported by DJ Lucas in http://debbugs.gnu.org/7489.
* src/split.c (usage, long_options, main): New options --number,
--unbuffered, --elide-empty-files.
(set_suffix_length): New function to auto increase suffix length
to handle a specified number of files.
(create): New function. Refactored from cwrite() and ofile_open().
(bytes_split): Add max_files argument to support byte chunking.
(lines_chunk_split): New function. Split file into chunks of lines.
(bytes_chunk_extract): New function. Extract a chunk of file.
(of_info): New struct. Used by functions lines_rr and ofile_open
to keep track of file descriptors associated with output files.
(ofile_open): New function. Shuffle file descriptors when there
are more output files than available file descriptors.
(lines_rr): New function to distribute lines round-robin to files.
(chunk_parse): New function. Parses K/N syntax.
* tests/misc/split-bchunk: New test for byte chunking.
* tests/misc/split-lchunk: New test for line delimited chunking.
* tests/misc/split-rchunk: New test for round-robin chunking.
* tests/Makefile.am: Reference new tests.
* tests/misc/split-fail: Add failure scenarios for new options.
* tests/misc/split-l: Fix a typo. s/ln/split/.
* doc/coreutils.texi (split invocation): Document --number.
* NEWS: Mention the new feature.
* .mailmap: Map new email address for shortlog.
Signed-off-by: Pádraig Brady <P@draigBrady.com>
* NEWS: Describe patch.
* bootstrap.conf (gnulib_modules): Add ftoastr.
* src/od.c: Include ftoastr.h, not float.h.
(FLT_DIG, DBL_DIG): Remove. No need to verify LDBL_DIG.
(FMT_BYTES_ALLOCATED): No need to worry about floating point now,
since this format is no longer used for floating point.
(PRINT_FIELDS): New macro, with most of the guts of the old PRINT_TYPE.
(PRINT_TYPE): Rewrite to use PRINT_FIELDS.
(PRINT_FLOATTYPE): New macro. This uses the new functions from
ftoastr.
(print_float, print_double, print_long_double): Reimplement
using PRINT_FLOATTYPE.
(decode_one_format): Calculate field widths based on ftoastr-supplied
macros.
* tests/Makefile.am (TESTS): Add misc/od-float.
* tests/misc/od-float: New file.
* src/csplit.c (free_buffer): Also free the line offsets buffers
(remove_line): Also free the containing structure
* tests/misc/csplit-heap: A new test to trigger with leaks of
this magnitude.
* tests/Makefile.am: Reference the new test
* NEWS: Mention the fix
Reported by David Hofstee
Without this fix, seq 1000 | csplit - /./ '{*}' would write
the NUL-terminated file name, xx1000, into a buffer of size 6.
* src/csplit.c (main): Use properly sized file name buffer.
* NEWS (Bug fixes): Mention it.
* tests/misc/csplit-1000: New test to trigger the bug.
* tests/Makefile.am (TESTS): Add misc/csplit-1000.
This reverts part of the recent commit 9069af45,
"stat: print timestamps to full resolution", which made %X, %Y, %Z
print floating point numbers. We prefer to retain portability of
%X, %Y and %Z uses, while still providing access to full-resolution
time stamps via modified format strings. Also make the new
%W consistent.
* src/stat.c: Include "xstrtol.h".
(print_it): Accept a new %...:[XYZ] format directive,
e.g., %:X, to print the nanoseconds portion of the corresponding time.
For example, %3.3:Y prints the zero-padded, truncated, milliseconds
part of the time of last modification.
(print_it): Update print_func signature to match.
(neg_to_zero): New helper function.
(epoch_time): Remove function; replace with...
(epoch_sec): New function; use timetostr.
(out_ns): New function. Use "09" only when no other modifier
is specified.
(print_statfs): Change type of "m" to unsigned int,
now that it must accommodate values larger than 255.
(print_stat): Likewise.
Map :X to a code of 'X' + 256. Likewise for Y, Z and W.
(usage): Update.
* tests/touch/60-seconds: Use %Y.%:Y in place of %Y.
* tests/misc/stat-nanoseconds: New file.
* tests/Makefile.am (TESTS): Add it.
* NEWS (Changes in behavior): Mention this.
With improvements by Pádraig Brady.
Thanks to Andreas Schwab for raising the issue.
* src/du.c (show_date): Fix call to fputs with a buffer that
contains some uninitialized data.
* tests/Makefile.am (TESTS): Add du/big-timestamp.
* tests/du/bigtime: New file, which checks for the bug.
* src/tail.c (tail_forever_inotify): Handle the case where
tail --follow=name with inotify, is not able to add a watch on
a specified directory. This may happen due to inotify resource
limits or if the directory is currently missing or inaccessible.
In all these cases, revert to polling which will try to reopen
the file later. Note inotify returns ENOSPC when it runs out
of resources, and instead we report a particular error message,
lest users think one of their file systems is full.
(main): Document another caveat with using inotify, where we
currently don't recheck directories recreated after the
initial watch is setup.
* tests/tail-2/F-vs-rename: Fix the endless loop triggered by
the above issue.
* tests/tail-2/inotify-hash-abuse: Likewise.
* tests/tail-2/wait: Don't fail in the resource exhaustion case.
* tests/tail-2/F-vs-missing: A new test for this failure mode
which was until now just triggered on older buggy linux kernels
which returned ENOSPC constantly from inotify_add_watch().
* NEWS: Mention the fix.
This valid translation spec aborted:
LC_ALL=en_US.iso-8859-1 tr '[:upper:]- ' '[:lower:]_'
This invalid translation spec aborted:
LC_ALL=en_US.iso-8859-1 tr '[:upper:] ' '[:lower:]'
This was caused by commit 6efd1046, 05-01-2008,
"Avoid tr case-conversion failure in some locales"
This misaligned conversion spec was allowed:
LC_ALL=C tr 'A-Y[:lower:]' 'a-z[:upper:]'
This was caused by commit af5d0c36, 21-10-2007,
"tr: do not reject an unmatched [:lower:] or [:upper:] in SET1"
This misaligned spec was allowed by extending the class:
LC_ALL=C tr '[:upper:] ' '[:lower:]'
* src/tr.c (validate_case_classes): A new function to check
alignment of case conversion classes. Also it adjusts the
length of the sets so that locales with different numbers of
upper and lower case characters, don't cause issues.
(string2_extend): Disallow extending the case conversion
class as in the above example. That is locale dependent
and most likely not what the user wants.
(validate): Do the simple test for "restricted" char classes
earlier, so we don't redundantly do more expensive validation.
(main): Remove the case class validation, and simplify.
* tests/misc/tr-case-class: A new test to test the various
alignment and locale issues, associated with case conversion.
* tests/misc/tr: Move case conversion tests to new tr-case-class.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the fixes.
* src/find-mount-point.c: A new file refactoring
find_mount_point() out from df.c
* src/find-mount-point.h: Likewise.
* src/df.c: Use the new find-mount-point module.
* src/stat.c (print_stat): Handle the new %m format.
(find_bind_mount): A new function to
return the bind mount for a file if any.
(out_mount_mount): Print the bind mount for a file, or else
the standard mount point given by the find-mount-point module.
(usage): Document the %m format directive.
* src/Makefile.am: Reference the refactored find-mount-point.c
* po/POTFILES.in: Add find_mount_point.c to the translation list
* doc/coreutils.texi (stat invocation): Document %m,
and how it may differ from the mount point that df outputs.
* test/misc/stat-mount: A new test to correlate mount points
output from df and stat.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the new feature
* THANKS: Add the author
Signed-off-by: Pádraig Brady <P@draigBrady.com>
This patch is by Gene Auyeung, Chris Dickens, Chen Guo, and Mike
Nichols, based off of a patch by Paul Eggert, Glen Lenker, et. al.,
with a basic heap implementation based off of the GDSL heap,
originally by Nicolas Darnis.
The number of sorts done in parallel is limited to the number
of available processors by default, or can be further restricted
with the --parallel option.
On a dual-die, 8 core Intel Xeon, results show sorting with
8 threads is almost 4 times faster than using a single thread.
Timings when sorting a 96MB file:
THREADS TIME (s)
1 5.10
2 2.87
4 1.75
8 1.31
Single threaded sorting has also been improved,
especially for cheaper comparison operations:
COMMAND BEFORE (s) AFTER (s)
sort 8.822 8.716
sort -g 10.336 10.222
sort -n 3.077 2.961
LANG=C sort 2.169 2.066
* bootstrap.conf: Add heap, pthread.
* coreutils.texi (sort): Describe the new --parallel option.
* gl/lib/heap.c: New file. Very basic heap implementation.
* gl/lib/heap.h: New file.
* gl/modules/heap: New file.
* src/Makefile.am: Add LIB_PTHREAD.
* src/sort.c: Include heap.h, nproc.h, pthread.h.
(MAX_MERGE): New macro.
(SUBTHREAD_LINES_HEURISTIC, PARALLEL_OPTION): New constants.
(MERGE_END, MERGE_ROOT): New constants.
(struct merge_node): New struct.
(struct merge_node_queue): New struct.
(sortlines temp): Remove declaration.
(usage, long_options, main): New option, --parallel.
(specify_nthreads): New function.
(mergelines): New signature, to emphasize the fact that the HI area
must be part of the destination. All callers changed.
(sequential_sort): New function, renamed from sortlines. Merge in
the functionality of sortlines_temp.
(compare_nodes): New function.
(lock_node, unlock_node): New functions.
(queue_destroy): New function.
(queue_init): New function.
(queue_insert): New function.
(queue_pop): New function.
(write_unique): New function.
(mergelines_node): New function.
(check_insert): New function.
(update_parent): New function.
(merge_loop): New function.
(sortlines): Rewrite to support and use parallelism, with a new
signature. All callers changed.
(struct thread_args): New struct.
(sortlines_thread): New function.
(sortlines_temp): Remove.
(sort): New argument NTHREADS. All uses changed. Output moved to
mergelines_node.
(main): disable threading if we are sorting at random.
* tests/Makefile.am (TESTS): Add misc/sort-benchmark-random.
* tests/misc/sort-benchmark-random: New file.
Signed-off-by: Pádraig Brady <P@draigBrady.com>
From there, they will be used by both test-lib.sh (as we phase it out)
and the newer init.sh, to which all tests will migrate.
* tests/test-lib.sh: Move most functions from here, ...
* tests/init.cfg: ...to here. New file.
* tests/Makefile.am (EXTRA_DIST): Add init.cfg.
* src/sort.c (usage): Mention --debug can output warnings to stderr.
Also split the translatable string to aid translation.
(default_key_compare): A new function refactored from main(),
and now also called from the new key_warnings() function.
(key_to_opts): A new function refactored from incompatible_options(),
and now also called from the new key_warnings() function.
(key_numeric): A new function refactored to test if key is numeric.
(key_warnings): A new function to output warnings to stderr,
about questionable use of various options. Currently it warns
about zero length keys and ineffective global options.
(incompatible_options): Refactor out key_to_opts()
(main): Use key_init() to initialize gkey. Refactor out
default_key_compare(). Call key_warnings() in debug mode.
* doc/coreutils.texi (sort invocation): Mention that warnings
are output by --debug.
* tests/misc/sort-debug-warn: A new test for debug warnings.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the new feature
* src/sort (usage): Add description for --debug.
(write_bytes): Pass a line structure so it can subsequently
be passed to compare to highlight the keys when in debug mode.
Also transform TAB and NUL characters written to stdout so
that the highlighting in debug mode aligns correctly.
(human_numcompare): Pass an "endptr" so we can record the extent
of the number matched.
(general_numcompare): Likewise.
(find_unit_order): Likewise.
(getmonth): Likewise.
(numcompare): Likewise. Note we reuse find_unit_order() for this,
which is a good enough approximation, and means we don't need to
change the strnumcmp() interface.
(check_mixed_SI_IEC): Return whether iec_present, so that can be
used to set the "endptr" in find_unit_order. Also make the key
parameter optional, which will be the case from numcompare().
(count_tabs): A new function to determine how much to adjust
the mbswidth() values by (TABs don't have a width).
(mark_key): A new function to output the key highlighting to stdout.
(debug_key): A new function to determine the offset and width
of the key highlighting.
(key_compare): Pass the show_debug parameter so the key highlighting
is only displayed when explicitly called. For each key type, set
the length (lena) and whether leading blanks are auto skipped (skipb)
which are then used by debug_key() to highlight the portion of the
key used in the comparison.
(compare): Pass the show_debug parameter so the key highlighting
is only displayed when explicitly called. Call debug_key() to
highlight the last resort comparison.
(check): Output highlighting for disorder line to stdout.
(main): Process the --debug option and make it mutually exlusive
with the -o option as I don't see it useful there, even potentially
harmful if someone left a --debug in by mistake when updating a file.
Also restricting debug output to stdout, simplifies the logic
for dealing with temporary files.
* doc/coreutils.texi (sort invocation): Describe the --debug option,
and reference it from the --key description.
* tests/misc/sort-debug-keys: A new test for highlighting keys.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the new feature.
* src/sort.c (general_numcompare): Use long doubles unconditionally,
and strtold when available, to convert numbers with greater range and
precision. Performance was seen to be on par with standard doubles.
* doc/coreutils.texi (sort invocation): Amend the -g description to
mention long double rather than double, and strtold rather than strtod.
* src/getlimits.c (main): Output floating point limits for use in tests.
* tests/misc/sort-float: A new test to ensure sort is using long
doubles when possible, and that locale specific floats are handled.
* tests/Makefile.am: Reference the new test.
* tests/test-lib.sh (getlimits_): Normalize indenting.
* NEWS: Mention the new behaviour.
Reported by Nelson Beebe.
* src/copy.c (copy_reg): Copy xattrs _after_ setting file ownership
so that capabilities are not cleared when setting ownership.
* tests/cp/capability: A new root test.
* tests/Makefile.am (root_tests): Reference the new test.
* NEWS: Mention the fix.
* src/sort.c (char fold_toupper[]): Change to unsigned
so as the correct comparisons are made in getmonth().
This fixes unibyte locales where abbreviated months
have characters that are > 0x7F, but it also works for
multibyte locales with the caveat that multibyte characters
are matched case sensitively.
With this change, the following example sorts correctly:
$ echo -e "1 márta\n2 Feabhra" | LANG=ga_IE.utf8 sort -k2,2M
2 Feabhra
1 márta
* src/sort.c (inittables): Since we ignore blanks around months
in the input, don't include them when they're present in the locale.
With this change, the following example sorts correctly:
$ echo -e "1 2月\n2 1月" | LANG=ja_JP.utf8 sort -k2,2M
2 1月
1 2月
* tests/misc/sort-month: A new test to exercise the above cases.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the fix.
Output the NORMAL attribute before non file name text.
This attribute will continue into file names that would
not otherwise be colored unless FILE is also set.
The regression was introduced with commit 483297d5, 28-02-2009,
"ls --color no longer outputs unnecessary escape sequences".
* src/ls.c (set_normal_color): A new function to output the
NORMAL attribute sequence if it's enabled.
(print_current_files): Output NORMAL before printing long format info.
(print_file_name_and_frills): Output NORMAL before printing file name.
(print_color_indicator): Reset the attributes before a file name with
attributes so that NORMAL attributes will not combine with them.
(print_name_with_quoting): Ensure attributes are reset after printing
the file name if NORMAL attributes were output.
* tests/ls/color-norm: A new test for NORMAL and FILE combinations.
* tests/Makefile.am: Reference the new test.
* NEWS: Mention the fix.
Reported in https://savannah.gnu.org/bugs/?26512
* src/wc.c (main): Set stdout to line buffered mode
to ensure parallel running instances don't intersperse
their output. This adds 6.5% to the run time in the worst case
of many zero length files, but has neglible impact for
standard sized files.
* tests/misc/wc-parallel: New test for atomic output.
* tests/Makefile.am: Reference it.
* NEWS: Mention the fix
This is similar to commit 710fe413, 20-10-2009,
"md5sum, sha*sum, sum: line-buffer the printed checksums"
* src/remove.c (rm_fts): Fix incorrect comparison of
device and inode numbers.
* tests/rm/one-file-system2: Add a separate test so
that it can be run as a normal user (It doesn't need to mount).
* tests/Makefile.am: Reference it.
* NEWS: Mention the fix.
Reported by Jan Larres.
Tailing forever and by-name (--follow=name, -F), tail would
sometimes fail to follow a file that had been removed via rename.
If you can't apply this patch and have tail 7.6 or newer, you can
work around the bug via the undocumented --disable-inotify option.
* src/tail.c (tail_forever_inotify): When tailing by name (-F),
do not un-watch a file upon receipt of the IN_MOVE_SELF event.
Reported by Arjan Opmeer in http://bugs.debian.org/548439.
* NEWS (Bug fixes): Mention it.
Also see http://marc.info/?l=coreutils-bug&m=125829031916515
* tests/Makefile.am (TESTS): Add tail-2/inotify-rotate.
* tests/tail-2/inotify-rotate: New test.
Allows for unambiguous processing when environment values (or even
non-portable names!) contain newline.
* src/env.c (longopts): Add new option.
(usage): Document it.
(main): Implement it.
* src/printenv.c (longopts): New variable.
(usage): Document new option.
(main): Implement it.
* doc/coreutils.texi (Common options): New macro optNull.
(du invocation, env invocation, printenv invocation): Use it.
* NEWS: Mention this.
* tests/misc/env-null: New test.
* tests/Makefile.am (TESTS): Run it.
If new data becomes available between the initial read and when tail
registers the inotify watch descriptors, ensure that it is read
before a new event happens on the file.
* tests/Makefile.am (TESTS): Add tail-2/inotify-race.
* tests/tail-2/inotify-race: New file.
* src/md5sum.c (main): Set stdout to line buffered mode
to ensure parallel running instances don't intersperse
their output. This adds 5% to the run time in the worst case
of many zero length files, or 2% with standard file sizes.
* src/sum.c (main): Likewise.
* tests/misc/md5sum-parallel: New test for atomic output.
* tests/Makefile.am: Reference it.
* NEWS: Mention the fix
* src/touch.c (no_dereference): New flag variable.
(longopts): Add -h/--no-dereference.
(touch): Add symlink handling.
(usage): Document new option.
(main): Accept new option.
* NEWS: Document it.
* doc/coreutils.texi (touch invocation): Likewise. Also mention
birthtime.
* tests/touch/no-dereference: New test.
* tests/Makefile.am (TESTS): Run it.
* src/stat.c (do_stat): Interpret a command line argument of "-"
to mean "standard input", like many other tools do.
(do_statfs): Fail upon any attempt to use "-".
* NEWS (Changes in behavior): Mention it.
* tests/misc/stat-hyphen: New test, to exercise the above.
* tests/Makefile.am (TESTS): Add misc/stat-hyphen.
A valid command like "touch -t 197101010000.60 F" would fail due
to the suffix of ".60". This bug is fixed via the latest change
to gnulib's posixtm module.
* tests/touch/60-seconds: New test.
* tests/Makefile.am (TESTS): Add it.
* NEWS (Bug fixes): Mention it.
* src/id.c (print_full_info) [POSIXLY_CORRECT]: Don't print context.
Reported by Ulrich Drepper.
* NEWS (Changes in behavior): Mention it.
* doc/coreutils.texi (id invocation): Document that id also prints the
security context, when possible, and when POSIXLY_CORRECT is not set.
* tests/id/no-context: New file. Test for this.
* tests/Makefile.am (TESTS): Add it.
Before this change, :>f; ln -T f no-such/ would succeed on Solaris 10.
After it, ln fails, as it should: ln: accessing `z/': Not a directory
The command, link f no-such/, had the same problem on that system.
* bootstrap.conf (gnulib_modules): Add "link".
* tests/ln/slash-decorated-nonexistent-dest: New test.
* tests/Makefile.am (TESTS): Add it.
* NEWS (Portability): Mention the improvement.
