This patch is by Gene Auyeung, Chris Dickens, Chen Guo, and Mike
Nichols, based off of a patch by Paul Eggert, Glen Lenker, et. al.,
with a basic heap implementation based off of the GDSL heap,
originally by Nicolas Darnis.
The number of sorts done in parallel is limited to the number
of available processors by default, or can be further restricted
with the --parallel option.
On a dual-die, 8 core Intel Xeon, results show sorting with
8 threads is almost 4 times faster than using a single thread.
Timings when sorting a 96MB file:
THREADS TIME (s)
1 5.10
2 2.87
4 1.75
8 1.31
Single threaded sorting has also been improved,
especially for cheaper comparison operations:
COMMAND BEFORE (s) AFTER (s)
sort 8.822 8.716
sort -g 10.336 10.222
sort -n 3.077 2.961
LANG=C sort 2.169 2.066
* bootstrap.conf: Add heap, pthread.
* coreutils.texi (sort): Describe the new --parallel option.
* gl/lib/heap.c: New file. Very basic heap implementation.
* gl/lib/heap.h: New file.
* gl/modules/heap: New file.
* src/Makefile.am: Add LIB_PTHREAD.
* src/sort.c: Include heap.h, nproc.h, pthread.h.
(MAX_MERGE): New macro.
(SUBTHREAD_LINES_HEURISTIC, PARALLEL_OPTION): New constants.
(MERGE_END, MERGE_ROOT): New constants.
(struct merge_node): New struct.
(struct merge_node_queue): New struct.
(sortlines temp): Remove declaration.
(usage, long_options, main): New option, --parallel.
(specify_nthreads): New function.
(mergelines): New signature, to emphasize the fact that the HI area
must be part of the destination. All callers changed.
(sequential_sort): New function, renamed from sortlines. Merge in
the functionality of sortlines_temp.
(compare_nodes): New function.
(lock_node, unlock_node): New functions.
(queue_destroy): New function.
(queue_init): New function.
(queue_insert): New function.
(queue_pop): New function.
(write_unique): New function.
(mergelines_node): New function.
(check_insert): New function.
(update_parent): New function.
(merge_loop): New function.
(sortlines): Rewrite to support and use parallelism, with a new
signature. All callers changed.
(struct thread_args): New struct.
(sortlines_thread): New function.
(sortlines_temp): Remove.
(sort): New argument NTHREADS. All uses changed. Output moved to
mergelines_node.
(main): disable threading if we are sorting at random.
* tests/Makefile.am (TESTS): Add misc/sort-benchmark-random.
* tests/misc/sort-benchmark-random: New file.
Signed-off-by: Pádraig Brady <P@draigBrady.com>
* gl/lib/dev-map.c, gl/lib/dev-map.h, gl/modules/dev-map: Remove.
* gl/lib/ino-map.c, gl/lib/ino-map.h, gl/modules/ino-map: New files.
* gl/modules/dev-map-tests, gl/tests/test-dev-map.c: Remove.
* gl/modules/ino-map-tests, gl/tests/test-ino-map.c: New files.
* gl/lib/di-set.h (struct di_set): Renamed from struct di_set_state,
and now private. All uses changed.
(_ATTRIBUTE_NONNULL_): Don't assume C99.
(di_set_alloc): Renamed from di_set_init, with no size arg.
Now allocates the object rather than initializing it.
For now, this no longer takes an initial size; we can put this
back later if it is needed.
* gl/lib/di-set.c: Include hash.h, ino-map.h, and limits.h instead of
stdio.h, assert.h, stdint.h, sys/types.h (di-set.h includes that
now), sys/stat.h, and verify.h.
(N_DEV_BITS_4, N_INO_BITS_4, N_DEV_BITS_8, N_INO_BITS_8): Remove.
(struct dev_ino_4, struct dev_ino_8, struct dev_ino_full): Remove.
(enum di_mode): Remove.
(hashint): New typedef.
(HASHINT_MAX, LARGE_INO_MIN): New macros.
(struct di_ent): Now maps a dev_t to a inode set, instead of
containing a union.
(struct dev_map_ent): Remove.
(struct di_set): New type.
(is_encoded_ptr, decode_ptr, di_ent_create): Remove.
(di_ent_hash, di_ent_compare, di_ent_free, di_set_alloc, di_set_free):
(di_set_insert): Adjust to new representation.
(di_ino_hash, map_device, map_inode_number): New functions.
* gl/modules/di-set (Depends-on): Replace dev-map with ino-map.
Remove 'verify'.
* gl/tests/test-di-set.c: Adjust to the above changes to API.
* src/du.c (INITIAL_DI_SET_SIZE): Remove.
(hash_ins, main): Adjust to new di-set API.
* gl/lib/mbsalign.c (mbsalign): Support the MBA_UNIBYTE_FALLBACK
flag which reverts to unibyte mode if one can't allocate memory
or if there are invalid multibyte characters present.
Note memory is no longer dynamically allocated in unibyte mode so
one can assume that mbsalign() will not return an error if this
flag is present. Don't calculate twice, the number of spaces,
when centering. Suppress a signed/unsigned comparison warning.
(ambsalign): A new wrapper function to dynamically allocate
the minimum memory required to hold the aligned string.
* gl/lib/mbsalign.h: Add the MBA_UNIBYTE_FALLBACK flag and
also document others that may be implemented in future.
(ambsalign): A prototype for the new wrapper.
* gl/tests/test-mbsalign.c (main): New test program.
* gl/modules/mbsalign-tests: A new index to reference the tests.
* .x-sc_program_name: Exclude test-mbsalign.c from this check.
Counterpart to commit 8fe40b84bd, since test-link.c uses rename,
and we override gnulib with a rename() replacement that can xalloc_die.
* gl/modules/link-tests.diff: New file.
A replacement getgroups is now guaranteed to exist, but it may
fail with ENOSYS. mgetgroups is moved to gnulib, and now takes
gid_t instead of GETGROUPS_T (but setgroups still needs GETGROUPS_T).
* gnulib: Update to latest.
* gl/modules/mgetgroups: Delete, moved to gnulib.
* gl/m4/mgetgroups.m4: Likewise.
* gl/lib/mgetgroups.h: Likewise.
* gl/lib/mgetgroups.c: Likewise.
* src/group-list.c (print_group_list): Adjust callers.
* src/id.c (print_full_info): Likewise.
cat, head, ptx, shuf, tac, tail, tee, tr, and uniq used freopen
on stdout, and were potentially vulnerable. dircolors, du, and
tsort only used it on stdin, which is unaffected by freopen_safer,
but this covers all uses for consistency.
* cfg.mk (sc_require_stdio_safer): New rule.
* gl/modules/xfreopen (Depends-on): Add freopen-safer.
* gl/lib/xfreopen.c (includes): Use stdio--.h.
* src/ptx.c (includes): Likewise.
* src/shuf.c (includes): Likewise.
* src/uniq.c (includes): Likewise.
* src/dircolors.c (includes): Likewise.
* src/du.c (includes): Likewise.
* src/tsort.c (includes): Likewise.
* gl/lib/mbsalign.c (mbsalign): Mark unused parameter.
* bootstrap.conf (gnulib_modules): Remove obsolete
rename-dest-slash.
* gnulib-tests/Makefile.am (AM_CFLAGS): Reduce set of warnings for
gnulib tests.
* gl/modules/rename-tests.diff (Makefile.am): New file, to add
LIBINTL to LDADD, since we avoid canonicalize-lgpl module.
* gl/lib/regcomp.c.diff (regerror, calc_next)
(build_collating_symbol, parse_bracket_element, build_equiv_class)
(free_tree): Mark unused parameters.
* gl/lib/regex_internal.h.diff (re_string_elem_size_at): New file,
to mark unused parameters.
* gl/lib/printf-args.c.diff (PRINTF_FETCHARGS): New file, to avoid
type mismatch.
* gl/lib/vasnprintf.c (VASNPRINTF): New file, to avoid shadowing
local variable name.
* .gitignore: Ignore temporary build artifacts.
* gl/lib/selinux-at.c: Remove file.
* gl/lib/selinux-at.h: Likewise.
* gl/modules/selinux-at: Likewise.
* gnulib: update to latest, to get the new module.
Reported by Samuel Thibault and Stéphane Raimbault, as the glibc fr_FR
locale has recently changed to use the official but variable width
abbreviated month names. Other glibc locales also have variable widths.
http://sourceware.org/ml/libc-locales/2008-q1/msg00035.htmlhttp://sourceware.org/bugzilla/show_bug.cgi?id=9859
* NEWS: Mention the fix
* gl/lib/mbsalign.c: A new module to align and truncate a
string in a specified number of screen cells, while handling
multi-byte characters appropriately.
* gl/lib/mbsalign.h: Ditto
* gl/modules/mbsalign: Ditto
* bootstrap.conf: Reference the new module
* src/ls.c (abmon_init): New function, precompute the abbreviated
months aligned left in a minimum width column <= 5 screen cells.
(align_nstrftime): New function, replace the first %b in the
format specification to strftime with the precomputed month string.
Note using the cached month strings speeds up `ls -lU` by around 17%
on glibc-2.7-2 on linux at least. Also if we implement this function
using heap storage rather than automatic storage, and use snprintf
instead of strcpy, ls will slow down by 2% and 1% respectively
(i.e. a net gain of 14% rather than 17%).
* tests/ls/abmon-align: A new test to test ls alignment for
various formats and locales
* tests/Makefile.am: Reference the new test
* gl/lib/argv-iter.h: New file.
* gl/lib/argv-iter.c: New file.
* gl/modules/argv-iter: New file.
With a suggestion for improved memory management by Pádraig Brady.
* gl/modules/selinux-at:
Ensure that LIB_SELINUX is cleared, in case it's set in the environment.
m4-quote the first two args to AC_SEARCH_LIBS.
Don't violate autoconf's ac_ namespace: s/ac_save/gl_save/
Drop the useless double quotes around a simple assignment RHS.
* gl/modules/selinux-h (Makefile.am)
[selinux/selinux.h, selinux/context.h]:
Remove temporary file and target, in case they're read-only.
Use $(MKDIR_P), not mkdir -p.
(License): Relax to LGPLv2+.
Remove vestigial comments.
* gl/modules/tempname (Depends-on): Add randint and stdbool.
* gl/lib/tempname.c: Include randint.h and stdbool.h.
(uint64_t): Remove definition. Not needed.
[_LIBC] (RANDOM_BITS): Remove this block, now that we have proper random bits.
(check_x_suffix): New function.
(gen_tempname_len): Rename from __gen_tempname.
Add a parameter, x_suffix_len, telling how many X's there must be at
the end of the template.
Use pseudo-random numbers all the way, rather than adding 7777
from one iteration to the next.
(__gen_tempname): New function, to call gen_tempname_len, requiring a
suffix length of 6.
* gl/lib/tempname.h: Add prototype for gen_tempname_len.
Signed-off-by: Jim Meyering <meyering@redhat.com>
* bootstrap.conf (gnulib_modules): Remove file-set, now that
it's in gnulib, and the canonicalize module requires it there.
* gl/lib/file-set.c, gl/lib/file-set.h, gl/modules/hash-triple: Remove.
* gl/lib/hash-triple.c, gl/lib/hash-triple.h, gl/modules/file-set: Remove.
* bootstrap.conf (gnulib_modules): Add file-set.
* gl/lib/file-set.c (record_file, seen_file): Functions from copy.c.
* gl/lib/file-set.h: Add prototypes.
* gl/lib/hash-triple.c (triple_hash, triple_hash_no_name):
(triple_compare, triple_free): Functions from copy.c.
* gl/lib/hash-triple.h (struct F_triple): Define. From copy.c.
Add prototypes.
* gl/modules/file-set: New module.
* gl/modules/hash-triple: New module.
* src/Makefile.am (copy_sources): New variable.
(ginstall_SOURCES, cp_SOURCES, mv_SOURCES): Use it.
* src/copy.c: Include hash-triple.h.
No longer include hash-pjw.h.
(copy_internal): Don't pass a NULL third argument to record_file,
since that function no longer accepts that.
(record_file): Move this function to file-set.c.
Along the way, remove the code to allow a NULL stat-buffer pointer.
Adjust sole caller.
(seen_file): Move this function to file-set.c.
(struct F_triple): Move declaration to hash-triple.h.
(triple_compare, triple_free, triple_hash, triple_hash_no_name):
Move these functions to hash-triple.c.
Signed-off-by: Jim Meyering <jim@meyering.net>
I wanted to use the xgetgroups function from id.c, so factored it out
and made it into a non-exiting function (hence the "m" prefix rather than "x").
* src/setuidgid.c (main): Use mgetgroups. Include "mgetgroups.h".
* src/id.c (xgetgroups): Remove function.
Include "mgetgroups.h".
(print_group_list): Use mgetgroups, not xgetgroups.
* gl/modules/mgetgroups: New module.
* gl/lib/mgetgroups.c: New file. mgetgroups is derived from
id.c's xgetgroups function.
* bootstrap.conf (gnulib_modules): Add mgetgroups.
* gl/m4/mgetgroups.m4: New file.
* gl/lib/mgetgroups.h: New file.
* gl/modules/selinux-at: New module. Check for libselinux and set
LIB_SELINUX here, unconditionally, rather than depending on
the configure-time --enable-selinux option.
* gl/modules/selinux-h: New module.
* bootstrap.conf (gnulib_modules): Add selinux-at.
* gl/lib/selinux-at.c, gl/lib/selinux-at.h: New files.
* gl/lib/se-selinux_.h: New file.
* gl/lib/se-context_.h: New file.
* gl/m4/selinux-selinux-h.m4: New file.
* gl/m4/selinux-context-h.m4: New file.
* src/Makefile.am (bin_PROGRAMS): Add chcon.
(chcon_LDADD): Define.
* README: Add chcon to the list of programs.
* src/chcon.c: Rewrite the original (Red Hat) chcon to use fts.
