There was slight change to libsmack such that positive values are
reserved for returning length of the label for smack_new_label_from_*
functions.
* m4/jm-macros.m4: Set HAVE_SMACK when both smack_new_label_from_self()
and recently added smack_new_label_from_path() are present.
The latter's presence indicates the newer API of the former.
* src/id.c (main): Check that smack_new_label_from_self() < 0,
and not just non-zero.
* src/getlimits.c (print_float): Adjust to use the ftoastr module,
which uses the appropriate precision so that no info is lost.
* cfg.mk (sc_prohibit_continued_string_alpha_in_column_1): Exclude od.c
fixes http://bugs.gnu.org/14650
* src/du.c (usage): Clarify that --separate-dirs doesn't exclude
all directories.
* doc/coreutils.texi (du invocation): Avoid implying that -S
excludes the size of any non directory entries for a directory.
Also don't mention st_size as it's dependent on --apparent-size.
Reported by C de-Avillez in <https://launchpad.net/bugs/1187044>
For a file of size 1234 bytes, commit ca9aa759 had the side effect
of changing 'stat -c "%'s" file' from outputting "?s" to the nicer
"1,234", depending on locale. This is worth mentioning in the NEWS.
Resolves part of http://bugs.gnu.org/14556.
* NEWS: Mention 8.7 improvement in stat.
* cfg.mk (old_NEWS_hash): Adjust accordingly.
Prompted by the continuous integration build failure at:
http://hydra.nixos.org/build/5221053
* tests/tail-2/retry.sh: Ensure the 'out' file is truncated,
as it's used to arbitrate the run order of commands.
Relying on the truncation in the background tail command
is racy because the truncation can occur after the fork
of the background shell and thus wait4lines would not wait
for output to occur in 'out', which would mean that the
'missing' file could be populated by the time tail(1)
gets to process it.
* src/od.c (PRINT_FIELDS): Declare "i" to be of type uintmax_t, so that
the numerator in the expression for "next_pad" does not overflow.
(print_named_ascii): Likewise.
(print_ascii): Likewise.
Bug introduced via commit v6.12-42-g20c0b87.
* tests/misc/od.pl: Exercise each of the three affected code paths.
* NEWS (Bug fixes): Mention it.
Reported by Rich Burridge.
* tests/misc/head-c.sh: Don't try to elide 1 exabytes, since on
32-bit systems, that number is not representable as a size_t.
This command would fail on 32-bit systems, where SIZE_MAX < 1E:
head --bytes=-E < /dev/null
Instead of "E", use $SSIZE_MAX.
For discussion, see http://bugs.gnu.org/13530
* tests/split/line-bytes.sh: Since we've limited virtual memory to
20MB, choose a smaller size, 1GiB (which is <= SIZE_MAX) rather than
1EiB, which is larger than SIZE_MAX on 32-bit systems.
I confirmed that this test still fails when the split.c-modifying
part of v8.21-58-gfec363c is backed out.
* tests/ls/block-size.sh (size_etc): The sed expression through which
we filtered the output of "ls -l ..." assumed that the user and group
name components of each line would not contain spaces. Avoid the
problem by using -og instead of -l, thus not printing either of those
fields. Adjust the sed expression accordingly.
* src/split.c (line_bytes_split): Rewrite to only buffer
when necessary. I.E. only increase the buffer when we've
already lines output in a split and we encounter a line
larger than the input buffer size, in which case a hold
buffer will be increased in increments of the input buffer size.
(lines_rr): Use the more abstract xalloc_die() just like
we did in line_bytes_split(), rather than explicitly
printing the "memory exhausted" message and exiting.
* tests/split/line-bytes.sh: Add a new test for this
function which previously had no test coverage.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/13537
* README: Update the email address best suited to discussing
feature requests, and also link to the list of previously
discussed and rejected requests.
This fixes Bug#14371, reported by Killer Bassist.
* NEWS: Document this.
* src/mkdir.c (struct mkdir_options): Remove member ancestor_mode.
New member umask_value. All uses changed.
* src/mkdir.c (make_ancestor): Fix umask assumption.
* src/mkdir.c, src/mkfifo.c, src/mknod.c (main):
Leave umask alone. This requires invoking lchmod after creating
the file, which introduces a race condition, but this can't be
avoided on hosts with "POSIX" default ACLs, and there's no easy
way with network file systems to tell what kind of host the
directory is on.
* tests/local.mk (all_tests): Add tests/mkdir/p-acl.sh.
* tests/mkdir/p-acl.sh: New file.
Adds an optional dependency on libsmack.
* m4/jm-macros.m4: Look for the smack library/header.
* src/id.c (main): Output the smack context if available.
* src/local.mk: Link with libsmack if available.
* NEWS: Mention the new feature.
Use a sentinel value that's checked implicitly, rather than
a bit array, to determine if an item should be output.
Benchmark results for this change are:
$ yes abcdfeg | head -n1MB > big-file
$ for c in orig sentinel; do
src/cut-$c 2>/dev/null
echo -ne "\n== $c =="
time src/cut-$c -b1,3 big-file > /dev/null
done
== orig ==
real 0m0.049s
user 0m0.044s
sys 0m0.005s
== sentinel ==
real 0m0.035s
user 0m0.032s
sys 0m0.002s
## Again with --output-delimiter ##
$ for c in orig sentinel; do
src/cut-$c 2>/dev/null
echo -ne "\n== $c =="
time src/cut-$c -b1,3 --output-delimiter=: big-file > /dev/null
done
== orig ==
real 0m0.106s
user 0m0.103s
sys 0m0.002s
== sentinel ==
real 0m0.055s
user 0m0.052s
sys 0m0.003s
eol_range_start: Removed. 'n-' is no longer treated specially,
and instead SIZE_MAX is set for the 'hi' limit, and tested implicitly.
complement_rp: Used to complement 'rp' when '--complement' is specified.
ADD_RANGE_PAIR: Macro renamed to 'add_range_pair' function.
* tests/misc/cut-huge-range.sh: Adjust to the SENTINEL value.
Also remove the overlapping range test as this is no longer
dependent on large ranges and also is already handled with
the EOL-subsumed-3 test in cut.pl.
This issue was introduced in commit v8.21-43-g3e466ad
* src/cut.c (set_fields): Process all range pairs when merging.
* tests/misc/cut-huge-range.sh: Add a test for this edge case.
Also fix an issue where we could miss reported errors due
to truncation of the 'err' file.
Ensure appropriate functions are inlined. This was seen to
be required with gcc 4.6.0 with -O2 on x86_64 at least.
It was reported that gcc 4.8.0 did inline these functions though.
Also reinstate the bit vector for the common case,
to further improve performance.
Benchmark results for both aspects of this change are:
$ yes abcdfeg | head -n1MB > big-file
$ for c in orig inline inline-array; do
src/cut-$c 2>/dev/null
echo -ne "\n== $c =="
time src/cut-$c -b1,3 big-file > /dev/null
done
== orig ==
real 0m0.088s
user 0m0.081s
sys 0m0.007s
== inline ==
real 0m0.070s
user 0m0.060s
sys 0m0.009s
== inline-array ==
real 0m0.049s
user 0m0.044s
sys 0m0.005s
* src/cut.c (set_fields): Set up the printable_field bit vector
for performance, but only when it's appropriate. I.E. not
when either --output-delimeter or huge ranges are specified.
(next_item): Ensure it's inlined and avoid unnecessary processing.
(print_kth): Ensure it's inlined and add a branch for the fast path.
Related to http://bugs.gnu.org/13127
print_kth() is the central function of cut used to
determine if an item is to be output or not,
so simplify it by moving some logic outside.
Benchmark results for this change are:
$ yes abcdfeg | head -n1MB > big-file
$ for c in orig split; do
src/cut-$c 2>/dev/null
echo -ne "\n== $c =="
time src/cut-$c -b1,3 big-file > /dev/null
done
== orig ==
real 0m0.111s
user 0m0.108s
sys 0m0.002s
== split ==
real 0m0.088s
user 0m0.081s
sys 0m0.007s
* src/cut.c (print_kth): Refactor a branch to outside the function.
Related to http://bugs.gnu.org/13127
The current implementation of cut, uses a bit array,
an array of `struct range_pair's, and (when --output-delimiter
is specified) a hash_table. The new implementation will use
only an array of `struct range_pair's.
The old implementation is memory inefficient because:
1. When -b with a big num is specified, it allocates a lot of
memory for `printable_field'.
2. When --output-delimiter is specified, it will allocate 31 buckets.
Even if only a few ranges are specified.
Note CPU overhead is increased to determine if an item is to be printed,
as shown by:
$ yes abcdfeg | head -n1MB > big-file
$ for c in with-bitarray without-bitarray; do
src/cut-$c 2>/dev/null
echo -ne "\n== $c =="
time src/cut-$c -b1,3 big-file > /dev/null
done
== with-bitarray ==
real 0m0.084s
user 0m0.078s
sys 0m0.006s
== without-bitarray ==
real 0m0.111s
user 0m0.108s
sys 0m0.002s
Subsequent patches will reduce this overhead.
* src/cut.c (set_fields): Set and initialize RP
instead of printable_field.
* src/cut.c (is_range_start_index): Use CURRENT_RP rather than a hash.
* tests/misc/cut.pl: Check if `eol_range_start' is set correctly.
* tests/misc/cut-huge-range.sh: Rename from cut-huge-to-eol-range.sh,
and add a test to verify large amounts of mem aren't allocated.
Fixes http://bugs.gnu.org/13127
The StorNext distributed file system was previously known as CVFS.
* src/stat.c (human_fstype): Add new file system ID definition.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/14251
* init.cfg (require_ulimit_v_): Renamed from require_ulimit_
as this only checks for ulimit -v support. Other uses of
ulimit -t and ulimit -n in tests shouldn't cause false failures
if not supported.
* cfg.mk (sc_prohibit_test_ulimit_without_require_): A new syntax check
to ensure that require_ulimit_v_() is used iff required.
* tests/misc/head-c.sh: Add missing call to require_ulimit_v_.
* tests/rm/many-dir-entries-vs-OOM.sh: Likewise.
* tests/split/r-chunk.sh: Remove non mandatory require_ulimit_ call.
* tests/misc/sort-merge-fdlimit.sh: Likewise.
* tests/cp/link-heap.sh: Adjust to renamed require_ulimit_v_.
* tests/dd/no-allocate.sh: Likewise.
* tests/misc/csplit-heap.sh: Likewise.
* tests/misc/cut-huge-to-eol-range.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* scripts/autotools-install (tarballs): Use http:// URLs rather
than ftp:// ones. The former are more likely to work, these days.
Update URLs to point to the latest versions.
As a side effect of the previous commit which fixes 'tail -f --retry'
to wait for a file to appear, tail would not exit when the last file
appears untailable and gives up on this file.
This can happen, for example, when the argument file name appears
as directory. Tail sets the 'ignore' flag of this file to true,
but instead of exiting the program, tail would continue the loop.
* src/tail.c (any_live_files): Change the function to return true
if any of the files is still tailable or if tail should continue to
try to check again.
(tail_forever): Change the condition to break the loop in the
"no files remaining" case, because now any_live_files() will care
about it, as mentioned above.
(parse_options): When --retry is used without any follow mode,
then reset reopen_inaccessible_files to false.
* tests/tail-2/retry.sh: Add test case.
The --retry option is indeed useful for both following modes
by name and by file descriptor. The difference is that in the
latter case, it is effective only during the initial open.
As a regression of the implementation of the inotify support,
tail -f --retry would immediately exit if the given file is
inaccessible.
* src/tail.c (usage): Change the description of the --retry option:
remove the note that this option would mainly be useful when
following by name.
(main): Change diagnosing dubios uses of --retry option:
when the --retry option is used without following, then issue
a warning that this option is ignored; when it is used together
with --follow=descriptor, then issue a warning that it is only
effective for the initial open.
Disable inotify also in the case when the initial open in tail_file()
failed (which is the actual bug fix).
* init.cfg (retry_delay_): Pass excess arguments to the test function.
* tests/tail-2/retry.sh: Add new tests.
* tests/local.mk (all_tests): Mention it.
* doc/coreutils.texi (tail invocation): Enhance the documentation
of the --retry option. Clarify the difference in tail's behavior
regarding the --retry option when combined with the following modes
name versus descriptor.
* NEWS (Bug fixes): Mention the fix.
Reported by Noel Morrison in:
http://lists.gnu.org/archive/html/coreutils/2013-04/msg00003.html
On OS X it was seen that the group ID used for new files,
are set to a that of the directory rather than the current user.
It's not currently understood when this happens, but it was confirmed
that ACLs, extended attributes and setgid bits are _not_ involved.
* init.cfg (skip_if_nondefault_group_): A new function to detect
and avoid this situation. Document with links to the discussions
for hopefully future clarification.
* tests/install/install-C-root.sh: Use the new function.
* tests/install/install-C-selinux.sh: Likewise.
* tests/install/install-C.sh: Likewise.
* doc/coreutils.texi (install invocation): Mention that install(1) may
not correctly determine the default user or permissions for installed
files, and so is best used with options specifying these attributes.
* src/head.c (elide_tail_bytes_pipe): Don't use calloc as that
bypasses memory overcommit due to the zeroing requirement.
Also realloc rather than malloc the pointer array to avoid
dependence on overcommit entirely.
* tests/misc/head-c.sh: Add a test case.
Fixes http://bugs.gnu.org/13530
* src/dd.c: Add new static global variable ibuf.
(alloc_ibuf, alloc_obuf): New functions factored from dd_copy().
(dd_copy): Call the new functions to allocate memory for
ibuf and obuf when necessary.
(skip): Likewise.
* tests/dd/no-allocate.sh: New test.
* tests/local.mk: Reference the test.
With --suppress-matched, the lines that match the pattern will not be
printed in the output files. I.E. the first line from the second
and subsequent splits will be suppressed.
* src/csplit.c: process_regexp(),process_line_count(): Don't output the
matched lines. Since csplit includes "up to but not including" matched
lines in each split, the first line (in the next group) is the matched
line - so just skip it.
main(): Handle new option.
usage(): Mention new option.
* doc/coreutils.texi (csplit invocation): Mention new option, examples.
* tests/misc/csplit-suppress-matched.pl: New test script.
* tests/local.mk: Reference the new test.
* NEWS: Mention new feature.
Use the more portable 'chmod a-w', instead of the 'chmod -w' form.
The latter is not always supported. Also its operation is
dependent on umask controlling the permissions bits for new files,
which is not the case in the presence of POSIX default ACLs for e.g.
In that case, chmod may print a warning like the following, and
exit with failure status:
chmod: man/hostid.1-t: new permissions are r--rw-r--, not r--r--r--
* man/local.mk: s/-w/a-w/
* init.cfg (require_gcc_shared_): A new function to check
that we can build shared libraries in the particular manner
we use in our tests.
* tests/cp/nfs-removal-race.sh: Use require_gcc_shared_.
Then fail rather than skip, if the actual shared lib build fails.
* tests/df/no-mtab-status.sh: Likewise.
* tests/df/skip-duplicates.sh: Likewise.
* tests/ls/getxattr-speedup.sh: Likewise.
Reported in http://bugs.gnu.org/14024
* src/tail.c (main): If -n0 or -c0 were specified without -f,
then no data would ever be output, so exit without reading input.
* tests/tail-2/tail-n0f.sh: Augment the related test with this case.
* src/shuf.c (main): If -n0 specified then no data would ever be output,
so exit without reading input.
* tests/misc/shuf.sh: Augment the related test with this case.
* doc/coreutils.texi (ln invocation): Describe how symlinks are
resolved with --relative, and give an example showing the greater
control available through realpath(1).
* tests/ln/relative.sh: Add a test to demonstrate full symlink
resolution, in a case where it might not be wanted.
Don't dereference an existing symlink being replaced.
I.E. generate the symlink relative to the symlink's containing dir,
rather than to some arbitrary place it points to.
* src/ln.c (convert_abs_rel): Don't consider the final component
of the symlink name when canonicalizing, as we want to avoid
dereferencing the final component.
* tests/ln/relative.sh: Add a test case.
* NEWS: Mention the fix.
Resolves http://bugs.gnu.org/14116
Reservoir sampling optimizes selecting K random lines from large or
unknown-sized input: http://en.wikipedia.org/wiki/Reservoir_sampling
Note this also avoids reading any input when -n0 is specified.
* src/shuf.c (main): Use reservoir-sampling when the number of output
lines is known, and the input size is large or unknown.
(input_size): A new function to get the input size for regular files.
(read_input_reservoir_sampling): New function to read lines from input,
keeping only K lines in memory, replacing lines with decreasing prob.
(write_permuted_output_reservoir): New function to output reservoir.
* tests/misc/shuf-reservoir.sh: An expensive_ test using valgrind to
exercise the reservoir-sampling code.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
* README-release: fix the `make` command, and mention how
to get the latest results without requring running a
system with the latest kernel.
* src/local.mk (src/fs-latest-magic.h): A new target to
document how/where to place the latest magic header.
(src/fs-kernel-magic): Adjust to include separately
downloaded header if available.
(src/fs-magic): Undefine MANPAGER as it may impact the
ability to pipe the output of man(1).
(fs-magic-compare): Don't echo the commands run as they're
distracting from the output which needs to be examined.
* m4/gmp.m4 (cu_GMP): Add an extra check that gmp.h is available
which is required on one Mac OS X 10.5.8 system at least,
where the lib was available but the header wasn't.
Also enable our GMP code on systems where GMP is not in a separate lib.
