2022-11-06 Jim Meyering version 4.9 * NEWS: Record release date. build: update gnulib to latest, and testsuite/init.sh tests: do not rely on unportable "grep -q" * testsuite/inplace-selinux.sh: Avoid unportable "grep -q". Instead, just redirect to /dev/null. 2022-10-23 Jim Meyering test: add a test case and mention the change in NEWS * testsuite/misc.pl: Add a test to exercise the preceding change. * NEWS (Changes in behavior): Mention it. 2022-10-23 Oğuz sed: handle the unspecified "n as delimiter alias" case more sensibly Print the less-surprising variant in a corner case of POSIX-unspecified behavior. Before, this would print "n". Now, it prints "X": printf n | sed 'sn\nnXn'; echo * sed/compile.c (match_slash): Remove special handling of 'n'. Reported in https://bugs.gnu.org/40242 2022-10-23 Jim Meyering build: update gnulib to latest, for Solaris static_assert fix 2022-10-16 Jim Meyering build: fix am_DISTCHECK_CONFIGURE_FLAGS typo * Makefile.am (AM_DISTCHECK_CONFIGURE_FLAGS): Correct typo in recent change: s/am_/AM_/ build: update gnulib to latest, for getdelim vs macOS workaround maint: avoid syntax-check failure * Makefile.am (am_DISTCHECK_CONFIGURE_FLAGS): Change DISTCHECK_CONFIGURE_FLAGS to am_DISTCHECK_CONFIGURE_FLAGS. maint: tell git to ignore new generated files in lib/ * lib/.gitignore: Add /assert.h and /stdckdint.h build: work when perl is installed but not in /usr/bin/perl Also support building on systems without perl and when cross-compiling. * configure.ac (AC_ARG_ENABLE [bold-man-page-references]): New configure-time option, using code copied from coreutils. (BUILD_MAN_PAGE, BUILD_DUMMY_MAN_PAGE): Remove definitions. (HAVE_PERL): Remove bogus conditional. * doc/dummy-man: New file, from coreutils, with slight adapation. * build-aux/help2man: Update from coreutils. * doc/local.mk (EXTRA_DIST): Distribute dummy-man. (run_help2man): New variable. (doc/sed.1): Use it rather than simply invoking help2man. Simplify to use only one copy of this rule, now that run_help2man works both with and without perl and when cross-compiling. Move the --name=... string argument into... * doc/sed.x: ... here. Also make the "Synopsis" appear in generated file. In https://lists.gnu.org/r/sed-devel/2022-10/msg00005.html, Bruno Haible reported build failure with perl in a different location and (in another report) another failure with no perl at all. build: fix THANKS generation * .mailmap: Building THANKS would fail due to thanks-gen finding two different email addresses for Mike Frysinger. Map the older one to the newer one. build: update gnulib to latest 2022-09-15 Marvin Schmidt tests: remove stray export-ending semicolon in test setup * testsuite/local.mk (TESTS_ENVIRONMENT): Remove the erroneous ";" that effectively elided all following var=val pairs from the test environment. Report and patch from http://bugs.gnu.org/57831 Bug introduced via v4.2.2-100-gb250bd4. 2022-07-03 Paul Eggert maint: pacify ‘make syntax-check’ * THANKS.in: Put J.T. before Jakub again, as apparently that’s how it’s done in Fedora 36 which is bleeding edge. * cfg.mk (sc_THANKS_in_sorted): Update for Fedora 36 sort. (sc_gitignore_missing): The lib files to ignore are now in lib/.gitignore, not .gitignore. sed: fix infloop with symlink cycles * bootstrap.conf (gnulib_modules): Add eloop-threshold, idx, minmax, readlink. * configure.ac: Do not check for lstat; no longer needed. (ENABLE_FOLLOW_SYMLINKS): Remove; all uses removed. (TEST_SYMLINKS): Depend only on readlink. * sed/utils.c: Include eloop-threshold.h, idx.h, minmax.h. (SSIZE_IDX_MAX): New macro. (follow_symlink): Rewrite to not loop when given a symlink cycle. Do not use lstat, since readlink suffices. Use just one memory buffer, not two; this simplifies memory management. * testsuite/follow-symlinks.sh: Adjust diagnostics to to match revised behavior. Test for symlink loops. 2022-07-02 Paul Eggert build: update gnulib submodule to latest 2022-07-02 Jim Meyering maint: exempt doc/fdl.texi from long-line check * cfg.mk (FILTER_LONG_LINES): Also exempt doc/fdl.texi. 2022-07-02 Paul Eggert sed: fix temp file cleanup Without this fix, the code would sometimes use FP after calling fclose (FP), which has undefined behavior in C. Problem found with --enable-gcc-warnings and GCC 12. * sed/execute.c (open_next_file): Do not register here, as it’s too late and this can cause the file to not be cleaned up. * sed/sed.c (G_file_to_unlink, register_cleanup_file, cancel_cleanup): Move from here to utils.c. (cleanup): Call remove_cleanup_file instead of doing it by hand. * sed/utils.c (struct open_file): Remove member temp (which was always false) and fclose_failed (which was not enough to prevent calling fclose with a bad pointer). All uses changed. (register_open_file): Do not access p->fp after it’s fclosed, as that has undefined behavior in C. Use xmalloc instead of xcalloc, since we initialize all members. (G_file_to_unlink, register_cleanup_file, cancel_cleanup): Move from utils.c to here. (remove_cleanup_file): New function. (ck_mkstemp): Fix a screwup when mkostemp succeeded but set_binary_mode or fdopen failed: we might misuse a null pointer, or forget to clean up the newly-created temp file. (ck_getdelim): Rename local to avoid confusion with global. (mark_as_fclose_failed): Remove. All uses removed. (ck_fclose): Remove entry from open_files before attempting to fclose it, so that panicking doesn’t try to fclose it again. (do_ck_fclose): New arg NAME so that there’s no need to call mark_as_fclose_failed, which inspected FP after fclosing it, which is undefined behavior. (ck_rename): Omit arg UNLINK_IF_FAIL. All callers changed. The cleanup handler removes this file now, as needed. .gitconfig: add gt-bug* for some test cases Remove unused BOOST.tests, PCRE.tests, SPENCER.tests Remove unused files testcases.h, ptestcases.h maint: stop using fdl module This is as per current Gnulib recommendations. Update COPYING, bootstrap from Gnulib Update, simplify, and sort .gitignore files Add NEWS item for dfa fix in Gnulib 2022-02-13 Chris Marusich tests: export CONFIG_HEADER to test scripts Before this change, the testsuite/inplace-selinux test fails in some cases when SELinux is available. The reason it fails is because the require_selinux_ function in init.cfg expects the CONFIG_HEADER environment variable to be defined, but in fact it is not defined. Even though we do invoke AM_CONFIG_HEADER in configure.ac, and even though the CONFIG_HEADER variable does get set in the resulting Makefile, nothing actually exports the CONFIG_HEADER variable, so the test is doomed to fail whenever it is not skipped. Fix this by exporting the variable. * testsuite/local.mk (TESTS_ENVIRONMENT): Add CONFIG_HEADER to the list of variables to export to the environment of the test scripts. Reported by Vineet Jain in https://bugs.gnu.org/36150 . The fix was suggested by Timothy Sample in the same bug report. 2022-01-01 Jim Meyering maint: make update-copyright build: update gnulib to latest; also bootstrap and init.sh 2021-12-24 Jim Meyering maint: syntax-check: disable the indent check * cfg.mk (local-checks-to-skip): Add sc_indent. Otherwise, "make syntax-check" would fail for me on a fedora 25 system. 2021-12-17 Jim Meyering build: non-recursive Automake in a less hacky way * bootstrap.conf (gnulib_modules): Remove non-recursive-gnulib-prefix-hack. (gnulib_tool_option_extras): Add --automake-subdir. (bootstrap_post_import_hook): No need to massage lib/gnulib.mk. * configure.ac [AM_INIT_AUTOMAKE]: Move subdir-objects onto the same line as the function name, so gnulib-tool sees it. 2021-10-02 Jim Meyering doc: fix more typos * doc/sed.texi: Fix typos. 2021-10-02 Antonio Diaz Diaz doc: fix a typo * doc/sed.texi (sed regular expressions): s/considrations/considerations/ This addresses https://bugs.gnu.org/50943 2021-09-05 Jim Meyering maint: avoid new syntax-check failure * doc/sed.texi: Split lines longer than 80. build: update gnulib to latest build: update gnulib to latest doc: tighten/clarify wording e.g., re "g" modifier * doc/sed.texi: When declaring that an example replaces all occurrences, use "g" and make the description more precise. Prompted by http://bugs.gnu.org/50361 Stop using "the word" when describing a match, to avoid the implication of a connection with "word"-delimited (\b) matching. Prefer to match "on lines ...", not "in lines". 2021-08-16 Assaf Gordon sed: allow '0rFILE' (insert FILE before the first line) The 'r' command can be used with address zero, effectively prepending a file to the beginning of the input file, e.g.: sed '0rA.TXT' B.TXT > C.TXT is equivalent to: cat A.TXT B.TXT > C.TXT With "sed -i", this allows safe in-place prepending of files. A typical example would be adding a license header to multiple source files: sed -i '0rLICENSE' *.c *.h find -iname '*.cpp' | xargs sed -i '0rLICENSE' A current cumbersome alternative is: sed -i -e 'x;${p;x};1rA.TXT' -e '1d' B.TXT * NEWS: Mention new feature. * sed/sed.h (struct readcmd): New struct. (struct sed_cmd): Use new struct instead of a char* for the filename. * sed/compile.c (compile_program): Expand conditional detecting invalid usage of "0" address to allow "0r"; Adjust '0r' to '1r' with prepending (instead of appending). * sed/execute.c (execute_program): 'r' command: support prepending. * sed/debug.c (debug_print_function): Use the new 'struct readcmd'. * testsuite/cmd-0r.sh: New test. * testsuite/local.mk (TESTS): Add new test. * doc/sed.texi (Zero Address): New section. (Adding a header to multiple files): New example section. 2021-08-11 Assaf Gordon sed: extract print_file function * sed/execute.c (dump_append_queue): Extract code to print a file to ... (print_file): new function. 2021-08-04 Renaud Pacalet doc: fix wording in 'line length adj.' example Reported in https://bugs.gnu.org/49680 . * doc/sed.texi: Fix wording, typo. 2021-08-01 Jim Meyering sed: avoid potential double-fclose Upon a failed temp file fclose, do_ck_fclose would call panic, which would then attempt to fclose and unlink that same pointer. Caught by gcc's new -Wanalyzer-double-fclose. * sed/utils.c (struct open_file) [fclose_failed]: New member. (panic): Don't double-close. (register_open_file): Clear new member. (mark_as_fclose_failed): Use new member to avoid double fclose. (do_ck_fclose): Call mark_as_fclose_failed upon fclose failure. maint: update obsolete constructs in configure.ac * configure.ac: Avoid autoconf warnings: Switch from obsolete AM_CONFIG_HEADER to AC_CONFIG_HEADERS, and quote. Use AC_PROG_CC_STDC, not AC_PROG_CC. Convert from obsolete AC_TRY_RUN to AC_RUN_IFELSE. maint: avoid new warning about deprecated security_context_t * sed/execute.c (open_next_file): Use char * in place of deprecated security_context_t. 2021-07-10 Jim Meyering build: update gnulib to latest * gnulib: update to latest * .gitignore files: Reflect gnulib changes. * bootstrap: update from gnulib 2021-01-05 Assaf Gordon build: restore build rules for manual page Reported by Kian Kasad in https://lists.gnu.org/r/sed-devel/2020-12/msg00000.html , (continued in https://lists.gnu.org/r/sed-devel/2021-01/msg00000.html ). * configure.ac: Enable buidling the manual page regardless of whether building from git or from a tarball. 2021-01-05 Assaf Gordon sed: silence -Wformat-nonoliteral warning GCC compilation fails with: sed/compile.c:206:3: error: format not a string literal, argument types not checked [-Werror=format-nonliteral] sprintf (unknown_cmd, msg, ch); ^~~~~~~ While there is suppression of "-Wformat-nonliteral" in configure.ac, it later '-Wformat=2' which turns "-Wformat-nonliteral" on. * sed/compile.c (bad_command): Add explicit "#pragma GCC ... ignore". The format string is fixed and known ahead of time. 2021-01-05 Assaf Gordon maint: update all copyright year number ranges Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * bootstrap.conf (gnulib_modules): Remove getopt module, depracated in https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=0abc38bd2a1398f0 * po/POTFILES.in: Remove getopt.c * sed/utils.h (panic): Rename _GL_ATTRIBUTE_FORMAT_PRINTF to _GL_ATTRIBUTE_FORMAT_PRINTF_STANDARD following gnulib's change: https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=387d654cabd7bc15 * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. 2020-03-23 Jim Meyering build: update to newer help2man * build-aux/help2man: Update from coreutils. This updates from 1.28 to 1.47.3 * cfg.mk (exclude_file_name_regexp--sc_long_lines): Exempt help2man from the 80-column-max limit. This addresses https://bugs.gnu.org/30471 2020-01-18 Tobias Stoeckmann sed: handle very long input lines with R (tiny change) It is possible to trigger an out of bounds memory access when using the sed command R with an input file containing very long lines. The problem is that the line length of parsed file is returned as a size_t by ck_getdelim, but temporarily stored in an int and then converted back into a size_t. On systems like amd64, on which this problem can be triggered, size_t and int have different sizes. If the input line is longer than 2 GB (which is parseable on amd64 or other 64 bit systems), this means that the temporarily stored int turns negative. Converting the negative int back into a size_t will lead to an excessively large size_t, as the conversion leads to a lot of leading 1 bits. Eventually ck_fwrite is called with this huge size_t which in turn will lead to an out of bounds access on amd64 systems -- after all the parsed text was just a bit above 2 GB, not near SIZE_MAX. You can trigger this issue with GNU sed on OpenBSD like this: $ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > long.txt $ sed Rlong.txt /etc/fstab Segmentation fault (core dumped) I was unable to trigger the bug on a Linux system with glibc due to a bug in glibc's fwrite implementation -- it leads to a short write and sed treats that correctly as an error. * sed/execute.c (execute_program) [case 'R']: Declare result to be of type size_t, not int. * NEWS (Bug fixes): Mention it. This addresses https://bugs.gnu.org/39166 2020-01-18 Tobias Stoeckmann sed: handle very long execution lines (tiny change) If sed is called with an excessively long execution line, then it is prone to an out of bounds memory access. The problem is that the length of the execution line, which is a size_t, is temporarily stored in an int. This means that on systems which have a 64 bit size_t and a 32 bit int (e.g. linux amd64) an execution line which exceeds 2 GB will overflow int. If it is just slightly larger than 2 GB, the negative int value is used as an array index to finish the execution line string with '\0' which therefore triggers the out of bounds access. This problem is probably never triggered in reality, but can be provoked like this (given that 'e' support is compiled in): $ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > e-command.txt $ sed -f e-command.txt /etc/fstab Segmentation fault (core dumped) Also adjust another int/size_t conversion, even though it is a purely cosmetic change, because it can never be larger than 4096. * sed/execute.c (execute_program) [case 'e']: Declare cmd_length to be of type size_t, not int. Likewise for "n" just below. * NEWS (Bug fixes): Mention it. This addresses https://bugs.gnu.org/39165 2020-01-14 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.8 * NEWS: Record release date. doc: mention changes inherited through gnulib * NEWS (Improvements): Mention gnulib. 2020-01-11 Jim Meyering build: update gnulib to latest (for updated perl.m4) tests: remove valgrind-added diagnostics from stderr * testsuite/bug32271-2.sh: Filter out lines starting with "==". This avoids unwarranted failures on e.g., windows systems that would emit this to stderr: +==16281== error calling PR_SET_PTRACER, vgdb might block * testsuite/newline-dfa-bug.sh: Likewise. Reported by Vagelis Prokopiou on Windows subsystem for Linux (Debian 10) in https://lists.gnu.org/r/sed-devel/2020-01/msg00018.html and by Bruno Haible for mingw in https://lists.gnu.org/r/sed-devel/2020-01/msg00005.html sed: avoid MSVC build failure * sed/sed.c (usage): Remove unnecessary forward declaration. Reported by Bruno Haible in https://lists.gnu.org/r/sed-devel/2020-01/msg00007.html tests: avoid hpux11 test failure * testsuite/title-case.sh: Correct printf usage not to trigger unspecified behavior for '\(\)'. Instead, just use sed with its -r option so those parentheses don't need any backslash at all. Reported by Bruno Haible in https://lists.gnu.org/r/sed-devel/2020-01/msg00008.html 2020-01-09 Jim Meyering tests: handle missing perl Before this change, the two perl-requiring tests would fail on systems without perl. Now, they are skipped, as intended. * bootstrap.conf (gnulib_modules): Add perl. * testsuite/no-perl: New file. From coreutils. * testsuite/local.mk (EXTRA_DIST): Add it. (TESTSUITE_PERL): Include testsuite/ prefix. 2020-01-06 Jim Meyering maint: once again distribute .tar.gz files * configure.ac: Reenable distribution of gzip-compressed tarballs, on request from Jan Nieuwenhuizen in https://lists.gnu.org/r/sed-devel/2020-01/msg00013.html * NEWS (Release): Mention this. 2020-01-05 Jim Meyering maint: space-before-open-paren: avoid syntax-check error * sed/sed.c (selinux_support): Insert space before open-paren. build: update gnulib to latest (for thread-related test fixes) * .gitignore: Add /lib/inttypes.h. 2020-01-01 Jim Meyering maint: update all copyright year number ranges Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. maint: remove explicit mention of gnulib's threadlib module * bootstrap.conf (gnulib_modules): Remove explicit mention of threadlib. It was added back in 2012 to avoid a bootstrap failure; that explicit mention has long been unnecessary. 2019-09-05 Assaf Gordon sed: set correct umask on temporary files "sed -i" now creates temporary files with correct umask (limited to u=rwx). Previously sed would incorrectly set umask, and combined with mkostemp creating file with mode 0600, the result would be a file with permission mode 0. Reported by Dr N.W. Filardo : https://lists.gnu.org/r/sed-devel/2019-08/msg00000.html "The net effect is that this patch does not do what it says on the tin: it does not improve the security story at all. Things continue to function because the subsequent operations are via f*() APIs, which take the open file handle, and in particular fchmod() will put the bits back to something sensible. However, when running atop, for example, fuse-style filesystems which do not keep open descriptors to underlying files, this is catastrophic: the underlying file will have I_SRWXU of zero, and so the filesystem server will be unable to open the file for the fchmod() and that's the end of that." "fuse-overlayfs" is an example of a filesystem with such issues. This change was made in commit 5156c19b23c41f438bf8658e1b9a43a5ff136835 and was released in sed 4.2.1. * NEWS: Mention change. * sed/utils.c (ck_mkstemp): Set correct umask. 2019-06-02 Assaf Gordon tests: inplace-selinux.sh: skip if SELinux is not supported Avoid false-positive failure on system with SELinux enabled when SED is built without SELinux suppport. Reported by Jeffrey Walton in https://bugs.gnu.org/35997 . * testsuite/inplace-selinux.sh: Call 'sed --version' to check if sed was built with SELinux support before continuing with the test. 2019-06-02 Assaf Gordon sed: report SELinux status in --version output Report whether sed was built with SELinux support (determined at compile time), and whether it is enabled on this system (determined at runtime). Relates to https://bugs.gnu.org/35997 . * sed/sed.c (selinux_support): New function, print SELinux status to stdout. (main): Call selinux_support on --version. 2019-01-20 Jim Meyering build: ensure no VLA is used Cause developer builds to fail for any use of a VLA. VLAs (variable length arrays) limit portability. * configure.ac (nw): Remove -Wvla from the list of disabled warnings, thus enabling the warning when configured with --enable-gcc-warnings. (GNULIB_NO_VLA) Define, disabling use of VLAs in gnulib. This commit is functionally equivalent to coreutils' v8.30-44-gd26dece5d. build: update gnulib to latest 2019-01-01 Jim Meyering build: update gnulib to latest 2019-01-01 Assaf Gordon maint: update copyright dates for 2019 * all files: Run "make update-copyright". 2018-12-20 Jim Meyering maint: use https: in gnu mirror URL prefix, not http This appears in the generated release announcement message. * cfg.mk (url_dir_list): Use https: prefix, not http:. maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.7 * NEWS: Record release date. sed: fix \b DFA-bug in C locale Under some conditions, \b would mistakenly fail to match. E.g., this would mistakenly print "123-x" instead of "123": echo 123-x|LC_ALL=C sed 's/.\bx//' * NEWS (Bug fixes): Mention it * gnulib: Update to latest, for DFA regression fix. * testsuite/word-delim.sh: New file, to test for the dfa.c regression. * testsuite/local.mk (T): Add it. Reported by Jan Palus in https://lists.gnu.org/r/sed-devel/2018-12/msg00022.html 2018-12-19 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.6 * NEWS: Record release date. 2018-12-17 Jim Meyering build: update gnulib to latest for linux/sparc c-stack fix 2018-12-16 Jim Meyering build: update gnulib to latest for config.sub fix, etc. * gnulib: Update for build- and test-related fixes, including the config.sub fix for Solaris 10: https://lists.gnu.org/r/grep-devel/2018-12/msg00013.html * bootstrap: Update to latest. 2018-12-13 Assaf Gordon tests: fix obinary test error due to 'wc' spaces On BSD systems "wc -c" prefixes the count with spaces, leading to test error. Discussed in https://lists.gnu.org/r/sed-devel/2018-12/msg00014.html . * testsuite/obinary.sh: Remove leading spaces from 'wc -c' output. 2018-12-12 Jim Meyering build: update gnulib to latest 2018-12-09 Assaf Gordon tests: add text/binary mode tests This test is skipped on most platforms (which do not support O_BINARY and do not differentiate between TEXT and BINARY files). On cygwin, the test is skipped because default cygwin installations do not use O_TEXT/O_BINARY modes (unless file systems are mounted with special options). In practice this test will only run on native Windows (e.g. when compiled with mingw or msvc). Note that almost every other test will fail with such a binary (due to excesssive "\r" in the output). Thus it is recommended to explictly run just this test alone: make check SUBDIRS=. TESTS=testsuite/obinary.sh The script tries to cover all cases mentioned in https://bugs.gnu.org/25459 . See also https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html . * testsuite/obinary.sh: New test. * testsuite/local.mk (TESTS): Add new test. 2018-12-05 Assaf Gordon sed: fix -b/--binary mode under windows/mingw Discussed in https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html . * NEWS: Mention change. * bootstrap.conf: Add gnulib's binary-io module. * sed/sed.c (main): Set stdin/stdout to binary mode if needed. * sed/utils.c (ck_mkstemp): Explicitly set binary mode on file descriptor. It seems that on (non-cygwin) Windows the fdopen(3) call ignores the 'b' in the 'mode' argument - and the file was always opened in O_TEXT mode. Thus "--binary" was not working with "--inplace". 2018-12-05 Assaf Gordon maint: update .gitignore Following gnulib recent update. * lib/.gitignore, m4/.gitignore: Update file list. 2018-12-05 Assaf Gordon build: update autoconf version requirements Require version 2.64 to bootstrap. This is already required in practice by bootstrap.conf, but was not updated in configure.ac. After recent gnulib update, gnulib-tool would complain: $ ./gnulib/gnulib-tool ./gnulib/gnulib-tool: *** minimum supported autoconf version is 2.63.\ Try adding AC_PREREQ([2.63]) to your configure.ac. ./gnulib/gnulib-tool: *** Stop. Hence this update. * configure.ac: Require autoconf version 2.64 or later. 2018-12-05 Assaf Gordon gnulib: update to latest (for dfa improvements) 2018-11-22 Jannick sed: fix memory leakage under lint The NULL-initialized char string in_place_extension is free'ed everytime it is redefined (using xstrdup) and at program exit with any return code. See: https://lists.gnu.org/r/sed-devel/2018-11/msg00005.html * sed/sed.c (main, cleanup): Free 'in_place_extension' if running with lint. 2018-11-01 Assaf Gordon build: rename stdin local variable Under windows/msys2, defining 'bool stdin' variable results in compilation errors: ../sed/execute.c:1252:8: error: expected declaration specifiers or '...' before numeric constant bool stdin = (input->fp && fileno (input->fp) == 0); ^~~~~ Reported by Jannick in https://lists.gnu.org/archive/html/sed-devel/2018-10/msg00019.html . * sed/execute.c (debug_print_input): Rename 'stdin' to 'is_stdin'. 2018-10-28 Jim Meyering maint: fix a syntax-check nit * sed/compile.c (MISSING_FILENAME): Insert a space before "(", and in a following comment. 2018-10-27 Jim Meyering tests: avoid unnecessary dependency on Data::Dump * testsuite/debug.pl: Don't use Data::Dump. Unnecessary. That package, perl-Data-Dump, appear not to be default-installed on at least a Fedora 29 beta system. sed: avoid UMR in --debug code path * sed/debug.c (debug_print_function) [b, t, T]: For a b, t or T command with no LABEL, do not access uninitialized memory. I.e., print the label name only when there is one. 2018-10-24 Assaf Gordon sed: add --debug feature $ seq 3 | sed --debug -e 's/./--&--/ ; 2d' SED PROGRAM: s/./--&--/ 2 d INPUT: 'STDIN' line 1 PATTERN: 1 COMMAND: s/./--&--/ MATCHED REGEX REGISTERS regex[0] = 0-1 '1' PATTERN: --1-- COMMAND: 2 d END-OF-CYCLE: --1-- INPUT: 'STDIN' line 2 PATTERN: 2 COMMAND: s/./--&--/ MATCHED REGEX REGISTERS regex[0] = 0-1 '2' PATTERN: --2-- COMMAND: 2 d END-OF-CYCLE: INPUT: 'STDIN' line 3 PATTERN: 3 COMMAND: s/./--&--/ MATCHED REGEX REGISTERS regex[0] = 0-1 '3' PATTERN: --3-- COMMAND: 2 d END-OF-CYCLE: --3-- Discussed in https://lists.gnu.org/r/sed-devel/2018-07/msg00006.html and https://lists.gnu.org/r/sed-devel/2018-10/msg00007.html . * NEWS: Mention new option. * doc/sed.texi (Program options): Mention new option. * sed/debug.c: New unit with debug printing functions. * sed/sed.h (debug_print_command, debug_print_char, debug_print_program, debug): Declare functions and global variable. (struct sed_cmd): Add label_name member variable. * sed/compile.c (compile_program): Save the label's name. (cleanup_program_filenames): extracted function to free filenames. (check_final_program) Don't delete the filenames, instead move it to ... (finish_program) ... here. * sed/execute.c (debug_print_end_of_cycle, debug_print_input, debug_print_line): New debug functions (cannot be defined in debug.c as execute's structures are private). (execute_program, process_files): Call debug functions. * sed/sed.c: (DEBUG_OPTION): New option for getoptlong. (debug): New global variable. (usage): Mention new option. (main): Process new option and call debug functions if needed. * testsuite/debug.pl: New tests. * testsuite/local.mk (T): Add new tests. 2018-10-19 Assaf Gordon sed: change internal storage for 'R' command For the 'R' command, keep a pointer to 'struct output' instead of just to FILE*. No change in functionality. This will help future debug code to know the filename of the associated file. As a side effect, fix invalid code in W/w execution: The previous code checked for 'cur_cmd->x.fp' (which was related to R command, not W/w). Since 'x' is a union, 'x.fp' was not relevant to the 'x.outf' (which is the struct associted with W/w commands). * sed/sed.h (struct sed_cmd): Replace 'FILE*' with 'struct *output' for R command. * sed/compile.c (compile_program): Adjust as needed. * sed/execute.c (execute_program): Adjust as needed. 2018-10-18 Assaf Gordon sed: reject r/R/w/W commands without a filename w/W (and s///w) commands Without a filename would print a confusing error message: $ sed w sed: couldn't open file : No such file or directory While r/R commands with empty file name were a silent no-op. With this change, sed programs with empty filename are rejected with a clear error: $ sed 's/1/2/w' sed: -e expression #1, char 7: missing filename in r/R/w/W commands $ sed r sed: -e expression #1, char 1: missing filename in r/R/w/W commands * NEWS: Mention change. * sed/compile.c (get_openfile): Exit with an error message if filename is missing. (compile_program): Same for 'r' command code. * testsuite/missing-filename.sh: New test. * testsuite/local.mk (T): Add new test. 2018-10-12 Clint Adams (tiny change) doc: use @key{TAB} in texinfo Discussed in https://bugs.gnu.org/22636 . * doc/sed.texi: Use '@kbd{@key{TAB}}' instead of '@kbd{tab}' to improve rendering in HTML and info formats. 2018-10-12 Bjarni Ingi Gislason (tiny change) doc: sed.x: some formatting corrections Reported in https://bugs.gnu.org/30479 . doc/sed.x: Change a HYPHEN-MINUS (code 0x55, 2D) to a dash (\-, minus) if it matches " -[:alpha:]" or \(aq-[:alpha:] (for options); Use a pronoun instead of a repeated noun. 2018-10-12 Assaf Gordon sed: update authors * sed/sed.c (AUTHORS): Update. * doc/sed.texi (@author): Update. 2018-08-23 Assaf Gordon maint: remove REG_PERL code Perl-regexp syntax (PCRE) in GNU Sed is shelved indefinitely. See https://bugs.gnu.org/22801 , https://bugs.gnu.org/22647 . Remove all (unused) REG_PERL related code. * sed/sed.c, sed/sed.h, sed/regexp.c, sed/compile.c: Remove REG_PERL code. 2018-08-23 Assaf Gordon maint: remove EXPERIMENTAL_DASH_N_OPTIMIZATION code The optimization was buggy and was never enabled in a released version. * sed/execute.c: Remove EXPERIMENTAL_DASH_N_OPTIMIZATION code. 2018-08-23 Assaf Gordon maint: replace street address with URL in license text * basicdefs.h, sed/compile.c, sed/execute.c, sed/mbcs.c, sed/regexp.c, sed/sed.c, sed/sed.h, sed/utils.c, sed/utils.h, testsuite/get-mb-cur-max.c, testsuite/test-mbrtowc.c (GPL License notice): Replace FSF's office address with 'https://www.gnu.org/licenses/'. 2018-08-23 Assaf Gordon build: use system's native sed during build Use the locally built sed binary only during 'check*' targets, but not during build/installation targets - it might be buggy, or contain temporary debugging/ASAN features that can break the build. See: https://lists.gnu.org/r/sed-devel/2018-08/msg00013.html . * cfg.mk (PATH): Add the 'sed' directory to the PATH only if this is a 'check' target. 2018-08-08 Assaf Gordon sed: mark function as attribute(malloc) Suggested by gcc-8.2.0. * sed/compile.c (read_label): Mark as _GL_ATTRIBUTE_MALLOC. 2018-08-08 Assaf Gordon maint: add undefined-behavior build target Using gcc-specific options, a recent gcc is required. build with: make build-ubsan CC=gcc-8.2 * cfg.mk (build-ubsan): New target. 2018-08-08 Assaf Gordon maint: add address-sanitizer build target use 'make build-asan' to rebuild sed with gcc's address sanitizer. * cfg.mk (build-asan): New target. 2018-08-08 Assaf Gordon gnulib: update to latest (for regex memory leaks) Specifically for gnulib commits 66b99e5259,c5e76a9560. see https://lists.gnu.org/r/bug-gnulib/2018-07/msg00127.html sed: fix memory leak * sed/regexp.c (match_regex): Free the previously allocated regex struct before re-building the regex during program execution. 2018-08-08 Assaf Gordon sed: free allocated memory under lint, remove DEBUG_LEAKS Under normal operation there is no need for explicit freeing, as all memory will be released when sed terminates. During development (and specifically, valgrind and address-sanitizing) enabling lint prevents false-positive warnings about memory leaks. Lint mode can be enabled with CFLAGS="-Dlint". It is also automatically enabled by default when building from git (as opposed to tarball). For consistency all instances of "#ifdef DEBUG_LEAKS" are replaced with "#ifdef lint". * sed/sed.h (struct subst): Add member variable to keep the address of allocated buffer in compile.c:setup_replacement(). (release_regex): Add declaration. (finish_program): Function now takes an argument: the sed program vector. * sed/sed.c (main): Adjust call to finish_program. * sed/compile.c (finish_program): Release program allocations. (setup_replacement): Remember the allocated buffer address. (compile_program): Free temporary array in 'y' command. * sed/execute.c (execute_program): Free allocated but unused buffer in 'R' command when reaching EOF. * sed/regexp.c (release_regex): Remove 'static', free the allocated dfa struct. * sed/utils.c (panic): Free open files linked-list elements. 2018-08-08 Assaf Gordon sed: do not close stderr on exit Not needed, and prevents leak-sanitizing from working. * sed/utils.c (ck_fclose): Do not close stderr. 2018-08-07 Jim Meyering doc: insert two missing words doc/sed.texi (BRE vs ERE): Insert "with a". 2018-08-04 Assaf Gordon maint: syntax-check: prohibit-operator-at-eol * cfg.mk (sc_prohibit_operator_at_end_of_line): Copied from coreutils' cfg.mk. * sed/execute.c (do_subst,execute_program): Adjust as needed. maint: syntax-check: fail-0 and fail-1 in shell script tests * cfg.mk (prohibit_fail_0,sc_prohibit_and_fail_1): Copied from coreutils' cfg.mk * testsuite/*.sh: Remove 'fail=0'. maint: syntax-check: add various test-related checks * cfg.mk (sc_prohibit_env_returns, sc_prohibit_perl_hash_quotes, sc_prohibit_verbose_version, sc_prohibit_framework_failure, sc_prohibit_test_backticks, sc_prohibit_test_empty): Copied from coreutils' cfg.mk. maint: syntax-check: add sc_prohibit_strncmp * cfg.mk (sc_prohibit_strncmp): Copied from coreutils' cfg.mk. * basicdefs.h (STREQ,STREQ_LEN,STRPREFIX): Copied from coreutils' system.h. * sed/execute.c (translate_mb): Replace strncmp with STREQ_LEN. maint: syntax-check: add sc_gitignore_missing * cfg.mk (sc_gitignore_missing): Copied from coreutils' cfg.mk. * .gitignore: Add missing files. maint: syntax-check: prohibit-form-feed * cfg.mk (sc_prohibit-form-feed): Copied from coreutils' cfg.mk. * sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.h, sed/utils.c: Remove form feed characters. maint: syntax-check: space-before-open-paren * cfg.mk (sc_space_before_open_paren): Copy target from coreutils' cfg.mk. * basicdefs.h, sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.c, sed/utils.c: Add space before parentheses. maint: syntax-check: fix tab indentation * sed/execute.c (open_next_file): Replace tab with spaces. 2018-08-03 Assaf Gordon sed: replace myname with gnulib's progname * compile.c (bad_prog): Use program_name instead of myname. * execute.c (open_next_file): Same. * utils.c (panic): Same. (myname): Remove variable. * sed.c (usage): Use program_name instead of myname. (main): Call set_program_name once, don't set variables directly. sed: replace ck_realloc with gnulib's xrealloc/xnrealloc * basicdefs.h (REALLOC): Call xnrealloc instead of ck_realloc. * sed/utils.c (ck_realloc): Remove. 2018-08-03 Assaf Gordon sed: replace MALLOC/ck_malloc with gnulib's XCALLOC ck_malloc zero'd out the allocated buffer, thus replaced with xcalloc/xzalloc (not xmalloc). * basicdefs.h (MALLOC): Remove. * utils.c (ck_malloc): Remove. * sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.c: Replace MALLOC with XCALLOC; ck_malloc with xzalloc. 2018-08-03 Assaf Gordon sed: replace ck_strdup with gnulib's xstrdup * sed/utils.{c,h} (ck_strdup): Remove. * sed/compile.c, sed/execute.c, sed/sed.c: Replace ck_strdup with xstrdup. sed: replace ck_memdup with gnulib's xmemdup * bootstrap.conf: Add gnulib's xalloc module. * basicdefs.h (MEMDUP): Remove. * sed/utils.{c,h} (ck_memdup): Remove. 2018-08-03 Assaf Gordon sed: fix heap buffer overflow from multiline EOL regex optimization sed would access invalid memory when matching EOF combined with s///n flag: $ yes 0 | fmt -w 40 | head -n2 | valgrind sed 'N;s/$//2m' ==13131== Conditional jump or move depends on uninitialised value(s) ==13131== at 0x4C3002B: memchr (vg_replace_strmem.c:883) ==13131== by 0x1120BD: match_regex (regexp.c:286) ==13131== by 0x110736: do_subst (execute.c:1101) ==13131== by 0x1115D3: execute_program (execute.c:1591) ==13131== by 0x111A4C: process_files (execute.c:1774) ==13131== by 0x112E1C: main (sed.c:405) ==13131== ==13131== Invalid read of size 1 ==13131== at 0x4C30027: memchr (vg_replace_strmem.c:883) ==13131== by 0x1120BD: match_regex (regexp.c:286) ==13131== by 0x110736: do_subst (execute.c:1101) ==13131== by 0x1115D3: execute_program (execute.c:1591) ==13131== by 0x111A4C: process_files (execute.c:1774) ==13131== by 0x112E1C: main (sed.c:405) ==13131== Address 0x55ec765 is 0 bytes after a block of size 101 alloc'd ==13131== at 0x4C2DDCF: realloc (vg_replace_malloc.c:785) ==13131== by 0x113BA2: ck_realloc (utils.c:418) ==13131== by 0x10E682: resize_line (execute.c:154) ==13131== by 0x10E6F0: str_append (execute.c:165) ==13131== by 0x110779: do_subst (execute.c:1106) ==13131== by 0x1115D3: execute_program (execute.c:1591) ==13131== by 0x111A4C: process_files (execute.c:1774) ==13131== by 0x112E1C: main (sed.c:405) ==13131== The ^/$ optimization code added in v4.2.2-161-g6dea75e called memchr() using 'buflen', ignoring the value of 'buf_start_offset' (which, if not zero, reduces the number of bytes available for the search). Reported by bugs@feusi.co (bug#32271) in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html . * NEWS: Mention the fix. * sed/regexp.c (match_regex): Use correct buffer length in memchr(). * testsuite/bug-32271-2.sh: Test using valgrind. * testsuite/local.mk (T): Add new test. 2018-08-03 Assaf Gordon sed: fix extraneous NUL in s///n command Under certain conditions sed would add an extraneous NUL: $ echo 0 | sed -e 's/$/a/2' | od -tx1 -An 30 00 0a This would happen when the regex is an empty (zero-length) match at the end of the line (e.g. '$' and 'a*$') and the substitute number flag ('n' in s///n) is higher than the number of actual matches (multiple EOL matches are possible with multiline match, e.g. 's/$/a/3m'). Details: The comment at the top of 'execute.c:do_subst()' says: /* The first part of the loop optimizes s/xxx// when xxx is at the start, and s/xxx$// */ Which refers to lines 1051-3: 1051 /* Copy stuff to the left of this match into the output string. */ 1052 if (start < offset) 1053 str_append(&s_accum, line.active + start, offset - start); The above code appends text to 's_accum' but does *not* update 'start'. Later on, if the s/// command includes 'n' flag, and if 'matched == 0' (an empty match), this comparison will be incorrect: 1081 if (start < line.length) 1082 matched = 1; Will in turn will set 'matched' to 1, and the 'str_append' call that follows (line 1087) will append an additional character. Because the empty match is EOL, the appended character is NUL. More examples that trigger the bug: echo 0 | sed -e 's/a*$/X/3' printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/4m' Examples that do not trigger the bug: # The 'a*' empty regex matches at the beginning of the line (in # addition to the end of the line), and the optimization in line # 1052 is skipped. echo 0 | sed -e 's/a*/X/3' # There are 3 EOLs in the pattern space, s///3 is not too large. printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/3m' This was discovered while investigating bug#32271 reported by bugs@feusi.co in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html . * NEWS: Mention the fix. * sed/execute.c (do_subst): Update 'start' as needed. * testsuite/bug-32271-1.sh: New test. * testsuite/local.mk (T): Add test. 2018-07-26 Assaf Gordon sed: do not flush output stream unless in unbuffered mode Previously sed would explicitly flush the output after every output line, except if the output was stdout in unbuffered mode. In practice this was equivalent to forcing line-buffering, and was espcially was noticable with "sed -i" (where the output is a temporary file). With this change, explicit flushing only happens with "sed -u", regardless of the type of output file, making "sed -i" much faster. This change also affect other write commands such as 'w'/'W' and 's///w'. Reported by Vidar Holen in https://lists.gnu.org/r/bug-sed/2018-07/msg00014.html . * NEWS: Mention this. * sed/execute.c (flush_output): Never flush output unless in unbuffered mode, regardless of which file it is. 2018-07-25 Bernhard Voelker doc: tiny fix * doc/sed.texi (Regexp Addresses): Remove the word 'two': there are 3 examples following. 2018-07-08 Assaf Gordon sed: fix heap buffer overflow from invalid references Under certain conditions sed would access invalid memory based on the requested back-reference (e.g. "s//\9/" would access the 9th element in the regex registers without checking it is at least 9 element in size). The following examples would trigger valgrind errors: seq 2 | valgrind sed -e '/^/s///p ; 2s//\9/' seq 2 | valgrind sed --posix -e '/2/p ; 2s//\9/' Reported by bugs@feusi.co in https://lists.gnu.org/r/bug-sed/2018-07/msg00004.html . * NEWS: Mention the bugfix. * sed/execute.c (append_replacement): Check number of allocated regex replacement registers before accessing the array. * sed/testsuite/bug32082.sh: Test sed for this behaviour under valgrind. * sed/testsuite/local.mk (T): Add new test. 2018-03-31 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.5 * NEWS: Record release date. 2018-03-30 Jim Meyering tests: port inplace-selinux.sh to Centos7 and Fedora 25 * testsuite/inplace-selinux.sh: Set the user (-u) portion of the security context rather than the type (-t), and use system_u and user_u, which appear to be reliably available, since some coreutils's tests use those. Assaf Gordon reported that this test failed on those systems in https://lists.gnu.org/r/sed-devel/2018-03/msg00018.html Also, correct copyright date to include this year. 2018-03-28 Jim Meyering build: update gnulib to latest maint: fix preceding change so "make dist" works * testsuite/local.mk (EXTRA_DIST): Also remove runtest, here. 2018-03-27 Jim Meyering tests: don't run $(check_PROGRAMS) directly testsuite/runtest was not portable to Solaris 10 /bin/sh, but was used only to keep automake test machinery from running two compiled binaries stand-alone. We can inhibit that more cleanly by removing those two files from the list of TESTS to run and then just remove the offending script. * testsuite/local.mk (LOG_COMPILER): Set to false. (TESTS): Don't put $(check_PROGRAMS) on this list. * testsuite/runtest: Remove file. Nelson H.F. Beebe reported the Solaris 10 /bin/sh failure. tests: port to solaris 10's /bin/sh * testsuite/runtest (test): Accommodate Solaris 10 /bin/sh by sourcing init.sh, as is already done in many other shell scripts in this directory, to handle (or skip test upon) use of this construct: ${1##*/}. Reported by Nelson H.F. Beebe. 2018-03-23 Jim Meyering maint: remove old definition of UNUSED: use _GL_UNUSED instead * sed/execute.c (UNUSED): Remove macro definition. (read_always_fail): Use _GL_UNUSED instead. 2018-03-23 Assaf Gordon sed: reject RE searches on buffers larger than INT_MAX Sed uses 'size_t' internally, but gnulib's re_search uses 'signed int'. If the buffer is larger than INT_MAX, reject it (panic, exit code 4). The test is marked 'very expensive' and skipped by default (due to creating a 2GB input file). To run it, use: make check SUBDIRS=. RUN_VERY_EXPENSIVE_TESTS=yes \ TESTS=testsuite/regex-max-int.sh Reported by YushiOMOTE in https://bugs.gnu.org/30520 . * NEWS: Mention this. * Makefile.am (check-expensive, check-very-expensive): New targets. * init.cfg (expensive_, very_expensive_): Copied from coreutils. * sed/regexp.c (match_regex): Check buffer length and panic if too large. * testsuite/regex-max-int.sh: New test. * testsuite/local.mk (T): Add new test. 2018-03-22 Jim Meyering maint: ignore more artifacts * .gitignore: Also ignore sed-*.tar.xz.sig files. * lib/.gitignore: Ignore more gnulib-provided sources. build: update gnulib to latest * gnulib: Update submodule. * bootstrap: Update from gnulib. * testsuite/init.sh: Likewise. 2018-03-14 0xddaa <0xddaa@gmail.com> sed: treat '\x5c' as literal backslash Sed incorrectly treated '\x5c' (ASCII 92, backslash) as an escape character. Old behavior: $ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/' z New behavior: $ echo z | sed -E 's/(z)/\x5c1/' \1 Reported in https://bugs.gnu.org/30794. * NEWS: Mention bug fix. * sed/compile.c (normalize_text): Pass backslash as literal character. * testsuite/misc.pl: Add tests. 2018-03-02 Mike Frysinger (tiny change) sed: improve sandbox description in --help output * sed/sed.c (usage): Add "(disable e/r/w commands)" to --sandbox text. 2018-02-15 Jim Meyering doc: small improvements * doc/sed.texi: s/only accepts/accepts only/. (BRE vs ERE): Tweak ERE and BRE definitions, and convert a passive-voice sentence to active voice. Insert a comma. 2018-02-13 Assaf Gordon doc: change URLs to https * doc/config.texi, doc/sed-dummy.1, doc/sed.texi: Change http to https for GNU URLs. External links (e.g. sed.sf.net, autsingroup.org) currently do not support https and kept as-is. maint: change http to https in license headers * all files: Change http to https for all licenses URL. sed: use https in usage screen contact information * sed/sed.c (contact): Change http to https. 2018-02-04 Jim Meyering build: update gnulib to latest * gnulib: This fixes a syntax-check failure that was due to the new use of "/proc/filesystems" in init.cfg. 2018-01-31 Jim Meyering maint: make cfg.mk slightly more generic * cfg.mk (announcement_Cc_): Don't hard-code "sed-devel@". Use "$(PACKAGE)-devel@" instead. 2018-01-09 Jakub Martisko sed: with --in-place and selinux, use the symlink's context When editing a file in place, the SELinux context (if exists) should be based on the link instead of the target file itself. --follow-symlinks option remains unchanged. Bug reported by Jakub Jelen, fix proposed by Petr Lautrbach, discussed in https://lists.gnu.org/r/sed-devel/2017-12/msg00000.html * NEWS: Mention this. * sed/execute.c (open_next_file): Use lgetfilecon (instead of getfilecon) to get the context of the symlink instead of the target file. * testsuite/inplace-selinux.sh: Test the above change. * testsuite/local.mk (T): Add new test file. * init.cfg (require_selinux_): Copied from coreutils, skip the test if selinux is not available. 2018-01-02 Assaf Gordon maint: update gnulib and copyright dates for 2018 * gnulib: Update to latest. * all files: Run "make update-copyright". 2017-11-22 Jim Meyering maint: update to work with GCC7's -Werror=implicit-fallthrough= * sed/sed.h (FALLTHROUGH): Define. * sed/execute.c (execute_program): Use new FALLTHROUGH macro in place of each comment. * sed/compile.c (mark_subst_opts, compile_program): Likewise. 2017-10-28 Assaf Gordon doc: fix incorrect line-wrapping example Reported by Bamber Ward in https://bugs.gnu.org/28140 . * doc/sed.texi (Line length adjustment): Fix sed script; Rewrite example to long script with inlined comments; Remove second example. 2017-10-17 Assaf Gordon doc: fix missing slash in example Reported by Nick Chambers in https://bugs.gnu.org/28881 . * doc/sed.texi (Invoking sed::overview): Add missing slash in 's/hello/world/' command. 2017-10-05 Assaf Gordon tests: skip utf8-ru test if required locale is not found Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 . * init.cfg (require_ru_utf8_locale_): New function. * testsuite/utf8-ru.sh: Skip test is required locale was not found. 2017-10-05 Assaf Gordon build: require texinfo v6.1 for development builds For development builds (using ./bootstrap), require texinfo version 6.1 or later (Same as GNU coreutils). Sed's manual uses texinfo's @U{} command, introduced in texinfo-6.0. This should not affect building from released tarballs, as the info manual is pre-built and pre-packaged in the tar archive. Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 . * bootstrap.conf: Require makeinfo 6.1 (up from 4.13). 2017-10-05 Assaf Gordon tests: remove non-functional regex C tests These C-based tests were not updated since 2004, and were not included by default. Enabling them with "./configure --enable-regex-tests" resulted in false positives: https://bugs.gnu.org/28665 , https://bugs.gnu.org/25381 . This patch removes them completely. * configure.ac: Remove "--with-regex-tests" option. * local.mk (check_PROGRAMS): Remove tests. * bug-regex10.c, bug-regex11.c, bug-regex12.c, bug-regex13.c, bug-regex14.c, bug-regex15.c, bug-regex16.c, bug-regex21.c, bug-regex27.c, bug-regex28.c, bug-regex7.c, bug-regex8.c, bug-regex9.c, runptests.c, runtests.c, tst-boost.c, tst-pcre.c, tst-regex.c, tst-regex2.c, tst-rxspencer.c: Remove files. 2017-05-13 Jim Meyering maint: stop using @acronym{...} in texinfo sources * doc/sed.texi, doc/config.texi: Remove all uses of @acronym{...}, per recommendation by Karl Berry. * cfg.mk (local-checks-to-skip): Remove exemption, enabling the @acronym{-prohibiting syntax-check rule. gnulib: update to latest 2017-05-10 Assaf Gordon tests: fix various misspellings Reported by Jim Meyering in https://lists.gnu.org/archive/html/sed-devel/2017-05/msg00001.html . * testsuite/BOOST.tests, testsuite/posix-mode-bad-ref.sh, testsuite/runptests.c, testsuite/runtests.c, testsuite/tst-pcre.c: Fix misspellings. 2017-05-10 Assaf Gordon tests: remove the 'Makefile.tests' used for old-style tests * testsuite/Makefile.tests: Removed. * testsuite/runtest: Remove code to run old-style tests. * testsuite/local.mk (EXTRA_DIST): Remove MAkefile.tests. tests: convert "noeolw" test to new framework * testsuite/misc.pl (Tests): Add noeolw. * testsuite/noeolw.good: Delete file. * testsuite/noeolw.1good: Likewise. * testsuite/noeolw.2good: Likewise * testsuite/noeolw.inp: Likewise. * testsuite/noeolw.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "noeolw". 2017-05-10 Assaf Gordon tests: convert "mac-mf" test to new framework All input files (mac-mf.{inp,good,sed}) are large and are kept as-is. * testsuite/mac-mf.sh: New test. * testsuite/local.mk (SEDTEST): Remove mac-mf. (T): Add mac-mf.sh 2017-05-10 Assaf Gordon tests: convert "newjis" test to new framework * testsuite/newjis.sh: New test. * testsuite/newjis.{good,inp,sed}: Removed. * testsuite/local.mk (EXTRA_DIST): Remove unneeded files. (SEDTEST): Remove newjis. (T): Add newjis.sh tests: convert "8to7" test to new framework * testsuite/8to7.sh: New test. * testsuite/8to7.{good,inp,sed}: Removed. * testsuite/local.mk (EXTRA_DIST): Remove unneeded files. (SEDTEST): Remove 8to7. (T): Add 8to7.sh 2017-05-10 Assaf Gordon tests: convert "madding" test to new framework All input files (madding.{inp,good,sed}) contain very long lines and are kept as-is (to avoid failing 'make syntax-check'). * testsuite/madding.sh: New test. * testsuite/local.mk (SEDTEST): Remove madding. (T): Add madding.sh 2017-05-10 Assaf Gordon tests: convert "badenc" test to new framework * testsuite/badenc.sh: New test. * testsuite/distrib.{good,inp,sed}: Removed. * testsuite/local.mk (EXTRA_DIST): Remove unneeded files. (SEDTEST) Remove badenc. (T): Add badenc.sh 2017-05-10 Assaf Gordon tests: convert "distrib" test to new framework The input file 'distrib.inp' has lines wider than 80 characters, and is kept as a separate file. The existing file 'distrib.sh' (which contained the same sed program as in 'distrib.sed') was never used, and is completely rewritten. * testsuite/distrib.sh: Rewritten as a new test. * testsuite/distrib.{good,sed}: Removed. * testsuite/local.mk (EXTRA_DIST): Remove unneeded files. (SEDTEST) Remove distrib. (T): Add distrib.sh 2017-05-10 Assaf Gordon tests: convert "inplace-hold" test to new framework * testsuite/inplace-hold.sh: New test. * testsuite/local.mk (SEDTEST) Remove inplace-hold. (T): Add inplace-hold.sh. tests: convert "stdin" test to new framework * testsuite/stdin.sh: New test. * testsuite/local.mk (SEDTEST) Remove stdin. (T): Add stdin.sh. tests: convert "eval" test to new framework * testsuite/eval.sh: New test. * testsuite/eval.{sed,inp,good}: Removed. * testsuite/local.mk (EXTRA_DIST): Remove unneeded files. (SEDTEST) Remove eval.sed. (T): Add eval.sh. 2017-05-10 Assaf Gordon tests: convert 'bsd' test to new framework The 'bsd.sh' is a long shell script (imported from NetBSD) which runs multiple tests, then compares the output to 'bsd.good'. The 'bsd.sh' is not modified. Instead, a new script 'bsd-wrapper.sh' is created to work with the newer init.sh-style testing framework. * testsuite/bsd-wrapper.sh: New test. * testsuite/local.mk (SEDTESTS): Remove old test. (T): Add new test. 2017-05-10 Assaf Gordon tests: convert 'xemacs' test to new framework Keep input/expected output files (xemacs.inp/xemacs.good) as they contain lines longar than 80 characters (which will fail 'make syntax-check'). * testsuite/xemacs.sed: Remove. * testsuite/xemacs.sh: New test. * testsuite/local.mk (EXTRA_DIST): Remove xemacs.sed. (SEDTESTS): Remove old xemacs test. (T): Add new xemacs.sh test. 2017-05-10 Assaf Gordon tests: convert 'help' test to new framework The 'help' test was implemented directly in 'Makefile.tests' - no external files to remove/adapt. * testsuite/help.sh: New tests. * testsuite/local.mk (SEDTESTS): Remove old test. (T): Add new test. 2017-05-10 Assaf Gordon tests: convert 'uniq' test to new framework Add a simple init.sh-stype wrapper to run the 'uniq' tests. 'uniq.sed' is a valid stand-alone script - keep it as-is. The input/expected output files (uniq.inp/uniq.good) are large - keep them as well instead of embedding them in the shell script. * testsuite/uniq.sh: New test. * testsuite/local.mk (SEDTESTS): Remove uniq.sed (T): Add uniq.sh. 2017-05-10 Assaf Gordon tests: convert 'utf8-*' tests to new framework Merge utf8-{1,2,3,4} into utf8-ru.sh script. * testsuite/utf8-ru.sh: New test. * testsuite/utf8-1.inp, testsuite/utf8-1.good, testsuite/utf8-1.sed, testsuite/utf8-2.inp, testsuite/utf8-2.good, testsuite/utf8-2.sed, testsuite/utf8-3.inp, testsuite/utf8-3.good, testsuite/utf8-3.sed, testsuite/utf8-4.inp, testsuite/utf8-4.good, testsuite/utf8-4.sed: Removed files. * testsuite/local.mk (EXTRA_DIST): Remove deleted inp/sed/good files. (SEDTESTS): Remove utf8-{1,2,3,4} tests. (T): Add new test. 2017-05-10 Assaf Gordon tests: convert "dc" test to new framework The 'dc.sed' program (implementing dc calculator in sed) is kept as-is. The input (dc.inp) and expected output (dc.good) are merged into the new shell script. The old-style test executed two calculations at once (Easter 2002 and square root of 2). They are now separated into two invocations of 'dc.sed'. * testsuite/dc.sh: New test. * testsuite/dc.inp, testuite/dc.good: Removed. * testsuite/local.mk (EXTRA_DIST): Remove inp/good files. (SEDTEST): Remove dc.sed. (T): Add dc.sh. 2017-05-10 Assaf Gordon tests: convert "binary/2/3" tests to new framework The sed scripts (binary.sed, binary2.sed, binary3.sed) are large and could function as stand-alone programs - there are kept as separate files. The input/expected output are removed, and a new script is added to run the tests. * testsuite/binary.sh: New script. * testsuite/binary.inp, testsuite/binary.good: Removed. * testsuite/local.mk (EXTRA_DIST): Remove inp/good files. (SEDTEST): Remove tests. (T): Add new test. 2017-05-10 Assaf Gordon tests: convert "8bit" test to new framework The input (8bit.inp) and expected output (8bit.good) contains non-ascii octets, and are kept as-is. The sed program is converted to newer init.sh-style tests. * testsuite/8bit.sed: Removed. * testsuite/8bit.sh: New test. * testsuite/local.mk (EXTRA_DIST): Remove 8bit.sed. (SEDTEST) Remove 8bit.sed. (T): Add 8bit.sh. 2017-05-10 Assaf Gordon tests: new function remove_cr_inplace in init.cfg When testing on Windows OSes, remove any CR (\r) characters from output files (before comparing them to expected output). This functionality was used by the old-style tests, embedded directly in 'Makefile.tests'. * init.cfg: (remove_cr_inplace): New function. 2017-05-10 Assaf Gordon tests: convert "subwrite" test to new framework * testsuite/misc.pl (Tests): Add subwrite. * testsuite/subwrite.inp: Likewise. * testsuite/subwrite.sed: Likewise. * testsuite/subwrt1.good: Delete file. * testsuite/subwrt2.good: Delete file. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "subwrite". tests: convert "writeout" test to new framework * testsuite/misc.pl (Tests): Add writeout. * testsuite/writeout.inp: Likewise. * testsuite/writeout.sed: Likewise. * testsuite/wrtout1.good: Delete file. * testsuite/wrtout2.good: Delete file. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "writeout". tests: convert 'quiet'/'file' tests to new framework * testsuite/local.mk: (SEDTESTS): Remove tests. * testsuite/misc.pl: Add tests. tests: convert "brackets" test to new framework * testsuite/misc.pl (Tests): Add brackets. * testsuite/brackets.good: Delete file. * testsuite/brackets.inp: Likewise. * testsuite/brackets.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "brackets". tests: convert "sep" test to new framework * testsuite/misc.pl (Tests): Add sep. * testsuite/sep.good: Delete file. * testsuite/sep.inp: Likewise. * testsuite/sep.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "sep". tests: convert "readin" test to new framework * testsuite/misc.pl (Tests): Add readin. * testsuite/readin.good: Delete file. * testsuite/readin.inp: Likewise. * testsuite/readin.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "readin". tests: convert "numsub" test to new framework * testsuite/misc.pl (Tests): Add numsub. * testsuite/numsub.good: Delete file. * testsuite/numsub.inp: Likewise. * testsuite/numsub.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "numsub". tests: convert "numsub5" test to new framework * testsuite/misc.pl (Tests): Add numsub5. * testsuite/numsub5.good: Delete file. * testsuite/numsub5.inp: Likewise. * testsuite/numsub5.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "numsub5". tests: convert "numsub4" test to new framework * testsuite/misc.pl (Tests): Add numsub4. * testsuite/numsub4.good: Delete file. * testsuite/numsub4.inp: Likewise. * testsuite/numsub4.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "numsub4". tests: convert "numsub3" test to new framework * testsuite/misc.pl (Tests): Add numsub3. * testsuite/numsub3.good: Delete file. * testsuite/numsub3.inp: Likewise. * testsuite/numsub3.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "numsub3". tests: convert "numsub2" test to new framework * testsuite/misc.pl (Tests): Add numsub2. * testsuite/numsub2.good: Delete file. * testsuite/numsub2.inp: Likewise. * testsuite/numsub2.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "numsub2". tests: convert "newline-anchor" test to new framework * testsuite/misc.pl (Tests): Add newline-anchor. * testsuite/newline-anchor.good: Delete file. * testsuite/newline-anchor.inp: Likewise. * testsuite/newline-anchor.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "newline-anchor". tests: convert "middle" test to new framework * testsuite/misc.pl (Tests): Add middle. * testsuite/middle.good: Delete file. * testsuite/middle.inp: Likewise. * testsuite/middle.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "middle". tests: convert "xabcx" test to new framework * testsuite/misc.pl (Tests): Add xabcx. * testsuite/xabcx.good: Delete file. * testsuite/xabcx.inp: Likewise. * testsuite/xabcx.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "xabcx". tests: convert "manis" test to new framework * testsuite/misc.pl (Tests): Add manis. * testsuite/manis.good: Delete file. * testsuite/manis.inp: Likewise. * testsuite/manis.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "manis". tests: convert "linecnt" test to new framework * testsuite/misc.pl (Tests): Add linecnt. * testsuite/linecnt.good: Delete file. * testsuite/linecnt.inp: Likewise. * testsuite/linecnt.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "linecnt". tests: convert "khadafy" test to new framework * testsuite/misc.pl (Tests): Add khadafy. * testsuite/khadafy.good: Delete file. * testsuite/khadafy.inp: Likewise. * testsuite/khadafy.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "khadafy". tests: convert "inclib" test to new framework * testsuite/misc.pl (Tests): Add inclib. * testsuite/inclib.good: Delete file. * testsuite/inclib.inp: Likewise. * testsuite/inclib.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "inclib". tests: convert "factor" test to new framework * testsuite/misc.pl (Tests): Add factor. * testsuite/factor.good: Delete file. * testsuite/factor.inp: Likewise. * testsuite/factor.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "factor". tests: convert "flipcase" test to new framework * testsuite/misc.pl (Tests): Add flipcase. * testsuite/flipcase.good: Delete file. * testsuite/flipcase.inp: Likewise. * testsuite/flipcase.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "flipcase". tests: convert "fasts" test to new framework * testsuite/misc.pl (Tests): Add fasts. * testsuite/fasts.good: Delete file. * testsuite/fasts.inp: Likewise. * testsuite/fasts.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "fasts". tests: convert "enable" test to new framework * testsuite/misc.pl (Tests): Add enable. * testsuite/enable.good: Delete file. * testsuite/enable.inp: Likewise. * testsuite/enable.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "enable". tests: convert "cv-vars" test to new framework * testsuite/misc.pl (Tests): Add cv-vars. * testsuite/cv-vars.good: Delete file. * testsuite/cv-vars.inp: Likewise. * testsuite/cv-vars.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "cv-vars". tests: convert "classes" test to new framework * testsuite/misc.pl (Tests): Add classes. * testsuite/classes.good: Delete file. * testsuite/classes.inp: Likewise. * testsuite/classes.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "classes". tests: convert "bkslashes" test to new framework * testsuite/misc.pl (Tests): Add bkslashes. * testsuite/bkslashes.good: Delete file. * testsuite/bkslashes.inp: Likewise. * testsuite/bkslashes.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "bkslashes". tests: convert "amp-escape" test to new framework * testsuite/misc.pl (Tests): Add amp-escape. * testsuite/amp-escape.good: Delete file. * testsuite/amp-escape.inp: Likewise. * testsuite/amp-escape.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "amp-escape". tests: convert "appquit" test to new framework * testsuite/misc.pl (Tests): Add appquit. * testsuite/appquit.good: Delete file. * testsuite/appquit.inp: Likewise. * testsuite/appquit.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "appquit". tests: convert "0range" test to new framework * testsuite/misc.pl (Tests): Add 0range. * testsuite/0range.good: Delete file. * testsuite/0range.inp: Likewise. * testsuite/0range.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "0range". tests: convert "dollar" test to new framework * testsuite/misc.pl (Tests): Add dollar. * testsuite/dollar.good: Delete file. * testsuite/dollar.inp: Likewise. * testsuite/dollar.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "dollar". tests: convert "xbxcx3" test to new framework * testsuite/misc.pl (Tests): Add xbxcx3. * testsuite/xbxcx3.good: Delete file. * testsuite/xbxcx3.inp: Likewise. * testsuite/xbxcx3.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "xbxcx3". tests: convert "xabcx" test to new framework * testsuite/misc.pl (Tests): Add xabcx. * testsuite/xabcx.good: Delete file. * testsuite/xabcx.inp: Likewise. * testsuite/xabcx.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "xabcx". tests: convert "xbxcx" test to new framework * testsuite/misc.pl (Tests): Add xbxcx. * testsuite/xbxcx.good: Delete file. * testsuite/xbxcx.inp: Likewise. * testsuite/xbxcx.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "xbxcx". 2017-05-09 Jim Meyering tests: convert "recall2" test to new framework * testsuite/misc.pl (Tests): Add recall2. * testsuite/recall2.good: Delete file. * testsuite/recall2.inp: Likewise. * testsuite/recall2.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "recall2". tests: convert "recall" test to new framework * testsuite/misc.pl (Tests): Add recall. * testsuite/recall.good: Delete file. * testsuite/recall.inp: Likewise. * testsuite/recall.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "recall". tests: convert "insert" test to new framework * testsuite/misc.pl (Tests): Add insert-nl. * testsuite/insert.good: Delete file. * testsuite/insert.inp: Likewise. * testsuite/insert.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "insert". tests: convert "allsub" test to new framework * testsuite/misc.pl (Tests): Add allsub. * testsuite/allsub.good: Delete file. * testsuite/allsub.inp: Likewise. * testsuite/allsub.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "allsub". tests: convert "y-newline" test to new framework * testsuite/misc.pl (Tests): Add y-newline. * testsuite/y-newline.good: Delete file. * testsuite/y-newline.inp: Likewise. * testsuite/y-newline.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "y-newline". tests: convert "y-zero" test to new framework * testsuite/misc.pl (Tests): Add y-zero. * testsuite/y-zero.good: Delete file. * testsuite/y-zero.inp: Likewise. * testsuite/y-zero.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "y-zero". tests: convert "y-bracket" test to new framework * testsuite/misc.pl (Tests): Add y-bracket. * testsuite/y-bracket.good: Delete file. * testsuite/y-bracket.inp: Likewise. * testsuite/y-bracket.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "y-bracket". tests: convert "noeol" to new framework * testsuite/misc.pl (preserve-missing-EOL-at-EOF): New test, replacing... * testsuite/noeol.good: Delete file. * testsuite/noeol.inp: Likewise. * testsuite/noeol.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "noeol". * testsuite/Makefile.tests (noeol): Remove target. We also require this additional change, since the noeolw test used the just-deleted noeol.inp input file: * testsuite/noeolw.inp: New file, renamed from testsuite/noeol.inp. * testsuite/Makefile.tests: Use noeolw.inp, not the other test's file. (EXTRA_DIST): Add testsuite/noeolw.inp. tests: convert "insens" to new framework * testsuite/misc.pl (case-insensitive): New test, replacing... * testsuite/insens.good: Delete file. * testsuite/insens.inp: Likewise. * testsuite/insens.sed: Likewise. * testsuite/local.mk (EXTRA_DIST): Remove their names. (SEDTESTS): Remove "insens". tests: begin migrating test triples into perl framework * testsuite/head.good: Delete file. * testsuite/head.inp: Likewise. * testsuite/head.sed: Likewise. * testsuite/space.good: Likewise * testsuite/space.inp: Likewise. * testsuite/space.sed: Likewise. * testsuite/empty.good: Likewise * testsuite/empty.inp: Likewise. * testsuite/empty.sed: Likewise. * testsuite/zero-anchor.good: Likewise * testsuite/zero-anchor.inp: Likewise. * testsuite/zero-anchor.sed: Likewise. * testsuite/misc.pl: New file, subsuming the above, with one test for each of the test file triples (input, output, sed commands). (empty, empty2, head, space, zero-anchor): Converted tests. * testsuite/local.mk: Remove references to deleted files. (T): Add misc.pl. 2017-05-09 Assaf Gordon tests: keep temporary directory if SAVE_TEMPS requested When developers run 'make check SAVE_TEMPS=yes' the 'Coreutils.pm' module does not delete the temporary files, but the CuTmpdir.pm still deletes the temp directory. With this change, the temp directory is kept, and printed to the log file (e.g. test-suite.log). * testsuite/CuTmpdir.pm: if $EMV{SAVE_TEMPS} is not empty, don't auto-delete the temp directory, and print it to the log. 2017-05-09 Assaf Gordon maint: fix wrong PATH in testsuite The switch to non-recursive make in v4.2.2-129-g3b29bec used a wrong directory name in the PATH construction of automake's TESTS_ENVIRONMENT variable ('/src/' instead of '/sed'). This had no effect as each test shell script appended the ./sed/ path independently. However, switching to perl-based tests - this will be noticed as the just-built sed binary will not be in the PATH. * testsuite/local.mk: Fix $PATH variable in test environment. 2017-04-22 Jim Meyering maint: ignore more * .gitignore: Ignore a bunch of doc/*.* texinfo artifacts and sort. tests: support perl-based tests like those of grep and coreutils * configure.ac: Define HAVE_PERL. * testsuite/Coreutils.pm: New file. From grep. * testsuite/CuSkip.pm: Likewise. * testsuite/CuTmpdir.pm: Likewise. * testsuite/local.mk (, TEST_EXTENSIONS): Add .pl. (TESTSUITE_PERL, TESTSUITE_PERL_OPTIONS): Define. (PL_LOG_COMPILER): Define. (EXTRA_DIST): Add the new .pm files. 2017-04-18 Assaf Gordon sed: enable special meaning of '+' with '-E --posix' in sed-4.2 and later, '--posix' option wrongly disabled the special meaning of '+' even in ERE mode (--posix should only disable it in BRE mode, since it is a GNU extension in BRE). Bug introduced in the original implementation of --posix in v4.1a-5-gba68fb4. Reported by Jordan Torbiak in https://bugs.gnu.org/26409 . * NEWS: Mention bug fix. * sed/regexp.c (compile_regex_1): In --posix mode, set RE_LIMITED_OPS only in BRE mode. * testsuite/posix-mode-ERE.sh: New test. * testsuite/local.mk: Add new test. 2017-03-21 Assaf Gordon tests: test comments, braces after commands Test multiple combinations of comments, spaces, braces after commands, to validate recently added function 'read_end_of_cmd()'. See discussion in https://bugs.gnu.org/22460 . * testsuite/command-endings.sh: New test, including y/// bug. * testsuite/local.mk: Add new test. 2017-03-21 Assaf Gordon sed: refactor end-of-line parsing Follow-up to bug#22460: extract repeated code to parse end of lines (while saving comments, braces) into a new function. See discussion in https://bugs.gnu.org/22460 . * sec/compile.c (read_end_of_cmd): New function, consumes sed script input until newline, EOF, semicolon, closing brace or comment is found. (compile_program): Call new function instead of duplicating code. 2017-03-21 Thorsten Heymann sed: allow comments, braces after y/// sed-4.4 and earlier rejected the following: $ sed 'y/1/a/ #f' sed: -e expression #1, char 8: extra characters after command See https://bugs.gnu.org/22460 . * sed/compile.c (compile_program): Handle comments, braces after 'y' command. * NEWS: Mention it. 2017-03-19 Kent Fredric (tiny change) tests: ensure tty device is readable before running test Unreadable tty's fail with a different permissions error instead of failing with "is a tty" error. Reported in https://bugs.gnu.org/25692 . * testsuite/panic-tests.sh: Run test if tty device is both readable and writable. 2017-03-07 Assaf Gordon doc: expand "locale considerations" (multibyte) section Show examples of processing valid and invalid characters. Mention \L,\U for s/// command. Combines reports from: https://bugs.debian.org/500501 https://lists.gnu.org/archive/html/coreutils/2017-02/msg00039.html * doc/sed.texi (Locale Consideration): Expand section. * doc/config.texi: Add variables to render unicode characters portably. 2017-02-23 Assaf Gordon doc: warn against misuse of -i with other options 'sed -iE' is not the same as 'sed -Ei'. 'sed -ni' is dangerous. From https://bugs.debian.org/832088 * doc/sed.texi (Command-Line Options): Explain and add examples to '-i/--in-place' item. 2017-02-23 Assaf Gordon doc: elaborate about regex matching on pattern space Regex addresses work on current pattern space, not on the original input lines. From https://bugs.debian.org/284646 * doc/sed.texi (Regexp Addresses): Add a paragraph and an example. * doc/sed.x (Addresses): Add a sentence about regexp. 2017-02-23 Assaf Gordon doc: mention escape-sequence precedence Unescaping takes place before passing the pattern to the regex engine: $ echo 'a^c' | sed -e 's/\x5e/b/' ba^c From: https://bugs.debian.org/605142 * doc/sed.texi (Escaping Precedence): Add new subsection under 'escape sequences' with examples. 2017-02-16 Assaf Gordon build: fix 'install-html' target Switching to non-recursive makefiles broke the 'install-html' target: The gettext plumbing requires an 'install-html' target in po/Makefile. This was fixed in gettext v0.19.8.1-41-ge5a008a, but packages using older gettext need to manually patch po/Makefile.in.in. Report and suggested fix by Eric Blake in https://debbugs.gnu.org/25690 . * bootstrap.conf (bootstrap_epilogue): Add 'install-{html,pdf,dvi,ps}' targets to po/Makefile.in.in (if needed). 2017-02-09 Alexandre Jasmin doc: reference the i command in /regexp/I Remove "TODO: add @code{pxref}" in the text and add the ref. * doc/sed.texi (selecting lines by text matching): Add reference. 2017-02-09 Alexandre Jasmin doc: rework BRE/ERE table to fit PDF manual Sample code in the basic/extended table was too wide to fit on a Letter size printout. Text was overlapping and unreadable. Remove margin, split commands in two lines and change the column with. * doc/sed.texi (Basic and extended regexp): Rework table. 2017-02-07 Alexandre Jasmin (tiny change) doc: change description of the r command "Reads text file a file" was probably a typo. Also remove "Example: " in the summary as there's no example there. * doc/sed.texi (sed scripts): Fix r command description. 2017-02-07 Alexandre Jasmin (tiny change) doc: fix copy-pasted examples of regexp The samples commands demonstrating '\B' and '\S' were both using '\w'. The commands output is correct. * doc/sed.texi (Regular Expression Extensions): Use the proper commands. 2017-02-03 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.4 * NEWS: Record release date. gnulib: update to latest maint: avoid %x-format argument type mismatch * sed/compile.c (savchar): Cast %x-format argument to "unsigned int" rather than to "unsigned char". Reported by Eric Blake in https://lists.gnu.org/archive/html/sed-devel/2017-02/msg00000.html 2017-02-02 Assaf Gordon doc: fix broken xref Remove an extraneous space after @xref. Older 'makeinfo' would fail with: ./doc/sed.texi:3461: @xref expected braces. ./doc/sed.texi:3461: ` {Execution Cycle} and @ref{Addresses ov...' is too long for expansion; not expanded. Introduced in sed-4.3-15-gb0009b8. * doc/sed.texi: Remove space after @xref. 2017-01-31 Jim Meyering gnulib: update to latest 2017-01-31 Assaf Gordon doc: add NEWS entry for recent newline bugfix Bugfix in sed-4.3-12-g44d99bf (for https://bugs.gnu.org/25390) did not included a corresponding NEWS entry. * NEWS: Mention bugfix. 2017-01-28 Assaf Gordon doc: new 'multiple commands syntax' section Explain which commands can be separated by semicolon and which require newlines. Requested by Ori Avtalion in https://bugs.gnu.org/21845 . * doc/sed.texi (Multiple commands syntax): New section in "sed scripts" chapter. 2017-01-24 Assaf Gordon doc: update 'other resources' chapter Remove broken link and update other resources in the manual. Reported by olszkocj@aol.com in https://bugs.gnu.org/24770 . * doc/sed.texi (Other Resources): Update wording. 2017-01-18 Jim Meyering maint: update bootstrap and init.sh * bootstrap: Update from gnulib. * testsuite/init.sh: Likewise. 2017-01-16 Assaf Gordon doc: add 'branching and flow control' section * doc/sed.texi (Branching and flow control): New section under 'Advanced Sed' chapter. doc: add 'join lines' example * doc/sed.texi (Joining lines): Add example of joining SMTP header lines. 2017-01-16 Norihiro Tanaka tests: new test for dfa newline bug sed-4.3 would segfault with: printf '$LINENO $LINEN\nB\n' | sed -e 'N;s/\$LINENO\(.*\n\)/\1/' Culprit is dfa crash if multi-lines are read in pattern space. It is fixed at gnulib commit v0.1-1117-g823b5cb . Reported by S. Gilles in https://bugs.gnu.org/25390 * testsuite/newline-dfa-bug.sh: New test. * testsuite/local.mk: Add the test. 2017-01-16 Assaf Gordon gnulib: update to latest (with fix for dfa newline bug) sed-4.3 would segfault due to dfa bug, see https://bugs.gnu.org/25390 . * gnulib: Update to latest with various dfa fixes and improvement. * m4/.gitignore: Add 'std-gnu11.m4', 'pthread_rwlock_rdlock.m4'. 2017-01-16 Assaf Gordon doc: improve README installation instructions Mention README-hacking for git users. Issue raised by Jean-Christophe Manciot in https://bugs.gnu.org/25377 . * README: Improve installation paragraph based on coreutils' README. 2017-01-16 Jim Meyering maint: avoid syntax-check-exposed nit * doc/local.mk (doc/sed.1): Use $(...) rather than @...@. 2017-01-15 Jim Meyering maint: do not distribute COPYING.DOC * COPYING.DOC: Remove this file, now that we have doc/fdl.texi. * Makefile.am (EXTRA_DIST): Don't distribute it. 2017-01-14 Jim Meyering tests: skip valgrind-running tests when ASAN-enabled * init.cfg (require_valgrind_): Skip when we detect that the binary is ASAN-enabled. 2017-01-13 Vagelis Prokopiou (tiny change) maint: insert missing word in comment * sed/sed.h (struct vector)[v_allocated]: Insert "of". 2017-01-09 Assaf Gordon build: skip building man page if needed Skip building the man page if building from tarball (where 'doc/sed.1' is already prebuilt). Build a stub if building from git but generating the man page isn't possible (if there's no Perl or cross-compiling). Cross-compilation issue reported by Vishal Biswas in https://bugs.gnu.org/25358 . Problematic 'perl' is related to https://bugs.gnu.org/25367 . * configure.ac: Check for Perl, cross-compilation and tarball build; Set automake variable BUILD_MAN_PAGE, BUILD_DUMMY_MAN_PAGE accordingly. * doc/local.mk (doc/sed.1): Build conditionally if BUILD_MAN_PAGE is true; Create a dummy man page if BUILD_DUMMY_MAN_PAGE is true. * doc/sed-dummy.1: Stub man page. 2017-01-08 Assaf Gordon build: fix out-of-tree builds without dep-tracking Create sed,lib,doc,testsuite subdirectories at the end of 'configure', preventing build problems when building out of tree with --disable-dependency-tracking. Reported by Juan M. Guerrero in https://bugs.gnu.org/25371 . * configure.ac: Call AS_MKDIR_P() to create subdirectories. 2017-01-07 Jim Meyering doc: fix NEWS typo * NEWS: Fix typo s/introducted/introduced/ * cfg.mk (old_NEWS_hash): Update. Reported by Karl Berry. 2017-01-01 Jim Meyering maint: update gnulib and copyright dates for 2017 * gnulib: Update to latest. * all files: Run "make update-copyright". maint: compile warning-free with -DDEBUG_LEAKS * sed/regexp.c (release_regex) [DEBUG_LEAKS]: Modernize declaration. 2016-12-30 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 4.3 * NEWS: Record release date. gnulib: update to latest, to fix parallel getopt test failure 2016-12-28 Assaf Gordon doc: new annotated example for N/P/D/b commands Illustrates how to restructure line breaks over multiple lines. * doc/sed.texi (Line length adjustment): New section. 2016-12-28 Assaf Gordon doc: new annotated example section for N/D commands Illustrate how to use N/D to search for doubled-word spanning multiple lines. * doc/sed.texi (Text search across mutliple lines): New section. 2016-12-28 Assaf Gordon doc: add 'multiline techniques' section Expand on using D,G,H,N,P commands to process multiple lines. * doc/sed.texi (Multiline techniques): New section. 2016-12-22 Assaf Gordon maint: update m4/.gitignore * m4/.gitignore: Add 'dirfd.m4', 'hard-locale.m4'. 2016-12-20 Jim Meyering maint: RE_ICASE, RE_NO_SUB: remove useless '#ifdef's * sed/regexp.c (compile_regex_1): Remove #ifdef RE_ICASE guard, since dfa.c uses it unconditionally. Suggested by Norihiro Tanaka. Also remove the nearby "#ifdef RE_NO_SUB". Those macros are guaranteed to be defined by virtue of configure-time tests that ensure we're using either a new-enough native glibc (RE_NO_SUB was added in 2004), or the included replacement. 2016-12-19 Jim Meyering gnulib: update to latest; dfa improvement and getopt-posix-tests build fix 2016-12-18 Jim Meyering gnulib: update to latest and adapt to modified dfa API * sed/regexp.c (compile_regex_1): Do away with use of DFA_CASE_FOLD. This gnulib update pulls in a dfa module in which that symbol no longer exists. Instead, it uses RE_ICASE in syntax bits. * m4/.gitignore: Add files created by running bootstrap. * lib/.gitignore: Likewise. tests: use just-built sed in more places * cfg.mk (PATH): Prepend $(srcdir)/src, so that we use the just- built sed also when running commands like those of "make distcheck". Tested by running this in a just-built directory: f=sed/sed; printf '%s\n' '#!/bin/sh' 'sleep 9h' > $f; chmod a+x $f and then verifying that nearly every "make syntax-check" rule hangs. 2016-12-14 Jim Meyering doc: fix sed.1-building rule to work on mingw * doc/local.mk (doc/sed.1): Add $(EXEEXT) suffix on "sed/sed" dependent. Patch by Miodrag Milanovic via Stephan T. Lavavej, in https://lists.gnu.org/archive/html/sed-devel/2016-12/msg00026.html 2016-12-12 Jim Meyering tests: port tests to avoid tr NUL-eliding behavior on Solaris * testsuite/Makefile.tests (elide_cr): Define this to use sed. Replace every prior use of "$(TR) -d \\r" with it. Our just-built sed can do that job (elide CR bytes) even when the input contains NUL bytes. Solaris 11.3's /bin/tr would remove any NUL byte from the result. Reported by Nelson H.F. Beebe in https://lists.gnu.org/archive/html/sed-devel/2016-12/msg00006.html gnulib: update to latest, for solaris-10-vs-configure fix doc: add a README-hacking file * README-hacking: New file: instructions for building from a git clone. Copied from grep's repo, but with s/grep/sed/. 2016-12-11 Jim Meyering build: avoid "make dist" failure due to missing dependencies * sed/local.mk ($(sed_sed_OBJECTS)): Depend on $(BUILT_SOURCES). Reported by Assaf Gordon in https://lists.gnu.org/archive/html/sed-devel/2016-12/msg00016.html 2016-12-10 Assaf Gordon doc: simplify documentation building Do not bother autogenerating sed.texi from sed-in.texi. All it did was to add a few '@group' commands. Since these are rarely modified, add them manually. This simplifies the documentation-building rules and lets us distribute only one copy of the texinfo manual. See http://lists.gnu.org/archive/html/sed-devel/2016-12/msg00013.html * doc/groupify.sed, doc/sed-in.texi: Remove files. * doc/sed.texi: Remove the 'auto-generated' warning, as this file is no longer auto-generated. * doc/local.mk (doc_sed_TEXINFOS): Rename from sed_TEXINFOS. (dist_noinst_DATA): The above renaming enables us to remove doc/config.texi and doc/fdl.texi from this list. (dist_noinst_SCRIPTS): Remove. (doc/sed.texi): Remove rule. (doc/s-texi): Likewise. 2016-12-06 Assaf Gordon maint: add missing fdl.texi to distribution Neglected to add it in commit v4.2.2-175-gc6a55aa. * doc/local.mk (sed_TEXINFOS, dist_noinst_DATA): Add doc/fdl.texi. 2016-12-06 Assaf Gordon doc: follow-up changes for manual improvements Fixes few missing issues from commit v4.2.2-176-g801a2c8, discussed in http://lists.gnu.org/archive/html/sed-devel/2016-12/msg00005.html * doc/config.texi: Add stub texinfo macros '@codequotebacktick' and '@codequoteundirected' (needed if using older versions of texinfo). * doc/sed-in.texi: Fix syntax that's accepted by newer texinfo but rejected by texinfo-4.13; Disable directed quote characters in examples. * doc/sed.texi: Auto-regenerated. 2016-12-05 Jim Meyering gnulib: update to latest 2016-12-05 Assaf Gordon doc: reorganize and expand manual Discussed in http://lists.gnu.org/archive/html/sed-devel/2016-11/msg00005.html * doc/sed-in.texi: Reorganize and expand. * doc/sed.texi: Auto-regenerated. 2016-12-02 Assaf Gordon maint: upgrade documentation license to GFDL-1.3 (or later) * bootstrap.conf: Add gnulib's fdl module. * doc/sed-in.texi: Update license blurb to latest recommended text. Remove duplicated copyright information from title page (copied from coreutil's texinfo style). Include fdl.texi in a new appendix. 2016-11-29 Jim Meyering tests: use "returns_ N env VAR=val ..." Use "returns_ N env VAR=val ..." rather than "VAR=val returns_ N ...". Some shells do not propagate envvar settings through our use of the "returns_" function, so set any envvar via use of "env". This was an issue at least on Ubuntu, Debian and *BSD-based systems. * testsuite/mb-bad-delim.sh: As above. * testsuite/mb-y-translate.sh: Likewise. gnulib: update to latest 2016-11-27 Jim Meyering tests: use "returns_ 1" rather than "&& fail=1" * testsuite/colon-with-no-label.sh: Rather than failing only upon success (exit 0), fail upon exit with any value other than 1. build: stop suppressing many -W... warnings This code in configure.ac added many -W options to the list of warnings that we do *not* enable. When I copied most of this code into this file from coreutils, I did not try to remove the options that happened not to be needed for sed. Now, compilers have improved and I have done that triage. Many -W options can be left enabled. * configure.ac (GNULIB_WARN_CFLAGS): There is no longer any need to use a different set of warnings for lib/ files, so remove this variable, along with many -W-related exemptions. build: avoid "make distcheck" failure due to leftover .Po files The following definitions caused trouble for no gain. They caused some .o file names to have a long additional prefix and even resulted in some .Po files not being removed by "make distclean" when building with recent automake. The only reason to use these definitions was -- long ago -- to require different compiler options for files in lib/ than elsewhere. That is no longer necessary. * lib/local.mk (lib_libsed_a_CPPFLAGS): Remove definition. (lib_libsed_a_CFLAGS, lib_libsed_a_LIBADD): Likewise. (lib_libsed_a_DEPENDENCIES): Likewise. maint: fix -Wformat-exposed errors in uses of "panic" * sed/utils.h (panic): Add __printf__ attribute, so that compilers can diagnose format/arg-type errors. This exposed the following: * sed/utils.c (ck_fwrite): Don't try to print size_t via %u. Instead, use %llu and a cast to unsigned long long. * sed/mbcs.c (is_mb_char): Cast an "int" to unsigned int, to avoid mismatch with %x. maint: remove unused parameters * sed/compile.c (finish_program): This function's sole argument was unused. Remove it and update callers. * sed/sed.h: Update prototype. * sed/utils.c (register_open_file): This functions's third argument, "temp", was unused. Remove it and update callers. * sed/sed.c: Update use. maint: fix two "make syntax-check" failures * cfg.mk (exclude_file_name_regexp--sc_prohibit_empty_lines_at_EOF): Also exempt zero-anchor.{inp,good}. * sed/regexp.c (match_regex): s/can not/cannot/ 2016-11-16 Assaf Gordon build: remove explicit 'html' makefile target The 'html' target was a left-over from the recursive Makefiles, which were converted to non-recursive in sed v4.2.2-129-g3b29bec. With this removed, the 'html' target in 'Makefile' is correctly genererated by automake (from doc/local.mk), and 'make html' works again. * Makefile.am: Remove explicit 'html' target. 2016-11-12 Jim Meyering tests: use "returns_ 1" rather than testing $? = 1 * testsuite/temp-file-cleanup.sh: As above. 2016-11-04 Assaf Gordon sed: standardize exit code values Use exit-code constants, and document them: 1=syntax error, 2=bad input files, 4=I/O error or panic. Add "Exit Status" section in manual. * sed/utils.h: Define new constants. * sed/utils.c, sed/compile.c, sed/execute.c, sed/sed.c: Use constants instead of hard-coded values. * doc/sed-in.texi: Add "Exit Status" section. * doc/sed.texi: Automatically re-generated. 2016-11-04 Assaf Gordon maint: regenerate sed.texi after doc change Regenerate sed.texi from sed-in.texi after doc change in v4.2.2-160-10549b7 ( https://bugs.gnu.org/24799 ). * doc/sed.texi: Regenerated with 'make info'. 2016-11-04 Norihiro Tanaka sed: fix multi-line regex matching with -z Sed incorrectly matches the anchor '^' to a newline character (ASCII 10) despite using NUL line terminators (-z). See https://bugs.gnu.org/24615 . Previous (incorrect) behavior: $ printf "a\0b\0" | sed -z 'N;s/^/X/mg;' | od -An -a X a nul b nul With this fix: $ printf "a\0b\0" | sed -z 'N;s/^/X/mg;' | od -An -a X a nul X b nul Current regex engine can not handle NUL multiline anchors. Implement such search manually (by breaking the pattern into separate NUL-terminated lines) and searching each one individually. * NEWS: Mention it. * sed/regexp.c (compile_regex_1): Don't use newline_anchor of regex, if the buffer delimiter is not newline character. (match_regex): Do above case line-by-line. * testsuite/zero-anchor.good, testsuite/zero-anchor.inp, testsuite/zero-anchor.sed: New test. * testsuite/local.mk: Add the test. * testsuite/Makefile.tests: Add the test. * bootstrap.conf: Add memrchr. * lib/.gitignore, m4/.gitignore: Ignore memrchr files. 2016-11-04 Norihiro Tanaka sed: optimize regex processing of lone ^ or $ Optimize /^/ and /$/ patterns and avoid invoking the regular expression engine if possible. See https://bugs.gnu.org/24615 . * sed/regex.c (compile_regex_1): Mark the patterns which consist of ^ or $. (match_regex): Handle the patterns which consist of ^ or $ manually. * sed/sed.h (struct regex): New members 'begline' and 'endline'. * testsuite/newline-anchor.good, testsuite/newline-anchor.sed, testsuite/newline-anchor.sed: New test. * testsuite/Makefile.tests: Add the test. * testsuite/local.mk: Add the test. 2016-10-26 Vincenzo Romano doc: make an example's description more precise * doc/sed-in.texi (Regular Expressions): Mention that the match must be at the end of a line. 2016-10-02 Jim Meyering maint: address syntax-check warnings about @pxref * doc/sed-in.texi: Improve cross-references. * doc/sed.texi: Regenerate. build: address warnings from GCC 7 * sed/sed.c (usage): Declare with the _Noreturn attribute to suppress this warning/error: sed/sed.c:359:11: error: this statement may fall through \ [-Werror=implicit-fallthrough] usage(0); ^~~~~~~~ * sed/execute.c (execute_program) [case 'q']: Add FALLTHROUGH comment to avoid warning from GCC 7's -Wimplicit-fallthrough. * sed/compile.c (compile_program): Likewise. gnulib: update to latest 2016-09-25 Norihiro Tanaka sed: fix incorrect match for title case * sed/regexp.c (compile_regex_1): Avoid building fastmap for case- insensitive matching. * testsuite/title-case.sh: New file, to test for this fix. * testsuite/local.mk (T): Add the new file name and sort alphabetically. * init.cfg (require_el_iso88597_locale_): Define. 2016-09-11 Jim Meyering dfa: reflect move of dfa code to new gnulib module * bootstrap.conf (gnulib_modules): Add dfa. * sed/dfa.c: Remove file. * sed/dfa.h: Likewise. * sed/local.mk (sed_sed_SOURCES): Remove dfa.c. (NOINST_HEADERS): Remove dfa.h. * sed/regexp.c (compile_regex_1): Use new dfasyntax API. * sed/sed.c (localeinfo): New global. (main): Call init_localeinfo to initialize it. * sed/sed.h: Include localeinfo.h and declare the new global. * lib/.gitignore: Ignore new gnulib-imported files. * m4/.gitignore: Likewise. * po/POTFILES.in: s,sed/dfa.c,lib/dfa.c, gnulib: update to latest, for new dfa module 2016-09-04 Assaf Gordon sed: new --sandbox option In sandbox mode, r/w/e commands are not allowed. This ensures sed operates only on files designated on the command line, and cannot execute external programs. * sed/sed.c (sandbox): New option variable. (usage): List new option. (main): Accept new option in getopt. * sed/sed.h (sandbox): New extern option variable. * sed/compile.c (DISALLOWED_CMD): New error message. (read_filename): Abort in sandbox mode (r/w commands). (compile_program): Abort on 'e' in sandbox mode. * testsuite/sandbox.sh: Test new option. * testsuite/local.mk: Add new test. * NEWS: Mention new option. * doc/sed-in.texi, doc/sed.texi: Document new option. 2016-08-27 Jim Meyering maint: fix distclean-vs-excess-.Po files more cleanly * doc/local.mk (distclean-local): Remove rule. Moved to ... * Makefile.am (distclean-local): ...here, and add a command to remove all .deps directories. It could have stayed in that .mk file, but since with this change, it is no longer specific to doc/, the top level makes more sense. * lib/local.mk (MOSTLYCLEANFILES): Remove this. Covered by the above. Reported by Assaf Gordon in https://lists.gnu.org/archive/html/sed-devel/2016-08/msg00019.html 2016-08-26 Assaf Gordon maint: update .mailmap Consolidate author's upper/lower case email addresses. Fixes the following problem: $ make THANKS GEN THANKS ./thanks-gen: THANKS.in: duplicate name: Ralf Wildenhues Due to: $ git log --pretty=format:'%aN %aE' | grep -i ralf | sort -u Ralf Wildenhues ralf.wildenhues@gmx.de Ralf Wildenhues Ralf.Wildenhues@gmx.de see http://lists.gnu.org/archive/html/sed-devel/2016-08/msg00023.html http://lists.gnu.org/archive/html/sed-devel/2016-08/msg00025.html * .mailmap: Add email mapping. 2016-08-24 Assaf Gordon doc: update README * README: Update information: remove mention of README.boot, mention AUTHORS, THANKS, website and help/usage information. maint: update program name in THANKS.in * THANKS.in: Replace 'grep' with 'sed'. maint: remove superfluous name from THANKS.in * THANKS.in: Remove Pádraig Brady: He is the author of two commits (v4.2.1-1-g737ca5e and v4.2.1-37-g629ef76) and will be automatically included in the auto-generated THANKS file. maint: remove README-alpha * README-alpha: Removed. 2016-08-22 Assaf Gordon maint: update AUTHORS * AUTHORS: Update information. 2016-08-18 Jim Meyering doc: work with read-only sed.1 * doc/local.mk (doc/sed.1): Also remove $@, so that mv does not prompt about read-only target. 2016-08-14 Jim Meyering build: fix sed.1 dependency; work with latest automake * doc/local.mk (doc/sed.1): Depend not on sed.c, but on the binary that help2man must run. Also depend on .version, to be sure we pick up a version change. Also, be careful not to write directly to $@: instead write to $@-t, and make read-only to mark as generated, and then, only if all is well, rename atomically to $@. Also, prefix commands with $(AM_V_GEN) and $(AM_V_at) and ensure that doc/ exists via $(MKDIR_P). (distclean-local): New target. Use this instead of ... (dist-hook-man-page): ... this. Remove rule. (SEDBIN): Remove unnecessary $(top_builddir)/ prefix. (doc/sed.texi, doc/s-texi): Remove some uses of $(top_builddir). Replace others with $(srcdir). * Makefile (dist-hook): remove dependency on dist-hook-man-page. * lib/local.mk (MOSTLYCLEANFILES): Arrange for lib/.deps/*.Po to be removed as part of "make distclean", so that make distcheck no longer fails when using latest automake. sed: remove obsolete advice from --help output * sed/sed.c (contact): Now that there is a sed-specific mailing list, there is no longer any need to include "sed" in the subject. Remove that advice. 2016-08-11 Assaf Gordon doc: mention \s,\S,\<,\> regex gnu-extensions * doc/sed-in.texi: Mention the extensions. * doc/sed.texi: Automatically re-generated. doc: expand on N command * doc/sed-in.texi: Expand on gnu-extension of 'N' command. * doc/sed.texi: Automatically re-generated. 2016-08-09 Jim Meyering sed: avoid one-byte heap buffer overrun * sed/execute.c (DFA_SLOP): Define. (resize_line): Use it in each line-related allocation. (line_init): Likewise. (line_copy): Likewise. 2016-08-08 Assaf Gordon sed: adjust line-terminator of F/l/= commands when -z is used Change F/l/= commands to use NUL line terminator if '-z' option is used. Previously F/l/= used '\n' regardless of -z option. See: http://lists.gnu.org/archive/html/sed-devel/2016-08/msg00000.html http://lists.gnu.org/archive/html/sed-devel/2016-08/msg00002.html . * sed/execute.c: (do_list,execute_program): Use 'buffer_delimiter' as line delimiter instead of hard-coded '\n'. * testsuite/nulldata.sh: Adjust tests accordingly. 2016-08-08 Norihiro Tanaka sed: use grep's DFA matcher to speed up regular expression matching When possible, sed now uses grep's DFA matcher to match regular expression, often resulting in a 10x speed-up. * NEWS (Improvements): Mention it. Move this section to the top. * sed/local.mk (sed_sed_SOURCES): Add dfa.c, and sort file names. (noinst_HEADERS): Add dfa.h, and sort file names. * sed/regexp.c (dfaerror, dfawarn): New functions. (compile_regex_1): Compile pattern with dfa. (match_regex): Use dfa. * sed/sed.h: Include dfa.h. (struct regex): New member dfa. * sed/dfa.c: New file, copied from grep. * sed/dfa.h: New file, copied from grep. * po/POTFILES.in: Add sed/dfa.c. 2016-08-08 Assaf Gordon sed: reject 's///e' in --posix mode 's///e' modifier is a gnu extension - reject it in --posix mode. * sed/compile.c: (mark_subst_opt): Exit upon 's///e' in posix mode. * testsuite/posix-mode-s.sh: Adjust tests accordingly. 2016-08-08 Assaf Gordon sed: allow multiple (non-conflicting) -E/-r parameters Accept multiple -E/-r parameters as they enable the same ERE operation mode. Previously, sed would reject multiple -E/-r/-R parameters (as preparation to supporting PERL syntax and wanting to avoid -E/-r/-R conflicts). This enables 'sed -E -E -r =' which previously would exit with a failure (and print the help screen). * sed/sed.c: (main): Don't exit on multiple -E/-r parameters. 2016-07-26 Assaf Gordon maint: add static-analysis target to cfg.mk Run 'make static-anslysis' to perform static code analysis using clang's 'scan-build' tool. * cfk.mk: (static-analysis,static-analysis-init,static-analysis-config, static-analysis-make): New targets. 2016-07-26 Assaf Gordon maint: add copyright/license information to configure.ac Starting year chosen as 1993 based on earliest 'configure.in' found in sed-1.18.tar.gz from that year. * configure.ac: Add standard copyright and license information. 2016-07-26 Assaf Gordon maint: update .gitignore after bootstrap Added by running ./bootstrap on a clean sed repository. * build-aux/.gitignore: Add config.rpath. * lib/.gitignore: Add additional gnulib C files. * m4/.gitignore: Add additional gnulib m4 files. * po/.gitignore: Add additional PO-related files. 2016-07-26 Assaf Gordon maint: update .gitignore * .gitignore: Add coverage files, .dirstamp files, testsuite files. 2016-07-25 Jim Meyering maint: skip a check when en_US.UTF-8 collation rules are broken * cfg.mk (sc_THANKS_in_sorted): This check would fail on systems for which "." is not ignored. Add a quick sort-based check for that error, and skip the check on any broken system. This also corrects the command to use $(srcdir)/THANKS.in, rather than just THANKS.in. 2016-07-24 Jim Meyering maint: sort THANKS.in * THANKS.in: Move J.T. Conklin's name "down" so it is in en_US.UTF-8-sorted order. 2016-07-24 Assaf Gordon build: switch to non-recursive makefile * .gitignore: Ignore '.dirstamp' files * bootstrap.conf: Add 'non-recursive-gnulib-prefix-hack' gnulib module. Add missing 'stdalign' module. (bootstrap_post_import_hook): Use module to adjust gnulib.mk file. * configure.ac: Add 'subdir-objects' automake option, don't generate makefiles in subdirectories. Adjust path of XFAIL tests. * Makefile.am: Include 'local.mk' files, adjust for non-recursive make. * doc/Makefile.am: Rename to ... * doc/local.mk: Adjust paths. Rename SED to SEDBIN variable, to avoid overriding the default system-wide SED value which is automatically set by autotools. * lib/Makefile.am: Rename to ... * lib/local.mk: Adjust paths. * lib/.gitignore: Ignore 'stdalign.h' from gnulib module. * sed/Makefile.am: Rename to ... * sed/local.mk: Adjust paths. * testsuite/Makefile.am: Rename to ... * testsuite/local.mk: Adjust paths. * testsuite/init.cfg: Move to ... * init.cfg: ... here. Moved to avoid changing the path in init.sh, and to keep the same directory structure as coreutils. (require_en_utf8_locale_): Adjust path for get-mb-cur-max executable. (require_ja_shiftjis_locale_,require_valid_ja_shiftjis_locale_): Adjust path for test-mbrtowc executable. * m4/.gitignore: Add gnulib's non-recusive-make module. * testsuite/runtest: Adjust paths of Makefile.tests and dir variables, Skip utility program 'get-mb-cur-max' which is not a standalone test. * testsuite/Makefile.tests: Adjust path of sed exeutable. * testsuite/eval.sed, testsuite/eval.good: Adjust path of sed executable. * testsuite/cmd-R.sh, testsuite/cmd-l.sh, testsuite/colon-with-no-label.sh, testsuite/comment-n.sh, testsuite/compile-errors.sh, testsuite/compile-tests.sh, testsuite/convert-number.sh, testsuite/execute-tests.sh, testsuite/follow-symlinks-stdin.sh, testsuite/follow-symlinks.sh, testsuite/help-version.sh, testsuite/in-place-hyphen.sh, testsuite/in-place-suffix-backup.sh, testsuite/invalid-mb-seq-UMR.sh, testsuite/mb-bad-delim.sh, testsuite/mb-charclass-non-utf8.sh, testsuite/mb-match-slash.sh, testsuite/mb-y-translate.sh, testsuite/normalize-text.sh, testsuite/nulldata.sh, testsuite/panic-tests.sh, testsuite/posix-char-class.sh, testsuite/posix-mode-N.sh, testsuite/posix-mode-addr.sh, testsuite/posix-mode-bad-ref.sh, testsuite/posix-mode-s.sh, testsuite/range-overlap.sh, testsuite/recursive-escape-c.sh, testsuite/regex-errors.sh, testsuite/stdin-prog.sh, testsuite/subst-mb-incomplete.sh, testsuite/subst-options.sh, testsuite/subst-replacement.sh, testsuite/temp-file-cleanup.sh, testsuite/unbuffered.sh: Adjust paths of init.sh, sed executable. 2016-07-17 Assaf Gordon tests: test --follow-symlinks option * configure.ac: (TEST_SYMLINKS): New AM_CONDITIONAL. * testsuite/follow-symlinks.sh: New tests. * testsuite/Makefile.am: (T): Conditionally add new test (and existing test 'follow-symlink-stdin.sh') based on TEST_SYMLINKS. tests: test y command in multibyte locales * testsuite/mb-y-translate.sh: New test. * testsuite/Makefile.am: Add new test. 2016-07-16 Assaf Gordon tests: test y,s non-slash multibyte delimiters Test various scenarios of using a multibyte delimiters (instead of slash) for s,y commands under UTF-8 and C locales. * testsuite/mb-bad-delim.sh: New test. * testsuite/Makefile.am: Add new test. 2016-07-06 Assaf Gordon tests: skip tests upon buggy mbrtowc implementations Sed tests purposefully use invalid non-utf8 multibyte sequences to test sed's handling of invalid input/program. Some implementations wrongly accept invalid sequences, causing false alarms. Add a test program to detect such buggy implementations and skip the tests if needed. See discussion in: http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00005.html http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00000.html http://lists.gnu.org/archive/html/sed-devel/2016-06/msg00031.html http://lists.gnu.org/archive/html/sed-devel/2016-06/msg00014.html * bootstrap.conf: Add gnulib's closeout module. * m4/.gitignore, lib/.gitignore, po/POTFILES.in: Adjust after adding closeout module. * testsuite/Makefile.am (check_PROGRAMS): Add new program. * testsuite/test-mbrtowc.c: New program. * testsuite/init.cfg (require_valid_ja_eucjp_locale_) (require_valid_ja_shiftjis_locale_): New functions. Call test-mbrtowc with known invalid multibyte sequences to ensure implementation is not buggy and rejects them; Skip test otherwise. * testsuite/invalid-mb-seq-UMR.sh: Use require_valid_ja_eucjp_locale_. * testsuite/mb-charclass-non-utf8.sh: Use require_valid_ja_shiftjis_locale_. 2016-07-04 Assaf Gordon sed: fix minor multibyte parsing bug Previously sed would parse multibyte characters incorrectly in two scenarios: 1. Slash following an incomplete-yet-valid multibyte sequence (match_slash): $ LC_ALL=en_US.UTF-8 sed $'s/\316/X/' sed: -e expression #1, char 6: unterminated `s' command 2. Open/close brackets as part of a valid mutilbyte string inside a character class (snarf_char_class). In the example below, '\203]' is a valid multibyte character in SHIFT-JIS locale: $ LC_ALL=ja_JP.shiftjis sed $'/[\203]/]/p' sed: -e expression #1, char #5: Unmatched [ or [^ Both cases stem from mbcs.c:brlen() being non-intuitive: It returned 1 for valid single-byte character, invalid multibyte-character, and a for the last byte of a valid multibyte sequence - making it non-trivial to use correctly. This commit replaces brlen() with a simpler is_mb_char() function: returns non-zero for multibyte sequences, zero for single/invalid sequences. * sed/sed.h: (BRLEN, brlen): Remove delaration. (IS_MB_CHAR,is_mb_char): Add macro and function declaration. * sed/mbcs.c: (brlen): Remove function. (is_mb_char): New function. * sed/compile.c: (snarf_char_class, match_slash): Use IS_MB_CHAR instead of BRLEN; Adjust local variables accordingly. * testsuite/mb-match-slash.sh: New test for scenario 1. * testsuite/mb-charclass-non-utf8.sh: New test for scenario 2, requires SHIFT-JIS locale. * testsuite/Makefile.am: Add new tests * testsuite/init.cfg: (require_ja_shiftjis_locale_): New function. * NEWS: Mention bug fix. 2016-06-26 Tristan Verniquet sed: use unlocked-io Sed already imports gnulib's unlocked-io module, but it was not included in 'utils.c'. Suggested in http://bugs.gnu.org/23848 . * sed/utils.c: Include 'unlocked-io.h' header. * NEWS: Mention improvement. 2016-06-05 Assaf Gordon tests: test text normalization (\X sequences) Sed understands escape-sequences such as \r \t \n \cX. Test various scenarios of text normalization. * testsuite/normalize-text.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-02 Assaf Gordon tests: test \dNNN \oNNN \xNN escape sequences Test different variations of \dNNN \oNNN \xNN escape sequences, exercising compile.c:convert_number(). * testsuite/convert-number.sh: new test. * testsuite/Makefile.am: add new test 2016-06-02 Assaf Gordon tests: test less-common compilation cases Test various odds-and-ends stemming from GNU sed implementation of the program compilation module. * testsuite/compile-tests.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test less-common execution cases * testsuite/execute-tests.sh: new test. * testsuite/Makefile.am: add new test. tests: test 'R' command * testsuite/cmd-R.sh: new test. * testsuite/Makefile.am: add new test. tests: test program file from STDIN * testsuite/stdin-prog.sh: new test. * testsuite/Makefile.am: add new test. tests: test auto-silent mode with '#n' comments * testsuite/comment-n.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test character-classes escaping in POSIX mode No \X escaping inside character classes in POSIX mode. Compare: $ printf "t\t\n" | sed 's/[\t]/X/' | od -a 0000000 t X nl $ printf "t\t\n" | sed --posix 's/[\t]/X/' | od -a 0000000 X ht nl * testsuite/posix-char-class.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test substitue replacements (s//\N/) Test standard POSIX (e.g. \1 \2 \3) and gnu extension (\U \u \L \l \E) replacements in s/// command. * testsuite/subst-replacement.st: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test s/// modifiers Test standard posix and gnu extension modifiers to s/// command. * testsuite/subst-options.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test compilation-related errors Cover code paths in copmile.c leading to errors. * testsuite/compile-errors.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test regex compilation errors Test scenarios leading to regex compilation errors. * testsuite/regex-errors.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: exercise 'panic' calls Test scenarios in which GNU sed terminates with a panic error message (and verify the error message text). * testsuite/panic-tests.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test -i/--inplace with backup suffix option GNU Sed accepts optional backup filename/suffix with -i option. If the name contains '*', sed replaces it with the input filename. Test various scenarios of suffix names. Example: sed -i'*-*-*.txt' = a will generate backup file 'a-a-a.txt'. * testsuite/in-place-suffix-backup.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test -u/--unbuffered mode Test GNU Sed behaviour in --unbuffered mode (reading only the minimum needed characters from STDIN, leaving output for next program when sed terminates). Example: seq 10 | (sed -u 1q ; sort -nr ) * testsuite/unbuffered.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test -z/--null-data option Test input/output line-terminators with -z option. * testsuite/nulldata.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test 'l' command Test output width of 'l' command, based on COLS envvar, lN (GNU Extension), '-l N' command-line parameter, and default output width. * testsuite/cmd-l.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test gnu address extension in gnu/posix modes The following addresses are gnu extensions: 0,/regexp/ addr1,+N addr2,~N * testsuite/posix-mode-addr.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test 's///' gnu extensions in gnu/posix modes s///[iImMe] should be rejected in --posix mode. s//\[lLuUN]// should have no special meaning in --posix mode. * testsuite/posix-mode-s.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test invalid backreferences in gnu/posix modes 's/foo/\1/' is rejected by default (gnu extension), silently accepted in POSIX. * testsuite/posix-mode-bad-ref.sh: new test. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: test N command in gnu/posix modes 'N' with insufficient output prints nothing in POSIX mode, prints the last read line in GNU Extension mode. Compare: echo a | sed N echo a | sed --posix N echo a | POSIXLY_CORRECT=y sed N * testsuite/posix-mode-N.sh: test output of N command under gnu and posix modes. * testsuite/Makefile.am: add new test. 2016-06-01 Assaf Gordon tests: invalid-mb-seq-UMR: avoid false-failure * testsuite/Makefile.am (TESTS_ENVIRONMENT): Propagate the LOCALE_JA envvar setting determined at configure time into the test environment. Without this, the test would fail on some systems: at least CentOS6 and 7, but not Fedora 24. 2016-06-01 Jim Meyering tests: don't let any envvar setting perturb tests * testsuite/envvar-check: New file. Identical to the one from grep, but with the addition of sed-specific "COLS". * testsuite/Makefile.am (EXTRA_DIST): Add it here. (TESTS_ENVIRONMENT): New prologue, mostly copied from grep. 2016-05-29 Assaf Gordon sed: reject recursive escaping after \c previously, sed 's/./\c\\/' or 's/./\c\d/' would produce incorrect results. Require two backslashes after \c to denote control sequence ^\ (ASCII 0x34), and reject recursive escaping (e.g. \c\x61). * sed/compile.c: (RECURSIVE_ESCAPE_C): New error message. (normalize_text): Check for \c-backslash, reject recursive escaping. * testsuite/recursive-escape-c.sh: New file. Test new behaviour. * testsuite/Makefile.am (T): Add new test. * NEWS (Bug fixes): Mention it. 2016-01-02 Jim Meyering revert v4.2.2-87-gc033bde, to make sed -i treat "-" as a file name Stephane Chazelas made a fine case for why sed's --in-place (-i) should treat "-" as a file name, and not as standard input in http://debbugs.gnu.org/21249 There was no specific test for the prior behavior, so rather than letting the revert remove the sole test, adapt and retain it: * testsuite/in-place-hyphen.sh: However, retain and adapt this file to test for the restored behavior. * testsuite/Makefile.am: Retain this change, too. 2016-01-01 Jim Meyering maint: update copyright year, bootstrap, init.sh Run "make update-copyright" and then... * gnulib: Update to latest. * tests/init.sh: Update from gnulib. * bootstrap: Likewise. 2015-12-27 Jim Meyering sed: do not elide an invalid byte in a substitution RHS Now, sed copies such bytes into the output. * sed/execute.c (str_append_modified): Copy each invalid byte into the result string, as the comment suggests. * testsuite/invalid-mb-seq-UMR.sh: Adjust this test to conform. Perl does the same thing. Also, remove an unnecessary "-e" option. * NEWS (Bug fixes): Mention it. Bug introduced prior to the switch to git in 2004, between the releases of sed-4.0.9 and sed-4.1. This addresses http://debbugs.gnu.org/22254. sed: fix a heap-clobbering buffer overrun * sed/execute.c (str_append_modified): Upon encountering an incomplete multi-byte sequence in a substitution replacement string, do not treat mbrtowc's return value of -2 as a large, positive number. * testsuite/subst-mb-incomplete.sh: New file/test. * testsuite/Makefile.am (T): Add it. Reported by Hanno Böck in http://debbugs.gnu.org/22127. Hanno used the AFL fuzzer to find the segfault-inducing input from which I derived the test's input. Introduced by v4.2.2-76-g78e8e58, this bug was never in a release. 2015-12-12 Jim Meyering tests: add helper prog, get-mb-cur-max and re_en_utf8_locale_ function 2015-08-23 Jim Meyering sed: reject a ":" command without a label Before, sed would accept it and treat it as a label whose name had length 0, and that could be referenced via a "b" or "t" command with no label. * sed/compile.c (errors): Add the new diagnostic. (COLON_LACKS_LABEL): Define. (compile_program): Reject ":" command with no label. * testsuite/colon-with-no-label.sh: New file. * testsuite/Makefile.am (T): Add it. * NEWS (Bug fixes): Mention it. Reported by Stephane Chazelas in http://bugs.gnu.org/21250 2015-08-01 Assaf Gordon build: comment out definitions of unused macros * sed/compile.c (OPEN_BRACE): Comment out definition, to prevent gcc-5.2 from failing with -Werror -Wunused-macros. (END_ERRORS): Likewise. 2015-07-30 Jim Meyering tests: avoid false-failure on system with no ja_JP.eucJP locale * testsuite/invalid-mb-seq-UMR.sh: Use the configure-detected locale name, $LOCALE_JA, rather than hard-coding ja_JP.eucJP. If the required type of locale was not detected, then skip this test. Reported by Assaf Gordon in http://bugs.gnu.org/21124 2015-07-24 Jim Meyering maint: remove dead code * sed/utils.c (ck_mkstemp): Remove $TMP/$TMPDIR-envvar-using code that would have been run only if this function's second parameter were to be NULL, but that could never happen, given the current and sole caller. Also, use mode_t as the type for the saved umask value, not "int". Also, move each declaration down to point of definition. * sed/utils.h (ck_mkstemp): Mark each of the four parameters as nonnull. 2015-07-15 Stanislav Brabec sed: fix --follow-symlinks to work when reading stdin When reading from stdin, use of --follow-symlinks would cause failure: $ echo abc | sed --follow-symlinks s/a/d/ sed: cannot stat -: No such file or directory * sed/execute.c (open_next_file): Set input->in_file_name earlier, so we can rearrange logic to avoid calling follow_symlink("-"). * testsuite/follow-symlinks-stdin.sh: New file. * testsuite/Makefile.am (T): Add it. * NEWS (Bug fixes): Mention it. Bug introduced by commit v4.2.1-13-g84066bf. http://bugs.gnu.org/20795 2015-06-23 Jim Meyering typo fix: s/singlebyte/single-byte/ * sed/compile.c (compile_program, normalize_text): Fix typo. * sed/mbcs.c (brlen): Likewise. 2015-06-20 Jim Meyering sed -i: do not treat "-" as a file name Most GNU utilities treat "-" as standard input, sed itself does that -- in most contexts. However, since the addition of support for --in-place (-i) in sed-4.0, sed -i has treated a "-" argument as a file name, i.e., like ./-. Now, that usage evokes a diagnostic: $ sed -i s/a/b/ - sed: couldn't edit -: is a terminal If you require the old behavior, specify the file name as "./-". Prompted by the report/patch from Stanislav Brabec in http://bugs.gnu.org/20796 to document the strangely inconsistent legacy behavior. * sed/execute.c (open_next_file): Call panic before even attempting to operate on the file descriptor. * testsuite/in-place-hyphen.sh: New file. Test for this. * testsuite/Makefile.am (T): Add it. * NEWS (Feature removal): Mention it. Admittedly, the old behavior feels more like a misfeature. 2015-06-20 Jim Meyering tests: initialize $fail to 0 * testsuite/range-overlap.sh: Otherwise, with $fail set to nonzero in the environment, this test would report failure, even when it should pass. 2015-05-10 Jim Meyering build: avoid warning about implicit declaration of unlink * sed/sed.c: Include , for declaration of unlink. Without this change, commit makes sed v4.2.2-84-g7689015 fail to compile when configured with --enable-gcc-warnings. sed -i: don't leave behind a temporary sedXXXXXX file For example, running a command like "sed -i s//b/ F" would fail to remove a temporary file named sedXXXXXX (for random XXXXXX) in the directory alongside F. * sed/sed.c (G_file_to_unlink): New global. (register_cleanup_file, cancel_cleanup, cleanup): New functions. (main): Call atexit. * sed/execute.c (open_next_file): Register for unlink. (closedown): Call cancel_cleanup right after the rename. * sed/sed.h: Declare two of the new functions. * NEWS (Bug fixes): Mention it. * testsuite/temp-file-cleanup.sh: New file. Test for this. * testsuite/Makefile.am (T): Add it. Reported by David Jones in http://bugs.gnu.org/20002. 2015-05-07 Jim Meyering maint: remove stray blank line in the middle of a comment * sed/utils.c (ck_mkstemp): As above. tests: skip the new test in presence of buggy valgrind * testsuite/invalid-mb-seq-UMR.sh: Upon failure due to a valgrind-internal assertion, skip this test. Reported by Norihiro Tanaka in http://debbugs.gnu.org/20490 2015-05-06 Norihiro Tanaka sed: fix mishandling of overlapping address ranges When the line number ranges of two or more editing commands overlap, sed applies all commands but the first to a line that is just beyond the union of all ranges. E.g., with this command, seq 9|sed '2,7d;3,6d', sed would mistakenly delete line 8. * sed/execute.c (match_an_address_p) [ADDR_IS_NUM]: Make this function work also in this case. (match_address_p): Move the ADDR_IS_NUM + ...->line_number == ...->addr_number comparison "up" into match_an_address_p, so we can hoist two similar if/return blocks out of "if" and "else" branches. Change so that this function returns false when the current line number is outside the specified range. * testsuite/range-overlap.sh: New file, to test for this. * testsuite/Makefile.am (T): Add it to the list. * NEWS (Bug fixes): Mention it. Reported as http://bugs.gnu.org/19899 2015-05-04 Jim Meyering build: do not require autoconf.git * configure.ac: Use AC_CONFIG_MACRO_DIR, not the new (as yet unreleased) AC_CONFIG_MACRO_DIRS. Reported by Hongyi Zhao in http://debbugs.gnu.org/20424 2015-05-03 Jim Meyering maint: housekeeping: sync a few rules from coreutils * cfg.mk (sc_preprocessor_indentation): New rule, from coreutils. (local-checks-to-skip): Exempt it, for now. (sc_THANKS_in_sorted): Another rule from coreutils. Enabled. (announcement_Cc_): Define, so the auto-generated announcement message template includes sed-devel@ as a recipient. (update-copyright-env) [UPDATE_COPYRIGHT_USE_INTERVALS]: Update to use =2, rather than=1, so that the next time we update copyright dates, it will collapse a few more ranges. maint: tweak comments and rearrange code slightly * sed/execute.c (str_append): Trivial NSC code transformation. (str_append_modified): Tweak another comment. tests: test for just-fixed UMR bug * testsuite/invalid-mb-seq-UMR.sh: New file. Test for just-fixed bug. * testsuite/Makefile.am (T): New variable. Add the two init.sh-using test names here, rather than in two separate lists. (SEDTESTS, EXTRA_DIST): Use $(T). (EXTRA_DIST): Add init.cfg. * testsuite/init.cfg: New file, to add functions from coreutils. (require_valgrind_, print_ver_): New functions. 2015-05-03 Norihiro Tanaka sed: avoid a UMR bug when processing an invalid multibyte sequence * sed/execute.c (str_append_modified): When attempting to append an invalid multibyte sequence, e.g., from the RHS of a substitution expression, we would read uninitialized memory. Don't capitalize first word of diagnostic; remove exclamation point. Wrap new translatable strings in _(...). * NEWS (Bug fixes): Mention it. This addresses http://debbugs.gnu.org/20490 2015-05-02 Jim Meyering maint: remove SED_FEATURE_VERSION; use PACKAGE_VERSION instead * configure.ac (SED_FEATURE_VERSION): Remove definition et al. * sed/compile.c (compile_program): Remove sole use. Use PACKAGE_VERSION instead. maint: add "/*~" to build-aux/ and m4/.gitignore files Otherwise, bootstrap would add entries for individual file names. * build-aux/.gitignore: Add /*~ * m4/.gitignore: Likewise. 2015-05-01 Jim Meyering maint: enable sc_file_system * cfg.mk (local-checks-to-skip): Remove its exemption. * BUGS (characters): s/filesystem/file system/ * ChangeLog-2014 (setup_jump): Likewise. * doc/sed-in.texi: Likewise. * doc/sed.texi: Likewise. build: require gettext 0.19.2, perl-5.5 * configure.ac: Require gettext-0.19.2. * bootstrap.conf: List perl and gettext version requirements. 2015-04-27 Jim Meyering tests: set sed's expected failure status to 4 * testsuite/help-version.sh (expected_failure_status_sed): As above. 2015-04-26 Jim Meyering maint: invoke make via $(MAKE), not via "make" * Makefile.am (html): Invoke make via $(MAKE). (full-distcheck): Likewise. use generated version.[ch] and add init.sh and help-version.sh * Makefile.am (DISTCLEANFILES): Initialize. (EXTRA_DIST): Add .version. (BUILT_SOURCES): Set to .version. (.version): New rule. (THANKS): Depend on .version. * sed/Makefile.am: Add rules to build version.[ch]. * sed/sed.c: Include version.h, and use the new variable, Version. * testsuite/Makefile.am: Arrange to distribute and run the new help-version.sh test. Also distribute the new init.sh. * testsuite/help-version.sh: New file. * testsuite/init.sh: Likewise. * .gitignore: Ignore the generated files, .version and src/version.[ch]. 2015-03-01 Jim Meyering maint: COPYRIGHT_YEAR: remove, along with tests that use it This is covered by other tests, and the general --help/--version checks provided by help-version.sh (about to be added) are more thorough. * configure.ac (COPYRIGHT_YEAR): Remove definition and AC_SUBST. * Makefile.am (dist-hook): Remove test. * testsuite/Makefile.am (SEDTESTS): Remove "version". * testsuite/version.gin: Remove file. * .gitignore: Remove mention of generated file, version.good. 2015-01-01 Jim Meyering maint: initialize an mbstate_t with "= { 0, };", rather than with memset I.e., use this: mbstate_t mbstate = { 0, }; rather than this: mbstate_t mbstate; memset (&cur_stat, 0, sizeof (mbstate_t)); Also, prefer "sizeof VAR" over sizeof(TYPE). * sed/execute.c (execute_program, translate_mb): As above. * sed/compile.c (match_slash, compile_program, normalize_text): Likewise. maint: enable sc_prohibit_long_lines * cfg.mk (local-checks-to-skip): Remove the overall exemption, but continue to exempt some files in testsuite/. * configure.ac: Split a long line. * doc/sed-in.texi: Split long lines. * doc/sed.texi: Likewise. * sed/compile.c: Likewise. * sed/execute.c: Split long lines. (execute_program): Reorder "#ifdef HAVE_POPEN" and "if (pipe_fp != NULL)" if/else blocks to make the code clearer and (incidentally) the lines shorter, due to a decrease in nesting level. * sed/sed.c: Split long lines. * sed/utils.c: Likewise. * sed/utils.h: Likewise. maint: translate_mb: new function factored out of 'y'-handling case * sed/execute.c (translate_mb): New function, factored out of the "case 'y':" part of... (execute_program): ...here. maint: remove more unnecessary declarations * sed/execute.c: Remove now-duplicate forward declarations. maint: update copyright year ranges to include 2015; update gnulib 2014-12-20 Jim Meyering maint: enable sc_prohibit_strncpy check * cfg.mk (exclude_file_name_regexp--sc_prohibit_strncpy): Remove its exemption. * sed/compile.c (compile_program): Remove two misuses of strncpy; use memcpy instead. * sed/execute.c (do_list): Likewise, for one instance. maint: begin to use "size_t" more, where appropriate * sed/execute.c (do_list): Update some "int" locals to "size_t", and combine a few declarations with needlessly separate initializations. maint: update .gitignore * .gitignore: Ignore build artifacts. maint: remove po/sed.pot from version control * po/sed.pot: Remove file. It is unnecessary and undesirable to version-control a file like this (mechanically-derived). build: update build tool version requirements * bootstrap.conf (buildreq): Blindly update tool-version number requirements from those required by coreutils. maint: fix "make distcheck" failure * Makefile.am (EXTRA_DIST): List THANKS.in, not THANKS. Add .mailmap, too. build: build-aux/texi2dvi: remove file * build-aux/texi2dvi: Remove file. It was out of date and could no longer parse our .texi files. * Makefile.am (EXTRA_DIST): Don't list it here. * cfg.mk: Remove an exemption for that file. * doc/Makefile.am (TEXI2DVI): Remove definition. 2014-12-16 Jim Meyering remove support for the 'L' (fmt/flow-paragraph) command * sed/Makefile.am (sed_SOURCES): Remove fmt.c. * sed/execute.c (execute_program): * sed/fmt.c: Remove file. * sed/sed.h (fmt): Remove declaration. * doc/sed.texi: Remove documentation for 'L' command. * doc/sed-in.texi: Likewise. * NEWS (Feature removal): Document it. Prompted by the report from Jodie Cunningham that using this command with a large number could cause sed to segfault: https://bugs.launchpad.net/ubuntu/+source/sed/+bug/1400575 2014-12-01 Jim Meyering doc: NEWS: mention the gzip->xz release tarball switch * NEWS (Build-related): Mention that we are now distributing only .tar.xz archives; no more gzip-compressed tarballs. build: avoid warning about unused fchown result * sed/execute.c: Include "ignore-value.h". (closedown): Explicitly ignore fchown return value. * bootstrap.conf (gnulib_modules): Add ignore-value. * lib/.gitignore: Ignore the new .h file. * .gitignore: Ignore all **~ backup files. 2014-11-30 Jim Meyering build: use gnulib's manywarnings module bootstrap.conf: Add the module name. configure.ac: Copy boilerplate and exclusions from coreutils. sed/Makefile.am (AM_CFLAGS): Define in terms of the two new variables, $(WARN_CFLAGS) and $(WERROR_CFLAGS). m4/.gitignore: Add the two new .m4 files. maint: add three casts-in-initialization to avoid warnings * sed/compile.c (special_files): When initializing, cast string literals to (char *) to avoid warnings. maint: avoid a gcc "const discard" warning * sed/compile.c (compile_program): Make a local "const" to avoid discarding "const" attribute in assignment. maint: avoid more gcc warnings: apply _GL_ATTRIBUTE_PURE * sed/utils.h (get_buffer, size_buffer): Make each functions sole parameter "const" and declare the function itself with the "pure" attribute. * sed/utils.c (get_buffer, size_buffer): Adjust definitions to match. maint: add _Noreturn attribute to two functions This avoids two gcc may-be-noreturn warnings: * sed/sed.h (bad_prog): Add _Noreturn attribute. * sed/utils.h (panic): Likewise. maint: manually convert K&R to ANSI style decls; and add "static" Convert many function definitions from K&R to ANSI style. When possible, make a function static and remove the immediately- preceding declaration of that same function. Also, add the const to a few declarations. * sed/compile.c: As above. (special_files): Likewise. * sed/execute.c: Likewise. * sed/fmt.c: Likewise. * sed/mbcs.c: Likewise. * sed/regexp.c: Likewise. * sed/sed.c: Likewise. * sed/sed.h: Likewise. * sed/utils.c: Likewise. maint: declare several "int" locals to be of type size_t * sed/compile.c (compile_program): Using "int" was wrong: not only too narrow, but would also evoke warnings from gcc when comparing with variables of unsigned type. maint: avoid gcc warning about unused macro * sed/regexp.c (END_ERRORS): Remove definition of unused macro. maint: avoid false-positive used-uninit. warning from gcc * sed/sed.h (IF_LINT): Define. * sed/compile.c (snarf_char_class) [lint]: Use it to initialize DELIM, so that gcc doesn't report it is used uninitialized. maint: enable sc_prohibit_empty_lines_at_EOF check * cfg.mk (exclude_file_name_regexp--sc_prohibit_empty_lines_at_EOF): * cfg.mk (local-checks-to-skip): Remove its exemption. Adjust many files to end in a single newline, but exempt a few test suite input and output files. * testsuite/fasts.sed: Remove three empty lines at EOF. * BUGS: Remove empty line at EOF. * sed/mbcs.c: Likewise. * sed/sed.h: Likewise. * testsuite/BOOST.tests: Likewise. * testsuite/dc.inp: Add a newline at EOF; there was none. * testsuite/flipcase.sed: Likewise. * testsuite/utf8-1.good: Likewise. * testsuite/utf8-1.inp: Likewise. * testsuite/utf8-2.good: Likewise. * testsuite/utf8-2.inp: Likewise. * testsuite/utf8-3.good: Likewise. * testsuite/utf8-3.inp: Likewise. * testsuite/utf8-4.good: Likewise. * testsuite/utf8-4.inp: Likewise. * testsuite/xbxcx.good: Likewise. * testsuite/xbxcx.inp: Likewise. * testsuite/xemacs.good: Likewise. * testsuite/xemacs.inp: Likewise. maint: enable sc_prohibit_tab_based_indentation syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. Exempt all files under testsuite/, for now, and a few others. * .gitmodules: Change each leading TAB to 8 spaces spaces. * BUGS: Likewise. * NEWS: Likewise. * configure.ac: Likewise. * sed/compile.c: Likewise. * sed/execute.c: Likewise. * sed/fmt.c: Likewise. * sed/regexp.c: Likewise. * sed/sed.c: Likewise. * sed/sed.h: Likewise. * sed/utils.c: Likewise. * sed/utils.h: Likewise. * COPYING: Update from coreutils. maint: enable sc_m4_quote_check syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * configure.ac: Fix the under-quoting problems thus exposed. maint: enable sc_cast_of_argument_to_free syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * testsuite/tst-rxspencer.c (mb_test): Remove a cast. maint: enable sc_cast_of_alloca_return_value syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * sed/regexp.c (match_regex): Do not cast return value of alloca. maint: enable sc_po_check syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * po/POTFILES.in: Adjust to reflect added/removed file names. maint: enable sc_prohibit_always_true_header_tests syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. Exempt configure.ac. maint: enable sc_makefile_at_at_check syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * testsuite/Makefile.am (LDADD, TESTS_ENVIRONMENT): Use $(...) rather than obsolescent @...@ notation. maint: enable sc_space_tab syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * doc/groupify.sed: Reverse SP-TAB sequences. * sed/compile.c (compile_program): Remove spaces before TAB. * sed/execute.c (append_replacement): Likewise. maint: enable sc_program_name syntax check * cfg.mk (local-checks-to-skip): Remove its exemption. * sed/sed.c: Include progname.h". (program_name): Remove now-unnecessary declaration. (main): Call set_program_name. * bootstrap.conf (gnulib_modules): Add progname. * lib/.gitignore: Append the two new names. maint: enable more checks * cfg.mk (local-checks-to-skip): Remove exemptions for immutable NEWS as well as the two config.h checks. Also accept "sed.h", since the first thing it does is to include . * sed/sed.h: Change "config.h" to , for consistency with other GNU tools. * sed/utils.c: Likewise, here and for each of the following. * testsuite/bug-regex10.c: * testsuite/bug-regex11.c: * testsuite/bug-regex12.c: * testsuite/bug-regex13.c: * testsuite/bug-regex14.c: * testsuite/bug-regex15.c: * testsuite/bug-regex16.c: * testsuite/bug-regex21.c: * testsuite/bug-regex7.c: * testsuite/bug-regex8.c: * testsuite/bug-regex9.c: * testsuite/runptests.c: * testsuite/runtests.c: * testsuite/tst-boost.c: * testsuite/tst-pcre.c: * testsuite/tst-regex.c: * testsuite/tst-regex2.c: * testsuite/tst-rxspencer.c: maint: enable sc_const_long_option syntax check * sed/sed.c (main): Add "const" to decl of longopts. * cfg.mk (local-checks-to-skip): Remove its exemption. maint: enable double-word prohibition * cfg.mk (local-checks-to-skip): Remove exemption for sc_prohibit_doubled_word. Exempt 4 files under testsuite/. doc: remove doubled word * doc/sed-in.texi: Remove doubled "the". * doc/sed.texi: Likewise. maint: add .prev-version * .prev-version: New file. maint: turn on the test that prohibits use of HAVE_CONFIG_H * cfg.mk (local-checks-to-skip): Remove the exemption here. * testsuite/bug-regex10.c: Remove the use of the offending macro. * testsuite/bug-regex11.c: Likewise. * testsuite/bug-regex12.c: Likewise. * testsuite/bug-regex13.c: Likewise. * testsuite/bug-regex14.c: Likewise. * testsuite/bug-regex15.c: Likewise. * testsuite/bug-regex16.c: Likewise. * testsuite/bug-regex21.c: Likewise. * testsuite/bug-regex7.c: Likewise. * testsuite/bug-regex8.c: Likewise. * testsuite/bug-regex9.c: Likewise. * testsuite/runptests.c: Likewise. * testsuite/runtests.c: Likewise. * testsuite/tst-boost.c: Likewise. * testsuite/tst-pcre.c: Likewise. * testsuite/tst-regex.c: Likewise. * testsuite/tst-regex2.c: Likewise. * testsuite/tst-rxspencer.c: Likewise. maint: generate THANKS from THANKS.in * THANKS.in: Renamed from THANKS, adjusted formatting and added header. Also remove from this list of names those that are now generated automatically. * THANKS: Removed, now that it's generated. * thanks-gen: New file. * Makefile.am (EXTRA_DIST): Add thanks-gen (THANKS): New rule. * .mailmap: New file. maint: add cfg.mk, derived from grep's * cfg.mk (local-checks-to-skip): Exempt many failing tests. (old_NEWS_hash): Define. * NEWS: Tweak wording to avoid semblance of doubled word: in in-place. tests: hook up gnulib-tests * Makefile.am (SUBDIRS): Add gnulib-tests. * gnulib-tests/Makefile.am: New file. * configure.ac (AC_CONFIG_FILES): Add gnulib-tests/Makefile. * .gitignore: Ignore all but Makefile.am. maint: .gitignore: ignore more maint: remove ACLOCAL_AMFLAGS in favor of AC_CONFIG_MACRO_DIRS * Makefile.am (ACLOCAL_AMFLAGS): Remove this assignment. * configure.ac: Use this macro instead. maint: do not version-control generated files. * doc/sed.1: Remove. * po/sed.pot: Likewise. * po/Makevars: Likewise. * .gitignore: Add their names. maint: begin to modernize configure.ac * configure.ac: Turn on silent rules, by default. Use git-derived versions all the time. Required a new, signed v4.2.2 tag, which I've pushed. Distribute .tar.xz tarballs, not .bzip2 or .gz. * Makefile.am (AUTOMAKE_OPTIONS): Remove. * bootstrap.conf (gnulib_modules): Add git-version-gen. maint: convert obsolescent @VAR@ notation to $(VAR) * sed/Makefile.am (sed_LDADD): Use $(VAR), no @VAR@. maint: update copyright dates; use gnulib's update-copyright module * bootstrap.conf (gnulib_modules): Add update-copyright. Then, run "make update-copyright". * Makefile.am: Add copyright header. maint: remove all trailing white space First, run this command: git grep -l ' $'|xargs perl -pi -e 's/[ \t]+$//' Then some minor fix-up to make the two newly-failing tests pass once again: * testsuite/mac-mf.sed: Append this, s/ $//, to eliminate trailing spaces in the actual output. * testsuite/y-newline.good: Manually remove a trailing space between two concatenated prompts. maint: update autogenerated .gitignore files maint: update build/gnulib infrastructure; generate ChangeLog * ChangeLog-2014: Renamed from ChangeLog. * po/ChangeLog-2014: Renamed from po/ChangeLog. * Makefile.am (EXTRA_DIST): Add the two new file names. (dist-hook): Generate ChangeLog at tarball-creation time. (gen-ChangeLog): New rule. Just like the one in coreutils and grep. * bootstrap.conf (gnulib_modules): Add the modules, readme-release and gitlog-to-changelog, and revamp to be much more like grep and coreutils. * lib/Makefile.am: Update. maint: update bootstrap from gnulib 2014-09-06 Jim Meyering fixup: add the properly-named new test files * testsuite/Makefile.am (EXTRA_DIST): Add y-zero.*, not y-NUL.* here.