Change Log for PCRE2

--------------------

Version 10.32-RC1 10-September-2018

-----------------------------------

1. When matching using the the REG_STARTEND feature of the POSIX API with a

non-zero starting offset, unset capturing groups with lower numbers than a

group that did capture something were not being correctly returned as "unset"

(that is, with offset values of -1).

2. When matching using the POSIX API, pcre2test used to omit listing unset

groups altogether. Now it shows those that come before any actual captures as

"<unset>", as happens for non-POSIX matching.

3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",

whatever the build configuration was. It now correctly says "\R matches all

Unicode newlines" in the default case when --enable-bsr-anycrlf has not been

specified. Similarly, running "pcre2test -C bsr" never produced the result

ANY.

4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing

multi-code-unit characters caused bad behaviour and possibly a crash. This

issue was fixed for other kinds of repeat in release 10.20 by change 19, but

repeating character classes were overlooked.

5. pcre2grep now supports the inclusion of binary zeros in patterns that are

read from files via the -f option.

6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.

7. Added --enable-jit=auto support to configure.ac.

8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit

modes for the benefit of m68k, where pointers can be 16-bit aligned. The

dummies force 32-bit alignment and this ensures that the structure is a

multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other

architectures, alignment requirements take care of this automatically.

9. When returning an error from pcre2_pattern_convert(), ensure the error

offset is set zero for early errors.

10. A number of patches for Windows support from Daniel Richard G:

  (a) List of error numbers in Runtest.bat corrected (it was not the same as in

      Runtest).

  (b) pcre2grep snprintf() workaround as used elsewhere in the tree.

  (c) Support for non-C99 snprintf() that returns -1 in the overflow case.

11. Minor tidy of pcre2_dfa_match() code.

12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer

use the stack for local workspace and local ovectors. Instead, an initial block

of stack is reserved, but if this is insufficient, heap memory is used. The

heap limit parameter now applies to pcre2_dfa_match().

13. If a "find limits" test of DFA matching in pcre2test resulted in too many

matches for the ovector, no matches were displayed.

14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as

EOF. The test looks to have come from a fuzzer.

15. If PCRE2 was built with a default match limit a lot greater than the

default default of 10 000 000, some JIT tests of the match limit no longer

failed. All such tests now set 10 000 000 as the upper limit.

16. Another Windows related patch for pcregrep to ensure that WIN32 is

undefined under Cygwin.

17. Test for the presence of stdint.h and inttypes.h in configure and CMake and

include whichever exists (stdint preferred) instead of unconditionally

including stdint. This makes life easier for old and non-standard systems.

18. Further changes to improve portability, especially to old and or non-

standard systems:

  (a) Put all printf arguments in RunGrepTest into single, not double, quotes,

      and use \0 not \x00 for binary zero.

  (b) Avoid the use of C++ (i.e. BCPL) // comments.

  (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of

      these now, if using MSVC or a standard C before C99, %lu is used with a

      cast if necessary.

19. Applied a contributed patch to CMakeLists.txt to increase the stack size

when linking pcre2test with MSVC. This gets rid of a stack overflow error in

the standard set of tests.

20. Output a warning in pcre2test when ignoring the "altglobal" modifier when

it is given with the "replace" modifier.

21. In both pcre2test and pcre2_substitute(), with global matching, a pattern

that matched an empty string, but never at the starting match offset, was not

handled in a Perl-compatible way. The pattern /(

a pattern. Because \G is in a lookbehind assertion, there has to be a

"bumpalong" before there can be a match. The automatic "advance by one

character after an empty string match" rule is therefore inappropriate. A more

complicated algorithm has now been implemented.

22. When checking to see if a lookbehind is of fixed length, lookaheads were

correctly ignored, but qualifiers on lookaheads were not being ignored, leading

to an incorrect "lookbehind assertion is not fixed length" error.

23. The VERSION condition test was reading fractional PCRE2 version numbers

such as the 04 in 10.04 incorrectly and hence giving wrong results.

24. Updated to Unicode version 11.0.0. As well as the usual addition of new

scripts and characters, this involved re-jigging the grapheme break property

algorithm because Unicode has changed the way emojis are handled.

25. Fixed an obscure bug that struck when there were two atomic groups not

separated by something with a backtracking point. There could be an incorrect

backtrack into the first of the atomic groups. A complicated example is

/(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP

shouldn't find a MARK (because is in an atomic group), but it did.

26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set

a list of modifiers for all subsequent patterns - only those that the script

recognizes are meaningful; (2) #subject lines can be used to set or unset a

default "mark" modifier; (3) Unsupported #command lines give a warning when

they are ignored; (4) Mark data is output only if the "mark" modifier is

present.

27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.

28. A (*MARK) name was not being passed back for positive assertions that were

terminated by (*ACCEPT).

29. Add support for \N{U+dddd}, but only in Unicode mode.

30. Add support for (?^) for unsetting all imnsx options.

31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose

code point was less than 256 and that were recognized by the lookup table

generated by pcre2_maketables(), which uses isspace() to identify white space.

Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,

U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by

Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.

32. In certain circumstances, option settings within patterns were not being

correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly

matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the

end of its group during the parse process, but without another setting such as

(?m) the compile phase got it right.) This bug was introduced by the

refactoring in release 10.23.

33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to

define memmove() as function call to bcopy(). This hasn't been tested for a

long time because in pcre2test the result of memmove() was being used, whereas

bcopy() doesn't return a result. This feature is now refactored always to call

an emulation function when there is no memmove(). The emulation makes use of

bcopy() when available.

34. When serializing a pattern, set the memctl, executable_jit, and tables

fields (that is, all the fields that contain pointers) to zeros so that the

result of serializing is always the same. These fields are re-set when the

pattern is deserialized.

35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated

negative class with no characters less than 0x100 followed by a positive class

with only characters less than 0x100, the first class was incorrectly being

auto-possessified, causing incorrect match failures.

36. Removed the character type bit ctype_meta, which dates from PCRE1 and is

not used in PCRE2.

37. Tidied up unnecessarily complicated macros used in the escapes table.

38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted

from distribution tarballs, owing to a typo in Makefile.am which had

testoutput8-16-3 twice. Now fixed.

39. If the only branch in a conditional subpattern was anchored, the whole

subpattern was treated as anchored, when it should not have been, since the

assumed empty second branch cannot be anchored. Demonstrated by test patterns

such as /(?(1)^())b/ or /(?(?=^))b/.

40. A repeated conditional subpattern that could match an empty string was

always assumed to be unanchored. Now it it checked just like any other

repeated conditional subpattern, and can be found to be anchored if the minimum

quantifier is one or more. I can't see much use for a repeated anchored

pattern, but the behaviour is now consistent.

41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint

(for an event that could never occur but you had to have external information

to know that).

42. If before the first match in a file that was being searched by pcre2grep

there was a line that was sufficiently long to cause the input buffer to be

expanded, the variable holding the location of the end of the previous match

was being adjusted incorrectly, and could cause an overflow warning from a code

sanitizer. However, as the value is used only to print pending "after" lines

when the next match is reached (and there are no such lines in this case) this

bug could do no damage.

Version 10.31 12-February-2018

------------------------------

1. Fix typo (missing ]) in VMS code in pcre2test.c.

2. Replace the replicated code for matching extended Unicode grapheme sequences

(which got a lot more complicated by change 10.30/49) by a single subroutine

that is called by both pcre2_match() and pcre2_dfa_match().

3. Add idempotent guard to pcre2_internal.h.

4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and

PCRE2_CONFIG_COMPILED_WIDTHS.

5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is

defined (e.g. by --enable-never-backslash-C).

6. Defined public names for all the pcre2_compile() error numbers, and used

the public names in pcre2_convert.c.

7. Fixed a small memory leak in pcre2test (convert contexts).

8. Added two casts to compile.c and one to match.c to avoid compiler warnings.

9. Added code to pcre2grep when compiled under VMS to set the symbol

PCRE2GREP_RC to the exit status, because VMS does not distinguish between

exit(0) and exit(1).

10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain

about a bad option only if the following argument item does not start with a

hyphen.

11. pcre2grep was truncating components of file names to 128 characters when

processing files with the -r option, and also (some very odd code) truncating

path names to 512 characters. There is now a check on the absolute length of

full path file names, which may be up to 2047 characters long.

12. When an assertion contained (*ACCEPT) it caused all open capturing groups

to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to

misbehaviour for subsequent references to groups that started outside the

assertion. ACCEPT in an assertion now closes only those groups that were

started within that assertion. Fixes oss-fuzz issues 3852 and 3891.

13. Multiline matching in pcre2grep was misbehaving if the pattern matched

within a line, and then matched again at the end of the line and over into

subsequent lines. Behaviour was different with and without colouring, and

sometimes context lines were incorrectly printed and/or line endings were lost.

All these issues should now be fixed.

14. If --line-buffered was specified for pcre2grep when input was from a

compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be

ignored for compressed files.)

15. Although pcre2_jit_match checks whether the pattern is compiled

in a given mode, it was also expected that at least one mode is available.

This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION

when the pattern is not optimized by JIT at all.

16. The line number and related variables such as match counts in pcre2grep

were all int variables, causing overflow when files with more than 2147483647

lines were processed (assuming 32-bit ints). They have all been changed to

unsigned long ints.

17. If a backreference with a minimum repeat count of zero was first in a

pattern, apart from assertions, an incorrect first matching character could be

recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set

as the first character of a match.

18. Characters in a leading positive assertion are considered for recording a

first character of a match when the rest of the pattern does not provide one.

However, a character in a non-assertive group within a leading assertion such

as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an

infelicity rather than an outright bug, because it did not affect the result of

a match, just its speed. (In fact, in this case, the starting 'a' was

subsequently picked up in the study.)

19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"

instead of "RRETURN" saves unwinding the backtracks in these cases (only one

didn't).

20. Allocate a single callout block on the stack at the start of pcre2_match()

and set its never-changing fields once only. Do the same for pcre2_dfa_match().

21. Save the extra compile options (set in the compile context) with the

compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS

to retrieve them, and update pcre2test to show them.

22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new

field callout_flags in callout blocks. The bits are set by pcre2_match(), but

not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts

if the callout_extra subject modifier is set. These bits are provided to help

with tracking how a backtracking match is proceeding.

23. Updated the pcre2demo.c demonstration program, which was missing the extra

code for -g that handles the case when \K in an assertion causes the match to

end at the original start point. Also arranged for it to detect when \K causes

the end of a match to be before its start.

24. Similar to 23 above, strange things (including loops) could happen in

pcre2grep when \K was used in an assertion when --colour was used or in

multiline mode. The "end at original start point" bug is fixed, and if the end

point is found to be before the start point, they are swapped.

25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT

matching (both pcre2_match() and pcre2_dfa_match()) and the matched string

started with the first code unit of a newline sequence, matching failed because

it was not tried at the newline.

26. Code for giving up a non-partial match after failing to find a starting

code unit anywhere in the subject was missing when searching for one of a

number of code units (the bitmap case) in both pcre2_match() and

pcre2_dfa_match(). This was a missing optimization rather than a bug.

27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a

pointer argument rather than a code unit value. This should not have affected

the generated code.

28. The JIT compiler has been updated.

29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().

This could not actually cause a crash because it was always used in a memcpy()

call with zero length.

30. Some internal structures have a variable-length ovector[] as their last

element. Their actual memory is obtained dynamically, giving an ovector of

appropriate length. However, they are defined in the structure as

ovector[NUMBER], where NUMBER is large so that array bound checkers don't

grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing

groups, making the ovector larger than this. The number has been increased to

131072, which allows for the maximum number of captures (65535) plus the

overall match. This fixes oss-fuzz issue 5415.

31. Auto-possessification at the end of a capturing group was dependent on what

follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused

incorrect behaviour when the group was called recursively from elsewhere in the

pattern where something different might follow. This bug is an unforseen

consequence of change #1 for 10.30 - the implementation of backtracking into

recursions. Iterators at the ends of capturing groups are no longer considered

for auto-possessification if the pattern contains any recursions. Fixes

Bugzilla #2232.

Version 10.30 14-August-2017

----------------------------

1. The main interpreter, pcre2_match(), has been refactored into a new version

that does not use recursive function calls (and therefore the stack) for

remembering backtracking positions. This makes --disable-stack-for-recursion a

NOOP. The new implementation allows backtracking into recursive group calls in

patterns, making it more compatible with Perl, and also fixes some other

hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because

the old code had a number of fudges to try to reduce stack usage. It seems to

run no slower than the old code.

A number of bugs in the refactored code were subsequently fixed during testing

before release, but after the code was made available in the repository. These

bugs were never in fully released code, but are noted here for the record.

  (a) If a pattern had fewer capturing parentheses than the ovector supplied in

      the match data block, a memory error (detectable by ASAN) occurred after

      a match, because the external block was being set from non-existent

      internal ovector fields. Fixes oss-fuzz issue 781.

  (b) A pattern with very many capturing parentheses (when the internal frame

      size was greater than the initial frame vector on the stack) caused a

      crash. A vector on the heap is now set up at the start of matching if the

      vector on the stack is not big enough to handle at least 10 frames.

      Fixes oss-fuzz issue 783.

  (c) Handling of (*VERB)s in recursions was wrong in some cases.

  (d) Captures in negative assertions that were used as conditions were not

      happening if the assertion matched via (*ACCEPT).

  (e) Mark values were not being passed out of recursions.

  (f) Refactor some code in do_callout() to avoid picky compiler warnings about

      negative indices. Fixes oss-fuzz issue 1454.

  (g) Similarly refactor the way the variable length ovector is addressed for

      similar reasons. Fixes oss-fuzz issue 1465.

2. Now that pcre2_match() no longer uses recursive function calls (see above),

the "match limit recursion" value seems misnamed. It still exists, and limits

the depth of tree that is searched. To avoid future confusion, it has been

renamed as "depth limit" in all relevant places (--with-depth-limit,

(*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still

available for backwards compatibility.

3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:

  (a) Check for malloc failures when getting memory for the ovector (POSIX) or

      the match data block (non-POSIX).

4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property

for a character with a code point greater than 0x10ffff (the Unicode maximum)

caused a crash.

5. If a lookbehind assertion that contained a back reference to a group

appearing later in the pattern was compiled with the PCRE2_ANCHORED option,

undefined actions (often a segmentation fault) could occur, depending on what

other options were set. An example assertion is (?

reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.

6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for

pcre2test to use it to output the frame size when the "framesize" modifier is

given.

7. Reworked the recursive pattern matching in the JIT compiler to follow the

interpreter changes.

8. When the zero_terminate modifier was specified on a pcre2test subject line

for global matching, unpredictable things could happen. For example, in UTF-8

mode, the pattern //g,zero_terminate read random memory when matched against an

empty string with zero_terminate. This was a bug in pcre2test, not the library.

9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out

of the section that is compiled when Unix-style directory scanning is

available, and into a new section that is always compiled for Windows.

10. In pcre2test, explicitly close the file after an error during serialization

or deserialization (the "load" or "save" commands).

11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.

12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with

a NULL pattern pointer when Unicode support is available.

13. When the 32-bit library was being tested by pcre2test, error messages that

were longer than 64 code units could cause a buffer overflow. This was a bug in

pcre2test.

14. The alternative matching function, pcre2_dfa_match() misbehaved if it

encountered a character class with a possessive repeat, for example [a-f]{3}+.

15. The depth (formerly recursion) limit now applies to DFA matching (as

of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA

matching to find the minimum value for this limit.

16. Since 10.21, if pcre2_match() was called with a null context, default

memory allocation functions were used instead of whatever was used when the

pattern was compiled.

17. Changes to the pcre2test "memory" modifier on a subject line. These apply

only to pcre2_match():

  (a) Warn if null_context is set on both pattern and subject, because the

      memory details cannot then be shown.

  (b) Remember (up to a certain number of) memory allocations and their

      lengths, and list only the lengths, so as to be system-independent.

      (In practice, the new interpreter never has more than 2 blocks allocated

      simultaneously.)

18. Make pcre2test detect an error return from pcre2_get_error_message(), give

a message, and abandon the run (this would have detected #13 above).

19. Implemented PCRE2_ENDANCHORED.

20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement

the --output=text (-O) option and the inbuilt callout echo.

21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and

single-branch conditions with a false condition (e.g. DEFINE) at the start of a

branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as

anchored.

22. Added an explicit limit on the amount of heap used by pcre2_match(), set by

pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the

heap limit along with other pattern information, and to find the minimum when

the find_limits modifier is set.

23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled

pattern is set up so as to initialize any padding the compiler might have

included. This avoids valgrind warnings when a compiled pattern is copied, in

particular when it is serialized.

24. Remove a redundant line of code left in accidentally a long time ago.

25. Remove a duplication typo in pcre2_tables.c

26. Correct an incorrect cast in pcre2_valid_utf.c

27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the

tests to improve coverage.

28. Some fixes/tidies as a result of looking at Coverity Scan output:

    (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.

    (b) Added some casts to avoid "suspicious implicit sign extension".

    (c) Resource leaks in pcre2test in rare error cases.

    (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge

        for checking at compile time that tables are the right size.

    (e) Add missing "fall through" comment.

29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.

30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.

31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in

pcre2test, a crash could occur.

32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so

that all the tests can run with clang's sanitizing options.

33. Implement extra compile options in the compile context and add the first

one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.

34. Implement newline type PCRE2_NEWLINE_NUL.

35. A lookbehind assertion that had a zero-length branch caused undefined

behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.

36. The match limit value now also applies to pcre2_dfa_match() as there are

patterns that can use up a lot of resources without necessarily recursing very

deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.

37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.

38. Fix returned offsets from regexec() when REG_STARTEND is used with a

starting offset greater than zero.

39. Implement REG_PEND (GNU extension) for the POSIX wrapper.

40. Implement the subject_literal modifier in pcre2test, and allow jitstack on

pattern lines.

41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.

42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit

of pcre2grep.

43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,

PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:

    (a) The -F option did not work for fixed strings containing \E.

    (b) The -w option did not work for patterns with multiple branches.

44. Added configuration options for the SELinux compatible execmem allocator in

JIT.

45. Increased the limit for searching for a "must be present" code unit in

subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are

much faster.

46. Arrange for anchored patterns to record and use "first code unit" data,

because this can give a fast "no match" without searching for a "required code

unit". Previously only non-anchored patterns did this.

47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.

48. Add the callout_no_where modifier to pcre2test.

49. Update extended grapheme breaking rules to the latest set that are in

Unicode Standard Annex #29.

50. Added experimental foreign pattern conversion facilities

(pcre2_pattern_convert() and friends).

51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE

is defined in a system header in cygwin. Also modified some of the #ifdefs in

pcre2grep related to Windows and Cygwin support.

52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a

character class is the last character in the class, Perl does not give a

warning. PCRE2 now also treats this as a literal.

53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was

not doing so for [\d-X] (and similar escapes), as is documented.

54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.

55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in

pcre2_compile() which could never actually trigger (code should have been cut

out when Unicode support is disabled).

Version 10.23 14-February-2017

------------------------------

1. Extended pcre2test with the utf8_input modifier so that it is able to

generate all possible 16-bit and 32-bit code unit values in non-UTF modes.

2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without

PCRE2_UCP set, a negative character type such as \D in a positive class should

cause all characters greater than 255 to match, whatever else is in the class.

There was a bug that caused this not to happen if a Unicode property item was

added to such a class, for example [\D\P{Nd}] or [\W\pL].

3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax

checking is now done in the pre-pass that identifies capturing groups. This has

reduced the amount of duplication and made the code tidier. While doing this,

some minor bugs and Perl incompatibilities were fixed, including:

  (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead

      of giving an invalid quantifier error.

  (b) {0} can now be used after a group in a lookbehind assertion; previously

      this caused an "assertion is not fixed length" error.

  (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with

      the name "DEFINE" exists. PCRE2 now does likewise.

  (d) A recursion condition test such as (?(R2)...) must now refer to an

      existing subpattern.

  (e) A conditional recursion test such as (?(R)...) misbehaved if there was a

      group whose name began with "R".

  (f) When testing zero-terminated patterns under valgrind, the terminating

      zero is now marked "no access". This catches bugs that would otherwise

      show up only with non-zero-terminated patterns.

  (g) A hyphen appearing immediately after a POSIX character class (for example

      /[[:ascii:]-z]/) now generates an error. Perl does accept this as a

      literal, but gives a warning, so it seems best to fail it in PCRE.

  (h) An empty \Q\E sequence may appear after a callout that precedes an

      assertion condition (it is, of course, ignored).

One effect of the refactoring is that some error numbers and messages have

changed, and the pattern offset given for compiling errors is not always the

right-most character that has been read. In particular, for a variable-length

lookbehind assertion it now points to the start of the assertion. Another

change is that when a callout appears before a group, the "length of next

pattern item" that is passed now just gives the length of the opening

parenthesis item, not the length of the whole group. A length of zero is now

given only for a callout at the end of the pattern. Automatic callouts are no

longer inserted before and after explicit callouts in the pattern.

A number of bugs in the refactored code were subsequently fixed during testing

before release, but after the code was made available in the repository. Many

of the bugs were discovered by fuzzing testing. Several of them were related to

the change from assuming a zero-terminated pattern (which previously had

required non-zero terminated strings to be copied). These bugs were never in

fully released code, but are noted here for the record.

  (a) An overall recursion such as (?0) inside a lookbehind assertion was not

      being diagnosed as an error.

  (b) In utf mode, the length of a *MARK (or other verb) name was being checked

      in characters instead of code units, which could lead to bad code being

      compiled, leading to unpredictable behaviour.

  (c) In extended /x mode, characters whose code was greater than 255 caused

      a lookup outside one of the global tables. A similar bug existed for wide

      characters in *VERB names.

  (d) The amount of memory needed for a compiled pattern was miscalculated if a

      lookbehind contained more than one toplevel branch and the first branch

      was of length zero.

  (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-

      terminated pattern, if a # comment ran on to the end of the pattern, one

      or more code units past the end were being read.

  (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.

      "{2,2") could cause reading beyond the pattern.

  (g) When reading a callout string, if the end delimiter was at the end of the

      pattern one further code unit was read.

  (h) An unterminated number after \g' could cause reading beyond the pattern.

  (i) An insufficient memory size was being computed for compiling with

      PCRE2_AUTO_CALLOUT.

  (j) A conditional group with an assertion condition used more memory than was

      allowed for it during parsing, so too many of them could therefore

      overrun a buffer.

  (k) If parsing a pattern exactly filled the buffer, the internal test for

      overrun did not check when the final META_END item was added.

  (l) If a lookbehind contained a subroutine call, and the called group

      contained an option setting such as (?s), and the PCRE2_ANCHORED option

      was set, unpredictable behaviour could occur. The underlying bug was

      incorrect code and insufficient checking while searching for the end of

      the called subroutine in the parsed pattern.

  (m) Quantifiers following (*VERB)s were not being diagnosed as errors.

  (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and

      PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.

  (o) If \Q was preceded by a quantified item, and the following \E was

      followed by '?' or '+', and there was at least one literal character

      between them, an internal error "unexpected repeat" occurred (example:

      /.+\QX\E+/).

  (p) A buffer overflow could occur while sorting the names in the group name

      list (depending on the order in which the names were seen).

  (q) A conditional group that started with a callout was not doing the right

      check for a following assertion, leading to compiling bad code. Example:

      /(?(C'XX))?!XX/

  (r) If a character whose code point was greater than 0xffff appeared within

      a lookbehind that was within another lookbehind, the calculation of the

      lookbehind length went wrong and could provoke an internal error.

  (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused

      an internal error. Now the hyphen is treated as a literal.

4. Back references are now permitted in lookbehind assertions when there are

no duplicated group numbers (that is, (?| has not been used), and, if the

reference is by name, there is only one group of that name. The referenced

group must, of course be of fixed length.

5. pcre2test has been upgraded so that, when run under valgrind with valgrind

support enabled, reading past the end of the pattern is detected, both when

compiling and during callout processing.

6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back

reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does

not recognize this syntax.

7. Automatic callouts are no longer generated before and after callouts in the

pattern.

8. When pcre2test was outputing information from a callout, the caret indicator

for the current position in the subject line was incorrect if it was after an

escape sequence for a character whose code point was greater than \x{ff}.

9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be

PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.

10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer

expansion when long lines are encountered. Original patch by Dmitry

Cherniachenko.

11. If pcre2grep was compiled with JIT support, but the library was compiled

without it (something that neither ./configure nor CMake allow, but it can be

done by editing config.h), pcre2grep was giving a JIT error. Now it detects

this situation and does not try to use JIT.

12. Added some "const" qualifiers to variables in pcre2grep.

13. Added Dmitry Cherniachenko's patch for colouring output in Windows

(untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment

variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.

14. Add the -t (grand total) option to pcre2grep.

15. A number of bugs have been mended relating to match start-up optimizations

when the first thing in a pattern is a positive lookahead. These all applied

only when PCRE2_NO_START_OPTIMIZE was *not* set:

    (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed

        both an initial 'X' and a following 'X'.

    (b) Some patterns starting with an assertion that started with .* were

        incorrectly optimized as having to match at the start of the subject or

        after a newline. There are cases where this is not true, for example,

        (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that

        start with spaces. Starting .* in an assertion is no longer taken as an

        indication of matching at the start (or after a newline).

16. The "offset" modifier in pcre2test was not being ignored (as documented)

when the POSIX API was in use.

17. Added --enable-fuzz-support to "configure", causing an non-installed

library containing a test function that can be called by fuzzers to be

compiled. A non-installed  binary to run the test function locally, called

pcre2fuzzcheck is also compiled.

18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and

which started with .* inside a positive lookahead was incorrectly being

compiled as implicitly anchored.

19. Removed all instances of "register" declarations, as they are considered

obsolete these days and in any case had become very haphazard.

20. Add strerror() to pcre2test for failed file opening.

21. Make pcre2test -C list valgrind support when it is enabled.

22. Add the use_length modifier to pcre2test.

23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and

'copy' modifiers.

24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it

is apparently needed there as well as in the function definitions. (Why did

nobody ask for this in PCRE1?)

25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to

PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard

compliant and unique.

26. pcre2-config --libs-posix was listing -lpcre2posix instead of

-lpcre2-posix. Also, the CMake build process was building the library with the

wrong name.

27. In pcre2test, give some offset information for errors in hex patterns.

This uses the C99 formatting sequence %td, except for MSVC which doesn't

support it - %lu is used instead.

28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to

pcre2test for testing it.

29. Fix small memory leak in pcre2test.

30. Fix out-of-bounds read for partial matching of /./ against an empty string

when the newline type is CRLF.

31. Fix a bug in pcre2test that caused a crash when a locale was set either in

the current pattern or a previous one and a wide character was matched.

32. The appearance of \p, \P, or \X in a substitution string when

PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL

dereference).

33. If the starting offset was specified as greater than the subject length in

a call to pcre2_substitute() an out-of-bounds memory reference could occur.

34. When PCRE2 was compiled to use the heap instead of the stack for recursive

calls to match(), a repeated minimizing caseless back reference, or a

maximizing one where the two cases had different numbers of code units,

followed by a caseful back reference, could lose the caselessness of the first

repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX

but didn't).

35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum

matching length and just records zero. Typically this happens when there are

too many nested or recursive back references. If the limit was reached in

certain recursive cases it failed to be triggered and an internal error could

be the result.

36. The pcre2_dfa_match() function now takes note of the recursion limit for

the internal recursive calls that are used for lookrounds and recursions within

the pattern.

37. More refactoring has got rid of the internal could_be_empty_branch()

function (around 400 lines of code, including comments) by keeping track of

could-be-emptiness as the pattern is compiled instead of scanning compiled

groups. (This would have been much harder before the refactoring of #3 above.)

This lifts a restriction on the number of branches in a group (more than about

1100 would give "pattern is too complicated").

38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern

auto_callout".

39. In a library with Unicode support, incorrect data was compiled for a

pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide

characters to match (for example, /[\s[:^ascii:]]/).

40. The callout_error modifier has been added to pcre2test to make it possible

to return PCRE2_ERROR_CALLOUT from a callout.

41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of

"<esc>[00m".

42. The limit in the auto-possessification code that was intended to catch

overly-complicated patterns and not spend too much time auto-possessifying was

being reset too often, resulting in very long compile times for some patterns.

Now such patterns are no longer completely auto-possessified.

43. Applied Jason Hood's revised patch for RunTest.bat.

44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.

45. Minor cosmetic fix to pcre2test: move a variable that is not used under

Windows into the "not Windows" code.

46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy

some of the code:

  * normalised the Windows condition by ensuring WIN32 is defined;

  * enables the callout feature under Windows;

  * adds globbing (Microsoft's implementation expands quoted args),

    using a tweaked opendirectory;

  * implements the is_*_tty functions for Windows;

  * --color=always will write the ANSI sequences to file;

  * add sequences 4 (underline works on Win10) and 5 (blink as bright

    background, relatively standard on DOS/Win);

  * remove the (char *) casts for the now-const strings;

  * remove GREP_COLOUR (grep's command line allowed the 'u', but not

    the environment), parsing GREP_COLORS instead;

  * uses the current colour if not set, rather than black;

  * add print_match for the undefined case;