Add cut operator (^) to grammar

[traviscross]

The cut operator (^) is a backtracking fence. Once the expression to its left succeeds, we become committed to the alternative; the remainder of the expression must parse successfully or parsing will fail. See Packrat Parsers Can Handle Practical Grammars in Mostly Constant Space, Mizushima et al., https://kmizu.github.io/papers/paste513-mizushima.pdf.

This operator solves a problem for us with C string literals. These literals cannot contain a null escape. But if we simply fail to lex the literal (e.g. c"\0"), we may instead lex it successfully as two separate tokens (`c "\0"), and that would be incorrect.

As long as we only use cut to express constraints that can be expressed in a regular language and we keep our alternations disjoint, the grammar can still be mechanically converted to a CFG.

Let's add the cut operator to our grammar and use it for C string literals and some similar constructs.

In the railroad diagrams, we'll render the cut as a "no backtracking" box around the expression or sequence of expressions after the cut. The idea is that once you enter the box the only way out is forward.

(H/t to @ehuss for suggesting the cut operator to solve this problem.)

cc @ehuss

---

This is stacked on #2097 and should merge after it.

[mattheww]

When you're defining a cut operator, it's important to specify the scope over
which it cancels re-attempts.

For the lexer's purposes it would be fine to make that scope unlimited, saying that if the right-hand side of a production containing a cut fails after reaching the cut then the entire lexing process fails.

But in this PR the nearest thing to a definition of the cut operator is the reference to https://kmizu.github.io/papers/paste513-mizushima.pdf .

That paper defines a cut operator with a narrower scope: it allows a cut only on the left-hand side of an ordered choice expression, and cancels only the re-attempt of the right-hand side of that expression.

That definition doesn't work for the positions in which this PR is placing cuts.

[mattheww]

If you're still planning to use prioritised choice more widely in the Reference, then (given that the Reference already has the notion of reserved forms) perhaps the simplest way to define cut is to say that:

FOO ← a ^ b

is a shorthand for

FOO ← a b
RESERVED_PREFIX_OF_FOO ← a

with RESERVED_PREFIX_OF_FOO appearing immediately after FOO in the lexical grammar's top-level ordered choice for tokens.

That characterisation also illustrates why adding the notion of a cut doesn't buy very much.

In particular if you use prioritised choice for the token rules then (for the lexing dialect used in Rust 2021 and later) you can simplify the existing reserved token rules, getting rid of the "except b or c or r or br or cr" business, and end up with something like this:

C_STRING_LITERAL ← c " (more stuff)
RESERVED_TOKEN_DOUBLE_QUOTE ← IDENTIFIER_OR_KEYWORD "
IDENTIFIER_OR_KEYWORD ← (same as at present)

This way c"\0" is rejected as a reserved form, and there's no need to bother the Reference's readers with a discussion of cuts.