Flex segfaults after reading EOF in `input()`

[nxg]

The program below works as expected when reading from stdin, but segfaults when it is instead lexing a buffer.

The key thing about this example is that one of the rules uses input() to gobble from "!" to EOF (yes, it looks as if I could use a "!".* pattern, but that doesn't produce the intended results in the real case; the lexer needs to balance braces, and if I hit EOF when trying to do that, I want to recover gracefully).

When run, reading from stdin, I get

$ flex -o eof.c eof.lex
$ cc -o eof eof.c
$ echo -n 'one two!three four' | ./eof
word:<one>
-> 1
-> 2
word:<two>
-> 1
buf=<three four>
-> 3

That's fine, but when I instead ./eof 'one two !three four', which scans the contents of a buffer set up by yy_scan_string, I get identical program output, followed by a segfault inside yy_get_next_buffer.

I can't work out which part of the flex manual is telling me I should expect that to happen.

The sequence of events seems to be that the lexer is finding its way to the end of file, as expected (and an <<EOF>> action confirms this), but not stopping there, despite the presence of the noyywrap option, and collapsing when it can't find a ‘next’ buffer.

Points:

• Option -d doesn't illuminate.
• It is, of course, a little hard to follow what the generated code is doing, but looking at the location of the segfault, it is indeed around the place where the code is checking for yywrap, so it should be getting the message that there is no more input coming.
• The only real illustration of using input(), in the flex manual, is in a case where hitting EOF is reported as an error. Here, I'm doing essentially the same as in that example, but regarding EOF as an acceptable end of the scan.
• The same behaviour appears when using a reentrant scanner.
• It's worth noting that input() returns 0, not EOF, at EOF, despite what Sect.8 illustrates (cf. flex repo issue, and links there), and despite the rather mysterious note about a ‘“real” end-of-file’ in Sect.20. I have a suspicion that this remark in Sect.20 is telling me something terribly important, but I can't work out what.
• This is with flex 2.6.4 and clang 15 on macOS, and 2.6.4 and gcc on (a RHEL-derived) Linux (I can confirm the precise gcc version if that would be helpful, but this doesn't look obviously compiler dependent)..

Program:

ALPHABETIC  [a-zA-Z]
WS      [^a-zA-Z!]

%option noyywrap nounput

%%

{ALPHABETIC}+   {
    printf("word:<%s>\n", yytext);
    return 1;
}
{WS}+   {
    return 2;
}

"!"         {  // gobble to end of input
    char buf[80];
    for (int idx=0; (buf[idx] = input()); idx++) /* empty */ ;
    printf("buf=<%s>\n", buf);
    // YY_FLUSH_BUFFER; /* makes no difference */
    return 3;
}

%%
int main(int argc, char** argv)
{
    switch (argc) {
      case 1: break;
      case 2:
        yy_scan_string(argv[1]);
        break;
      default:
        fprintf(stderr, "Usage: %s [string]\n", argv[0]);
        exit(1);
    }

    int token;
    while ((token = yylex()) != 0) {
        printf("-> %d\n", token);
    }
}

[Mightyjo]

I can't find a spot in the docs that explains this behavior clearly. The best hints I could find are in the sections on multiple buffers, yywrap, and EOF rules.

You need an <> rule that calls yyterminate or sets up the next buffer. That rule will take the place of yywrap in your use case.

I'm away from my computer but I'll post an example when I'm back.

[nxg]

Thanks for clarifying.

In case it's useful when thinking about the docs, my mental model, when writing what I did, was that when I arrive at EOF using input(), I'm doing so ‘legitimately’ (ie, as opposed to my being illegitimately creative with yyinput, or something like that). It was on that basis that I presumed yywrap would Do The Right Thing, and that when flex subsequently asked for more input from input(), it would be told calmly ‘no’.

Or, put another way, my mental model is that flex is itself using input() to get input, or something equivalent to that, so that I'm working in concert with it if I read from it separately.

If those are bad intuitions, it might be useful for the docs to disabuse the reader fairly explicitly.

[nxg]

Thanks for the suggestion. I tried adding that rule:

<<EOF>> {
    fprintf(stderr, "EOF!\n");
    yyterminate();
}

But this produces the same result, apparently without the <<EOF>> rule being matched.

Adding unput(0) in the "!" rule doesn't make a difference, either. Side remark: this isn't surprising , since the docs do say ‘you cannot put back ‘EOF’ to attempt to mark the input stream with an end-of-file’, but in context that could be read as saying simply that you can't ‘fake’ an EOF by putting this back ahead the actual EOF.

It's taken a while to get back to this – sorry for the silence. I think I gave up on this last year, and worked around it somehow (possibly with a by-hand lexer, possibly by taking a different approach with lexer patterns), but I can't reconstruct from the relevant repo what I actually did! Then I ran into it again with a different lexer, thought ‘I've been here before’, and...

[nxg]

Just for info/future-googlebait: the workaround in my real case was not to use input()! I was able to get the same effect by matching the entire thing to be examined in a rule, and working through that. That may or may not be better lex style, but it can feel more natural to step through the input with input(). Onward and upward; share and enjoy.

(There does still appear to be something amiss with input(), though...)