Update to ANTLR Runtime 3.5.3

[HannesWell]

A naive attempt to resolve #3352.

Most changes are adaptions to the removal of the Token.EOF_TOKEN constant:
antlr/antlr3@484368a

It's replaced by a corresponding check for the token's type. Alternatively, null could be returned in case EOF is reached, but I assumed that's not the preferred solution. The advantage of the current approach is, that is compatible for new and old versions of Xtext, if there are any downstream consumers referencing the adapted code.

[github-actions]

Test Results

 22 files   -   8 054   22 suites   - 8 054   7m 2s ⏱️ - 3h 44m 32s
118 tests  -  43 123  116 ✅  -  42 541  0 💤  -   584  2 ❌ +2 
122 runs   - 212 623  120 ✅  - 209 704  0 💤  - 2 921  2 ❌ +2 

For more details on these failures, see this check.

Results for commit 9d1a856. ± Comparison against base commit 5bee596.

This pull request removes 43123 tests.

com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] entrySet [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] keys [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] values [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: several] entrySet [collection size: several]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: several] keys [collection size: several]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedAllPresent[Guava-based UnmodifiableMergingMapView tests [collection size: several] values [collection size: several]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedNonePresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] entrySet [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedNonePresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] keys [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedNonePresent[Guava-based UnmodifiableMergingMapView tests [collection size: one] values [collection size: one]]
com.google.common.collect.testing.testers.CollectionAddAllTester ‑ testAddAll_unsupportedNonePresent[Guava-based UnmodifiableMergingMapView tests [collection size: several] entrySet [collection size: several]]
…

♻️ This comment has been updated with latest results.

[cdietrich]

I also wonder what happens if you run the workflows for all (test) languages

https://github.com/eclipse-xtext/xtext/blob/main/dev-doc/RegenerateLanguages.md

as we have code / workflow fragments that patch around in the generated code using regexes

Am also not sure if we have a generate all composite launch config meanwhile

[HannesWell]

I also wonder what happens if you run the workflows for all (test) languages

After I've fixed the first round of failures because BufferedTokenStream.fill() now also adds the EOF token to the list of tokens. Alternatively we could make sure that the EOF token has the indices properly set all the time.

But currently this still fails with

[ERROR]: GeneratorException: (Element: -UNKNOWN-; Reported by: XtextGenerator)
	 org.eclipse.emf.common.util.WrappedException: java.lang.reflect.InvocationTargetException
	at org.eclipse.xtext.xtext.generator.parser.antlr.AntlrToolFacade.runWithEncodingAndParams(AntlrToolFacade.java:187)
	at org.eclipse.xtext.xtext.generator.parser.antlr.XtextAntlrGeneratorFragment2.runAntlr(XtextAntlrGeneratorFragment2.java:243)
	at org.eclipse.xtext.xtext.generator.parser.antlr.XtextAntlrGeneratorFragment2.generateProductionGrammar(XtextAntlrGeneratorFragment2.java:183)
	at org.eclipse.xtext.xtext.generator.parser.antlr.XtextAntlrGeneratorFragment2.doGenerate(XtextAntlrGeneratorFragment2.java:147)
	at org.eclipse.xtext.xtext.generator.parser.antlr.AbstractAntlrGeneratorFragment2.generate(AbstractAntlrGeneratorFragment2.java:104)
	at org.eclipse.xtext.xtext.generator.CompositeGeneratorFragment2.generate(CompositeGeneratorFragment2.java:43)
	at org.eclipse.xtext.xtext.generator.XtextGenerator.invokeInternal(XtextGenerator.java:228)
	at org.eclipse.emf.mwe.core.lib.AbstractWorkflowComponent.invoke(AbstractWorkflowComponent.java:133)
	at org.eclipse.emf.mwe.core.lib.Mwe2Bridge.invoke(Mwe2Bridge.java:35)
	at org.eclipse.emf.mwe.core.lib.AbstractWorkflowComponent.invoke(AbstractWorkflowComponent.java:213)
	at org.eclipse.emf.mwe2.runtime.workflow.AbstractCompositeWorkflowComponent.invoke(AbstractCompositeWorkflowComponent.java:38)
	at org.eclipse.emf.mwe2.runtime.workflow.Workflow.run(Workflow.java:21)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:103)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:63)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:53)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher.run(Mwe2Launcher.java:78)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher.main(Mwe2Launcher.java:36)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.eclipse.xtext.xtext.generator.parser.antlr.AntlrToolFacade.runWithEncodingAndParams(AntlrToolFacade.java:185)
	... 16 more
Caused by: java.lang.NoSuchFieldError: Class org.antlr.runtime.Token does not have member field 'org.antlr.runtime.Token EOF_TOKEN'
	at org.antlr.grammar.v3.ActionAnalysis.nextToken(ActionAnalysis.java:67)
	at org.antlr.grammar.v3.ActionAnalysis.analyze(ActionAnalysis.java:46)
	at org.antlr.tool.Grammar.examineAllExecutableActions(Grammar.java:1842)
	at org.antlr.tool.Grammar.checkNameSpaceAndActions(Grammar.java:712)
	at org.antlr.tool.CompositeGrammar.defineGrammarSymbols(CompositeGrammar.java:356)
	at org.antlr.Tool.process(Tool.java:451)
	at de.itemis.xtext.antlr.toolrunner.AntlrToolRunner.runWithEncodingAndParams(AntlrToolRunner.java:168)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	... 18 more

I'll investigate it further.

[szarnekow]

@HannesWell I wonder if this is the best way to spend your time.

The content assist parser relies heavily on the internal state of the Antlr parser and by transitivity on the generated code patterns. What do you think will be the benefit of that effort? It's not that Antlr 3.2 has any known CVEs, has it?

[HannesWell]

I wonder if this is the best way to spend your time.

Probably not.
Still I'd suggest to have the changes that avoid the references to the now removed Token.EOF_TOKEN.
I can create a separate PR with just that, without changing the required antlr.runtime version.

The content assist parser relies heavily on the internal state of the Antlr parser and by transitivity on the generated code patterns.

Yes, but I was hoping that we are lucky and the internals didn't change (much) with this respect in that increment. If you have analyzed and state that already, I've missed that.

Caused by: java.lang.NoSuchFieldError: Class org.antlr.runtime.Token does not have member field 'org.antlr.runtime.Token EOF_TOKEN'

Looking at that error it seems like the class org.antlr.grammar.v3.ActionAnalysis has to be adjusted too to not reference the EOF_TOKEN. But I'm not sure where this the bundle org.antlr.generator is actually coming from.
The name indicates antlr, but I didn't found it in https://github.com/antlr/antlr3.
But I found https://mvnrepository.com/artifact/org.xtext/antlr-generator/3.2.1.
Out of curiosity, I wonder if this bundle is actually an Xtext project? But then I wonder where it's sources are?

What do you think will be the benefit of that effort? It's not that Antlr 3.2 has any known CVEs, has it?

I'm not aware of any, but I also haven’t checked it. But in general I think it's got to be able to be able to follow the latest available, even if it's not yet urgent. One day the latest version might be needed urgently for whatever reason and then it's of course simple if the increment is smaller.
But of course we can hope that this day will never come and the currently used version can be used forever.

[HannesWell]

I'd suggest to have the changes that avoid the references to the now removed Token.EOF_TOKEN.
I can create a separate PR with just that, without changing the required antlr.runtime version.

@cdietrich, @szarnekow, should I do that or just close this PR without extracting anything?

[cdietrich]

i propose to discuss in one of the next calls