feat: Add support for custom code templates (fix #987)

[travkin79]

This PR fixes issue #987 and adds

• a preferences dialog for editing custom code templates for languages available through registered TextMate grammars
• first language-independent code templates
• support for adding new custom code templates
• code proposals based on available custom code templates
• syntax highlighting in the code templates preview based an the settings for the registered language grammars

[angelozerr]

@travkin79 to be consistent with other existing class, I suggest that you use TM* instead of Tm4e*

[sebthom]

Please add some screenshots to this PR. Thanks!

[angelozerr]

@travkin79 I noticed that you have changed the Tm4e with Tm, please use TM like other classes (TMPlugin, etc)

[angelozerr]

Please add some screenshots to this PR. Thanks!

I think it should be very cool to update the docs with screenshot.

[travkin79]

Please add some screenshots to this PR. Thanks!

Hi @sebthom,
Here are some screenshots that illustrate the new features implemented so far (it's still work in progress).

New preferences dialog (very similar to the "Templates" preferences in JDT and CDT, due to reusing code from the Eclipse platform):

Users can add custom code templates for registered languages by pressing the "New" button in the preferences dialog. Here is how the dialog for new code templates looks like (again very similar to the corresponding dialog in JDT and CDT):

The context (context types) list is automatically created by checking registered grammars / languages. In the "New Template" dialog, you'll get a list of all available languages / scopes:

Pushing the button "Insert variable" in the "New Template" dialog shows available code template variables (currently only the language-independent ones):

The preview source viewer in the preferences dialog uses the syntax highlighting associated with the registered grammar corresponding to the template's context type, here Java:

In an editor with syntax highlighting activated through TM4E, users get code proposals based on existing code templates:

[angelozerr]

@travkin79 it looks super great! Congrats!

But IMHO I think you should also support vscode snippet. TM4E already supports:

• textmate
• language-configuration.json

that you can share between vscode and Eclipse IDE.

VSCode snippets is the last part which misses.

For instance instead of using cursor or selection, you should use snippet syntax

If you can support that, you could copy/paste for instance https://github.com/redhat-developer/vscode-java/blob/main/snippets/server.json and use it directly.

And it would be very nice to provide an extension point to contribute to TM4E with snippet like we have for textmate and language configuration.

If you can manage that, you could share snippets from all existing vscode extension and it will be very easy for developer to manages snippets for vscode and Eclipse IDE both.

Please note that IntelliJ supports also vscode snippets.

[travkin79]

Hello @angelozerr,

I see the benefits of VSCode syntax support. But I think, it would also demand a lot of work to add VSCode snippet support. For that reason I prefer to begin with a smaller feature set, without VSCode snippet syntax support for now. We can add that feature later with a separate PR (see my issue comment on that).

In my implementation, I tried to reuse as many code template support code from the Eclipse platform as possible and though, it was already quite some work to get that far. Now, I'd like to clarify if I'm on the right way with my concept of using

1. using custom code templates for registered TM grammars / languages
2. using common code templates for comments, ordinary comments and documentation comments (so that such code templates can be used for all languages, no need to specify copies for each language)

In addition, I'd like to finish my implementation and clarify / improve a few things in code. Most of these things can be found in TODO comments in my changed files.

• Does my algorithm in TMTemplateCompletionProcessor#retrieveTemplateContextType(TMToken) correctly detect comments and documentation comments in tokenized source code?
• TMUIPlugin got a new inner class ContextTypeRegistryWrapper for wrapping a deprecated ContextTypeRegistry. The null-safety check (the compiler) complains about incompatible return types in the overriden method contextTypes(). I didn't find a solution, yet. Any tip is appreciated. Are other null-safety annotations and conversions in my PR correctly used?
• In TMUIPlugin#getTemplateContextRegistry() I'm loading registered grammars and registering a new context type for each grammar. Since the list of grammars becomes quite long and I didn't find language names for each of them, are there any grammars that I can / should safely skip in this step? Maybe the grammars embedded in other languages like various languages from code snippets in Markdown or LaTeX?
• In TMTemplateCompletionProcessor#computeCompletionProposals(ITextViewer, int) I have to call the overridden method via Display.syncExec(). Otherwise I get an invalid thread access exception. Am I doing something wrong or might there be a bug in the parent class?
• I'm trying to apply the same syntax highlighting in the code template preview and editing source viewers. Is there an elegant way of adding special syntax highlighting for the code template variables (e.g. ${user}) without having to create a modified version of each registered TM grammar?
• Extensions
  • In case we want to support language-specific code template variables in our code templates, how can we add additional variables to the pre-defined context types (see TMLanguageTemplateContextType)? Maybe that could also be a feature for a separate PR.
  • In case we're going to use some kind of extensions, we might need to override context types by loading more specific ones from extensions or load language-specific variables (or variable resolvers) from extensions.

[angelozerr]

Please add some screenshots to this PR. Thanks!

Hi @sebthom,
Here are some screenshots that illustrate the new features implemented so far (it's still work in progress).

New preferences dialog (very similar to the "Templates" preferences in JDT and CDT, due to reusing code from the Eclipse platform):

Users can add custom code templates for registered languages by pressing the "New" button in the preferences dialog. Here is how the dialog for new code templates looks like (again very similar to the corresponding dialog in JDT and CDT):

The context (context types) list is automatically created by checking registered grammars / languages. In the "New Template" dialog, you'll get a list of all available languages / scopes:

Pushing the button "Insert variable" in the "New Template" dialog shows available code template variables (currently only the language-independent ones):

The preview source viewer in the preferences dialog uses the syntax highlighting associated with the registered grammar corresponding to the template's context type, here Java:

In an editor with syntax highlighting activated through TM4E, users get code proposals based on existing code templates:

Those screenshot and description should be included in the doc that @sebthom has written

[travkin79]

Hi @angelozerr,

I think, I'd add screenshots to the documentation as soon as the implementation is (more or less) final. Otherwise, the UI / screenshots are likely to change.

[angelozerr]

Hi @angelozerr,

I think, I'd add screenshots to the documentation as soon as the implementation is (more or less) final. Otherwise, the UI / screenshots are likely to change.

Thanks!

[travkin79]

Hi @sebthom,
Could you please take a look at this PR? Do you see things that we should definitely change? Do you have some answers to the questions in the TODO comments I added in changed files?

[sebthom]

I haven't reviewed the code yet but some feedback from trying to use it:

1. When I create a new template the name field is not visible:

Probably because the dropdown menu expands to much when it contains wide entries:

2. Could you also enable syntax highlighting in pattern editor and not only in the preview?

3. Please document this feature in docs/userguide.md

4. What is the "Use code formatter" option for? Since tm4e does not provide a formatter should it better be called something like "Use code formatter after insert (if available)"

5. How is context "Document Comment" defined? In JS it seems to be any /** */ block comment but not // style line comment.

6. are you planning to include some more generic built-in templates next to the "author" template?

[travkin79]

Hi @sebthom,

Thank you very much for your feedback. I've tried solving the issues / answering your questions.

1. When I create a new template the name field is not visible:

In my case the text field was visible, but resizing the window could result in not showing the name text field. I fixed that by setting a minimum size, but I had to use a (hacky) work-around to find the text field from the parent class, since the corresponding variable was private. Suggestions for improvements are welcome.

dropdown menu expands to much when it contains wide entries

Since I'm reusing code from the Eclipse platform, I'm not sure if I can shorten the labels in case they are too long.

Besides, it might be useful to remove some entries of the list, e.g. grammars for embedded languages. But I'm not sure which filter criteria might be reasonable.

2. Could you also enable syntax highlighting in pattern editor and not only in the preview?

I'd like to, but I didn't yet find a way to modify the source viewer configuration from the parent class org.eclipse.ui.texteditor.templates.TemplatePreferencePage.EditTemplateDialog.

3. Please document this feature in docs/userguide.md

You'll now find a first version of the new documentation in docs/user-guide.md.

4. What is the "Use code formatter" option for? Since tm4e does not provide a formatter should it better be called something like "Use code formatter after insert (if available)"

I originally thought that the custom code template support implementation from the Eclipse platform would somehow support formatting the code templates after application, but it turns out that's not the case. So I switched the "Use code formatter" checkbox off in the preferences page and removed the unused setting.

5. How is context "Document Comment" defined? In JS it seems to be any /** */ block comment but not // style line comment.

My idea was to offer two special, generic context types for custom code templates:

• Comment: an ordinary line comment or block comment (which is not a documentation comment)
• Documentation comment: a comment (usually a block comment) that according to the grammar is used for documentation, i.e. has a token type containing the strings "comment" and "documentation". In java for example, these are comments surrounded by /** and */ and I think languages like C/C++ and JavaScript use similar syntax.

These context types allow for specifying text templates that can be used in arbitrary languages, e.g. a copyright text block, a TODO comment with the user's name, a license reference or text, etc. That avoid re-specifying the same code templates for each language separately.

6. are you planning to include some more generic built-in templates next to the "author" template?

I didn't plan to add more pre-built code templates, but if you have any suggestions for more useful code templates that are not very user-specific, I'd happily add those.

[sebthom]

2. Could you also enable syntax highlighting in pattern editor and not only in the preview?

I'd like to, but I didn't yet find a way to modify the source viewer configuration from the parent class org.eclipse.ui.texteditor.templates.TemplatePreferencePage.EditTemplateDialog.

I think you could reuse the org.eclipse.tm4e.ui.internal.widgets.TMViewer source viewer in EditTemplateDialog's and TemplatePreferencePage's SourceViewer createViewer(..) methods like here

tm4e/org.eclipse.tm4e.ui/src/main/java/org/eclipse/tm4e/ui/internal/preferences/GrammarPreferencePage.java

Lines 375 to 409 in 4a927cc

	private void createThemePreview(final Composite parent) {
	final var label = new Label(parent, SWT.NONE);
	label.setText(TMUIMessages.GrammarPreferencePage_preview);
	final var data = new GridData();
	data.horizontalSpan = 2;
	label.setLayoutData(data);
	grammarPreview = new TMViewer(parent, SWT.BORDER | SWT.V_SCROLL | SWT.H_SCROLL);
	// Don't set caret to 'null' as this causes https://bugs.eclipse.org/293263
	// viewer.getTextWidget().setCaret(null);
	final var control = grammarPreview.getControl();
	control.setLayoutData(GridDataFactory.fillDefaults()
	.align(SWT.FILL, SWT.FILL)
	.grab(true, true)
	.hint(SWT.DEFAULT, convertHeightInCharsToPixels(5))
	.create());
	}
	private void preview(final IGrammarDefinition definition, final @Nullable IThemeAssociation selectedAssociation) {
	// Preview the grammar
	final IGrammar grammar = grammarManager.getGrammarForScope(definition.getScope());
	if (selectedAssociation != null) {
	setPreviewTheme(selectedAssociation.getThemeId());
	}
	grammarPreview.setGrammar(grammar);
	final ISample[] samples = TMUIPlugin.getSampleManager().getSamples(definition.getScope().getName());
	if (samples.length == 0) {
	grammarPreview.setText("");
	} else {
	// TODO: manage list of sample for the given scope.
	grammarPreview.setText(samples[0].getContent());
	}
	}

[travkin79]

2. Could you also enable syntax highlighting in pattern editor and not only in the preview?

I'd like to, but I didn't yet find a way to modify the source viewer configuration from the parent class org.eclipse.ui.texteditor.templates.TemplatePreferencePage.EditTemplateDialog.

I think you could reuse the org.eclipse.tm4e.ui.internal.widgets.TMViewer source viewer in EditTemplateDialog's and TemplatePreferencePage's SourceViewer createViewer(..) methods like here

tm4e/org.eclipse.tm4e.ui/src/main/java/org/eclipse/tm4e/ui/internal/preferences/GrammarPreferencePage.java

Lines 375 to 409 in 4a927cc

	private void createThemePreview(final Composite parent) {
	final var label = new Label(parent, SWT.NONE);
	label.setText(TMUIMessages.GrammarPreferencePage_preview);
	final var data = new GridData();
	data.horizontalSpan = 2;
	label.setLayoutData(data);
	grammarPreview = new TMViewer(parent, SWT.BORDER | SWT.V_SCROLL | SWT.H_SCROLL);
	// Don't set caret to 'null' as this causes https://bugs.eclipse.org/293263
	// viewer.getTextWidget().setCaret(null);
	final var control = grammarPreview.getControl();
	control.setLayoutData(GridDataFactory.fillDefaults()
	.align(SWT.FILL, SWT.FILL)
	.grab(true, true)
	.hint(SWT.DEFAULT, convertHeightInCharsToPixels(5))
	.create());
	}
	private void preview(final IGrammarDefinition definition, final @Nullable IThemeAssociation selectedAssociation) {
	// Preview the grammar
	final IGrammar grammar = grammarManager.getGrammarForScope(definition.getScope());
	if (selectedAssociation != null) {
	setPreviewTheme(selectedAssociation.getThemeId());
	}
	grammarPreview.setGrammar(grammar);
	final ISample[] samples = TMUIPlugin.getSampleManager().getSamples(definition.getScope().getName());
	if (samples.length == 0) {
	grammarPreview.setText("");
	} else {
	// TODO: manage list of sample for the given scope.
	grammarPreview.setText(samples[0].getContent());
	}
	}

Unfortunately, re-using the TMViewer doesn't work. I'm subclassing org.eclipse.ui.texteditor.templates.TemplatePreferencePage.EditTemplateDialog from the Eclipse platform UI in order to reuse the implementation for handling templates. That includes having to use their SourceViewer and/or SourceViewerConfiguration because of the pre-defined TemplateVariableProcessor. I thought, I could override TemplatePreferencePage.EditTemplateDialog#createViewer(Composite) and extend its behavior, but it's complicated.

Nevertheless, with a few tricks (that I am not proud of) I managed to activate syntax highlighting in the edit code template dialog. But if someone sees a better way of implementing this, I'd happily improve the code.

What is still missing is syntax highlighting for template variables. I don't know if there is a way to extend each of the registered grammars with grammar rules for template variables, only for the code templates preferences page and dialog.

Update
I found a way to highlight code template variables in the preview and in the edit template dialog:

[sebthom]

the value of languageGrammar.name is the raw value from the textmate file which may not match under which name the grammar is registered in Eclipse or may not be defined at all (e.g. for PHP). So toContextTypeName(..) should not rely on IGrammar#getName but needs to get the name from TMEclipseRegistryPlugin.getGrammarRegistryManager().getContentTypesForScope(languageScope).iterator().next().getName();

[travkin79]

Thank you for this important hint. I changed the implementation. Some questions remain open though:

• Can we somehow shorten the (unique) code template context type names? (It has to be unique since the code in TemplatePreferencePage.EditTemplateDialog uses the edit template dialog's context drop-down list entries, i.e. the context type names, to determine the selected code template context type.)
• Does it make sense to ignore some grammars, e.g. those without a grammar name / content type name?

[sebthom]

I think for now I would leave it as is. I don't know a reliable way to decide which grammars to filter out and which not.

[sebthom]

Thanks a lot for this contribution!