[GTK, Wayland] Shell can be resized smaller than set minimum size

[tmssngr]

Describe the bug
When setting the minimum size of a shell, it is expected that the user can't make it smaller than this by resizing the window. This works as expected on Windows, macOS and Linux with X11, but not on Linux with Wayland.

To Reproduce

import org.eclipse.swt.*;
import org.eclipse.swt.graphics.*;
import org.eclipse.swt.widgets.*;

public class MinimumSizeShell {

	public static void main(String[] args) {
		final Display display = new Display();

		final Shell shell = new Shell(display);
		final Composite composite = new Composite(shell, 0);
		composite.addListener(SWT.Paint, event -> {
			final Point size = composite.getSize();
			event.gc.setForeground(new Color(255, 0, 0));
			event.gc.setLineWidth(5);
			event.gc.drawRectangle(0, 0, size.x - 1, size.y - 1);
		});
		shell.setLayout(new Layout() {
			@Override
			protected Point computeSize(Composite unused, int wHint, int hHint, boolean flushCache) {
				return new Point(400, 300);
			}

			@Override
			protected void layout(Composite unused, boolean flushCache) {
				composite.setSize(400, 300);
			}
		});

		final Point size = shell.computeSize(SWT.DEFAULT, SWT.DEFAULT);
		shell.setSize(size);
		shell.setMinimumSize(size);
		shell.open();

		while (!shell.isDisposed()) {
			if (!display.readAndDispatch()) {
				display.sleep();
			}
		}

		display.dispose();
	}
}

• launch and try to make the shell smaller

Expected behavior
It must not be possible to make the shell smaller than the set minimum size.

Screenshots
Initially:

After making smaller:

Environment:

1. Select the platform(s) on which the behavior is seen:

• • All OS
• • Windows
• • Linux
• • macOS

2. Additional OS info (e.g. OS version, Linux Desktop, etc)
   Tried on Ubuntu 26.04 and Fedora Silverblue 43.

[Phillipus]

As with the other issue I tested this on Ubuntu Wayland and I can't reproduce this. I wonder if anyone else has access to Ubuntu Wayland to test it?

[tmssngr]

I've taken the screenshots with the Window option. It looks like the "shadow area" around the visible shell belongs to the shell for Wayland - the size of the small screenshot seems to be the same size as the client area of the visible shell in the large screenshot. Hence, I reckon, the minimum size should be increased inside SWT by the "shadow area" for GTK.

[wnesensohn]

Also seeing this, same configuration as in the other issue, that is, Fedora 43 with Gnome 49.

Wayland debugging output shows that the minimum size is off significantly: {Default Queue} -> xdg_toplevel#42.set_min_size(348, 248)

The adjustments, on my machine, to get correct minimum size behavior are 52px in width, and 97px in height if GTK3 is used, or 0px in width and 45px in height for GTK4 (enabled by SWT_GTK4=1). The height discrepancy looks like it's exactly matching the height of the title bar.

So it looks like two similar issues, one which is resolved using GTK4, and the other (not accounting for the title bar) present with both GTK3 and GTK4.