SWT and Project Panama aka native interconnect

[laeubi]

The upcoming Project Panama aims at a close integration of java and native code as an alternative for hand-crafted JNI code.

One very interesting part is the jextract tool (you can download a pre build jdk-19-ea with jextract included here. This is based on the JEP 412: Foreign Function & Memory API already part of Java-17.

Even though I don't know if Java 17 is yet an option (@akurtakov ?) and java 19 even a bit far away I'd like to bring the topic to SWT right now so we can investigate and maybe improve and give feedback (e.g. can all relevant SWT native APIs be parsed by jextract?), so I can only encourage every "native swt devloper" to give it a try right now, so hopefully in a glory future we no longer need to build native SWT libraries and can use a pure java interface what will greatly improve the circle of people that could work in that area.

[akurtakov]

Simrel will require Java 17 in 2022-09 release and all EPPs are already shipping with Java 17 JVM so from end user POV this shouldn't be an issue. Some layered products may raise concerns.
For SWT it will be big rewrite to change that so we will probably start with some smaller parts - e.g. webkitgtk on gtk 4.x is not yet ready and porting to it is smth that hopefully we align well - aka swt itself could be fine with Java 11 but browser integration requires Java XX . There are other parts of Eclipse like network proxy and security storage that are done using JNA now and thus would be easier to switch initially. Furthermore they are higher up in the stack and will not enforce everything to move to Java 17 immediately.

[HannesWell]

Java-22 is released today with the Foreign Function & Memory (FFM) API as a final feature through JEP 454.
Compared to the its third preview via JEP 442 in Java-21 only a few changes happened so with some luck the code generated by jextract for Java-22 can also be used on a JDK-21 only with --enable-preview as VM argument.
With Java-21 being the required Java version for the Eclipse SDK and SWT probably in the not too distant future I think this could be considered again.

Using the FFM API would replace the usage of native libaries in SWT and would have several major advantages:

• Java developers would not have to deal with C code anymore, only with pure Java code
• One fragment per OS would be enough since we could save the architecture specific fragments
  • We could even consider that the code for all platforms could go into one bundle, but the generated classes can also be numerous and large so this could become quite big. But I think that's something that can and should be discussed later once the migration is completed and we know exact numbers.
  • No architecture specific content also means that SWT can run on an architecture as long as somebody provides a JDK for it and a user can obtain it. So new architectures (if they arise some day) or exotic ones would be supported without changes.
• No need to build C code anymore and thus a much simpler setup to build SWT from scratch in the CI and locally
  • Probably we would commit the sources generated by jextract into git and just check in the CI if the generated sources are up-to (but that's a detail to be discussed later).

Migrating the entire SWT native code to the FFM API will be a major task, but it can be done gradually. We can just replace native call by native call until all JNI native calls are replaced by FFM calls and no native libraries are necessary anymore.

I assume the most controversial point is if it is ok to require the --enable-preview VM argument to run SWT? For the EPP packages we can add it easily by our own but of course custom products have to be adapted as well. But maybe this is a good opportunity to improve the tooling to simplify the addition of the --enable-preview flag.
Of course the same discussion can be conducted for other parts of the Eclipse platform that use JNI/JNA at the moment too or we have a combined discussion for that.

[laeubi]

I also think one should just check in the generated code, there is no need to generate it over and over again as jextract always requires a special JVM and the native headers and often one wants to adjust something here and there.

And yes one can simply remove the native keyword and redirect the call to the FFM equivalent. I even tested to use the same binding for windows and linux in one example and it worked like a charm, so its not really a question of windows/linux/... but more that the library is the same, e.g. once in the past there was even a swt.win.gtk, now you can use the same code for win/linux as it is all java code, its just a matter of some if / else chains even if some calls diverge.

[akurtakov]

I strongly believe we should start on smaller scale (e.g. https://github.com/eclipse-equinox/equinox/tree/master/bundles/org.eclipse.equinox.security.linux or https://github.com/eclipse-platform/eclipse.platform/tree/master/team/bundles/org.eclipse.core.net.linux) with JNA based bundles so we can get the infrastructure proper (all build JVMs and etc. are still at version 17 and some tools might even fail with Java 21 (it happened during the 11->17 )) and even get more familiar with Panama ourselves.

[laeubi]

We can even start smaller:

• Make Java 21 the primary JVM to execute the maven build eclipse.platform.releng.aggregator#1919

:-)

[tmssngr]

This basically sounds interesting, but I want to emphasize one issue: currently, it is possible to compile a Java/SWT applications to a native executable using GraalVM Native Image. I do not know whether the FFI of the newer Java versions already can be compiled, so I only would consider to change SWT to use the FFI, if it also could be compiled to native executables.

…

-- Thomas Singer

On 2024-03-19 23:31, Hannes Wellmann wrote: Java-22 is released today with the Foreign Function & Memory (FFM) API as a final feature through [JEP 454](https://openjdk.org/jeps/454). Compared to the its third preview via [JEP 442](https://openjdk.org/jeps/442) in Java-21 only a few changes happened so with some luck the code generated by [jextract](https://github.com/openjdk/jextract?tab=readme-ov-file#getting-jextract) for Java-22 can also be used on a JDK-21 only with `--enable-preview` as VM argument. With Java-21 being the required Java version for the Eclipse SDK and SWT probably in the not too distant future I think this could be considered again. Using the FFM API would replace the usage of native libaries in SWT and would have several major advantages: - Java developers would not have to deal with C code anymore, only with pure Java code - One fragment per OS would be enough since we could save the architecture specific fragments - We could even consider that the code for all platforms could go into one bundle, but the generated classes can also be numerous and large so this could become quite big. But I think that's something that can and should be discussed later once the migration is completed and we know exact numbers. - No architecture specific content also means that SWT can run on an architecture as long as somebody provides a JDK for it and a user get it. So new architectures (if they arise some day) or exotic ones would be supported without changes. - No need to build C code anymore and thus a much simpler setup to build SWT from scratch in the CI and locally - Probably we would commit the sources generated by `jextract` into git and just check in the CI if the generated sources are up-to (but that's a detail to be discussed later). Migrating the entire SWT native code to the FFM API will be a major task, but it can be done gradually. We can just replace native call by native call until all JNI native calls are replaced by FFM calls and no native libraries are necessary anymore. I assume the most controversial point is if it is ok to require the `--enable-preview` VM argument to run SWT? For the EPP packages we can add it easily by our own but of course custom products have to be adapted as well. But maybe this is a good opportunity to improve the tooling to simplify the addition of the `--enable-preview` flag. Of course the same discussion can be conducted for other parts of the Eclipse platform that use JNI/JNA at the moment too or we have a combined discussion for that.

[laeubi]

That's more a question for the Graalvm team than something SWT/Platform can solve.

[tmssngr]

Hi Christoph, The first step in the direction of FFI would render SWT useless for me.

…

-- Best regards, Thomas Singer

On 2024-03-20 07:33, Christoph Läubrich wrote: That's more a question for the Graalvm team than something SWT/Platform can solve.

[laeubi]

You can still use older releases of SWT without a problem but FFM is the same as any new Java version / Dependency Upgrade / ... SWT can simply not wait for others to wait as no one is waiting for SWT as well.

So if GraalVM is important for you I suggest to look if there are already discussions and try to bring focus on the topic of Java 21/22 support to it as it will become used sooner or later.

[tmssngr]

Please reconsider your decision as our company has supported SWT in the last years by paying Alexandr Miloslavski.

…

-- Best regards, Thomas Singer

On 2024-03-20 08:31, Christoph Läubrich wrote: You can still use older releases of SWT without a problem but FFM is the same as any new Java version / Dependency Upgrade / ... SWT can simply not wait for others to wait as no one is waiting for SWT as well. So if GraalVM is important for you I suggest to look if there are already discussions and try to bring focus on the topic of Java 21/22 support to it as it will become used sooner or later.

[akurtakov]

I don't see any decision here nor any code change/PR nor a concrete issue.

It's an idea and a good one in terms of reducing maintenance cost and make it easier for more people to contribute to SWT.

If/when/how/what happens is totally up in the air as there is a very long path to be walked before there is even proof of concept, pros vs cons evaluated and so on.
With the above said, we can not shoot an idea that might give fruit in the future because there are problems with it in the moment.

[guw]

@tmssngr The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

[laeubi]

@tmssngr The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

Thanks for giving an update here a possible mitigation then would be for example to only use down calls for the time being (what should be the was majority).

[HannesWell]

The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

Yes it was also mentioned on the Panama-Mailing-List, but I guess/hope that the support in the Graal-VM will improve in the future and since this isn't something that will be applied in the near future of SWT Graal has hopefull catched up until then.

[mickaelistria]

If the idea of using FFI is consensual enough is supported by a clear majority part of the active contributors, then the idea will be implemented anyway.
But SWT can probably start this effort by dealing with 2 branches, or 2 sets of artifacts in the master branch: 1 with FFI (becoming reference/master at some point), and 1 with JNI. The work about FFI could even start right now in a dedicated branch in SWT repo (or again even in the master branch as long as it's isolated enough).
If such an approach for "concurrent" development is chosen, the tricky part is to decide about how to deal with divergence; for that, separated branches might actually be simpler as it's often easy to just rebase.

[iloveeclipse]

I wonder if we can run this SWT experiment with GTK4 port by providing it as a separated fragment, which shouldn't affect anyone as the fragment would not force everyone else to use Java 22+.

In general, a good target for getting rid of native libraries would be next LTS release of Java.

• launcher
• file system access
• SWT + browser

Would be something for the IDE working group to plan for.

[vogella]

Are there plans to start this work? With Java 25 LTS this should be possible.

[vogella]

@HeikoKlare when in Java 25 planned / possible for platform?

[HeikoKlare]

Earliest once it's allowed for SimRel:

• Ask SimRel project not to bump BREEs to Java 25 eclipse-simrel/.github#75 (comment)

[HannesWell]

I just know that @HannesWell had a look (or wanted to have a look) into FFM adoption.

Yes I generally do, since I expect (or at least hope) for significant simplifications in the RelEng process after a complete migration away from JNI.

But as already noted, I also think we should do it incrementally.

I already started locally to migrate the Win32Handler from JNA (which also has its difficulties) to FFM.
My goals is not only to do the migration but also built a toolchain (around jextract) to simplify the generation of FFM bindings further and to streamline the generated binding. The defaults from jextract can be quite verbose. This should then also simplify to regenerate, updated and maintain the bindings.
The adaption of the calling code is not that difficult on a small scale. Preparing the toolchain in advance should then help to ease the migration on larger scales like in SWT.

After the Win32Handler, I planned to migrate the remaining code in the Equinox launcher to FFM with the goal/hope to get rid of the native equinox.launcher fragments. I assume it's much simpler after the code for the splash is not necessary anymore, e.g. via

• Show splash-screen using standalone SWT in Equinox launcher eclipse-equinox/equinox#1279

IMO then makes sense to look at the smaller parts of the Eclipse codebase that use either JNI or JNA, as suggested in #175 (reply in thread).
Since I'm mainly on Windows, I would then appreciate help from somebody on Linux and macOS, though one can probably also generate the bindings for other OS with just using GH workflows.

SWT would then probably be the last step, but once completed, the one with the largest gain (but also the most work).

[HannesWell]

Over the last year, I've already started the FFM migration for GTK on a different project, but on a much smaller scale. But even that was a real pain.

Did you use jextract for it? And could you elaborate on your pain-points in more detail?

[ptziegler]

Did you use jextract for it? And could you elaborate on your pain-points in more detail?

No. To get a feeling on how this generally works, most was written by hand. It might've made sense for the native Cairo calls, but I'm not sure how well it would've worked for the GTK3/GTK4 header files, because they contain a lot of overlap.

Regarding the pain points:

• The FFM method calls all throw a Throwable. Which is quite a bother when migrating a code base that previously expected Exceptions. Which means that all calls are now within an embedded call that converts everything into a RuntimeException.

• You no longer recompile your own binary which you then reference via System.loadLibrary(...). Instead you reference the binaries provided by the OS directly. Which means you now have to know the exact name that it'll have on every distribution. For example libcairo.so.2 or libgtk-3.so.0. You also have to be careful about aliases. On my system, I can use libgtk-3.so instead of libgtk-3.so.0, but this doesn't work on the GitHub build nodes.

• I never managed to get the multi-release to work properly inside the IDE. Even when using a Java 25, the "old" code is executed. So for testing, I always have to update the project classpath to only contain the FFM source folder.

For the next release cycle I wanted to try out jextract for the Windows DLLs though.

[vogella]

@HannesWell I'm happy to help with Linux if you lead with Windows. The slow Windows file system usually kills my desire to work on Eclipse under windows. For reference, my aggregator SDK build on Linux runs with -T6 in ~ 4 min on my machine and IIRC ~ 30 min on Windows.

I don't think Equinox is a good starting point, Equinox is usually the last to move its BREE, IIRC parts of it are still on Java 8 or Java 11. Maybe that is different with the launcher, just cc me into the tickets or PRs for Equinox and I can start helping.

[merks]

If you talk to @tjwatson, I believe its a small core that is of concern, i.e., these

I think he'd rather the launcher not even exist in equinox. 😁

So I do think the equinox launcher is reasonable small place to start...

[tjwatson]

I don't think Equinox is a good starting point, Equinox is usually the last to move its BREE, IIRC parts of it are still on Java 8 or Java 11. Maybe that is different with the launcher, just cc me into the tickets or PRs for Equinox and I can start helping.

I've mentioned this other places many times. Only 5 core Equinox bundles need to stay on Java 8 for me. That is:

1. org.eclispe.osgi
2. org.eclipse.equinox.metatype
3. org.eclipse.equinox.region
4. org.eclipse.equinox.console
5. org.eclipse.equinox.coordinator (EDIT: forgot this one)

Ed is correct, I would have preferred the eclipse launcher not even be in Equinox. I see very little value in the launcher being used for anything other than RCP/SWT UI based applications.

[HannesWell]

I don't think Equinox is a good starting point, Equinox is usually the last to move its BREE, IIRC parts of it are still on Java 8 or Java 11. Maybe that is different with the launcher, just cc me into the tickets or PRs for Equinox and I can start helping.

Besides what has been said with respect to the EE requirements of different parts of Equinox, just the Equinox launcher is actually the least sensitive part when it comes to early adoption of new JDK requirements. An increased EE requirements of the launcher doesn't imply the same change to any downstream bundle (because there are non), just on the overall products being launched.
So as long as anything else in the product uses a newer Java version, the launcher wouldn't be the driver anymore.