Groupware #1994

pbleser-oc · 2025-12-09T08:30:50Z

Description

Ongoing implementation of the Groupware backend service, which initially happened on its own branch but should now occur on the main branch.

Changes are almost exclusively on packages that are distinct to the Groupware backend, namely

pkg/jmap, pkg/jscalendar and pkg/jscontact: contain the implementation of the JMAP protocol and data models for Core, Mail, Contacts, Calendars, Blobs, ...
services/groupware: contains the Groupware backend service which is currently configured to not be started by default, and must be explicitly included in the START_ADDITIONAL_SERVICES property

Changes to common areas include:

devtools/deployments/opencloud_full: addition of a container of the Stalwart mail server which is used for Groupware functionality
pkg/structs: add more helper functions that are then used in pkg/jmap and services/groupware

Note that it is not fully functional yet and is going to be under continued and ongoing development along with its UI counterparts.

Specifically, the following aspects are lacking and only implemented in a skeletal way as a proof of concept:

API documentation: originally developed using go-swagger and redocly-cli, but which is having severe limitations and might be superseded by an ongoing implementation of a custom source code parser which renders OpenAPI
unit tests: very little coverage if at all, which is mostly due to most of the implementation of the Groupware backend doing relatively little logic and is mostly implementing the JMAP model
integration tests: some coverage exists that performs tests against a Stalwart container using testcontainers, but will be made more exhaustive
k6 black box tests: none implemented yet
system documentation: none implemented yet, too early to do so

The Groupware backend is not meant to be used productively yet.

…enLDAP container as a directory for user authentication

…ince 'admin' exists as a regular user in LDAP and thus won't have access to the administration

…80 directly to access the userinfo endpoint using HTTP since the certificates in traefik are self-signed and end up being rejected by Stalwart with no option to bypass the certificate check

…h API endpoint with it

…ernal authentication API for third party services such as Stalwart

* primitive implementation to demonstrate how it could work, still to be considered WIP at best * add new dependency: MicahParks/jwkset and MicahParks/keyfunc to retrieve the JWK set from KeyCloak to verify the signature of the JWTs sent as part of Bearer authentication in the /auth API * (minor) opencloud/.../service.go: clean up a logging statement that was introduced earlier to hunt down why the auth-api service was not being started

* refactored the models to be strongly typed with structs and mapstruct to decompose the dynamic parts of the JMAP payloads * externalized large JSON strings for tests into .json files under testdata/ * added a couple of fantasy Graph groupware APIs to explore further options * added k6 scripts to test those graph/me/messages APIs, with a setup program to set up users in LDAP, fill their IMAP inbox, activate them in Stalwart, cleaning things up, etc...

…iner repository to avoid Hub limits

…arlier implementation attempt, which also fixes compilation issues due to changes in main

* after having decided that the Groupware API should be a standalone independent custom REST API that is using JMAP data models as much as possible, * removed Groupware APIs from the Graph service * moved Groupware implementation to the Groupware service, and refactored a few things accordingly

…d error handling in jmap and services/groupware

…ctory, and started Swagger documentation implementation

…f they are missing

* jmap/EmailCreate: add more attributes that were omitted: Headers, InReplyTo, References, Sender * add jmap GetEmailSubmissionStatus * improve email integration tests by adding a thorough test for email submission * jmap integration tests: provision principals and domains using the Stalwart Management API, switching from an in-memory to an internal directory

…ion properties

…efaults and config sub-structures

butonic · 2025-12-10T13:22:16Z

services/auth-api/pkg/service/http/v0/service.go

+
+	svc, err := NewAuthenticationApi(options.Config, &options.Logger, options.Metrics, options.TraceProvider, m)
+	if err != nil {
+		panic(err) // TODO p.bleser what to do when we encounter an error in a NewService() ?


The authentication api should be passed in as an option so the creation cannot fail here.

butonic · 2025-12-10T15:52:22Z

services/groupware/DEVELOPER.md

+Run the following command on your host (requires the `ldap-tools` package with the `ldapsearch` CLI tool), which should output a list of DNs of demo users:
+
+```bash
+ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \


this only works with the url for me

Suggested change

ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \

ldapsearch -H ldap://localhost -D 'cn=admin,dc=opencloud,dc=eu' \

butonic · 2025-12-10T16:06:38Z

services/groupware/DEVELOPER.md

+curl -ks -D- -X POST \
+"https://keycloak.opencloud.test/realms/openCloud/protocol/openid-connect/token" \
+-d username=alan -d password=demo -d grant_type=password \
+-d client_id=groupware -d scope=openid


when working from top to bottom this fails. using a client_id=web works ... so I need to add a client id for the goupware somewhere ...

butonic · 2025-12-10T16:08:15Z

services/groupware/DEVELOPER.md

+to which one should receive the following response:
+
+```java
+A OK [CAPABILITY IMAP4rev2 ...] Authentication successful


hm, I need to dig into:

stalwart-1 | 2025-12-10T16:07:05Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = 172.39.0.1, remotePort = 45034, size = 19, contents = "A LOGIN alan demo\r\n" stalwart-1 | 2025-12-10T16:07:05Z ERROR LDAP error (store.ldap-error) listenerId = "imaptls", localPort = 993, remoteIp = 172.39.0.1, remotePort = 45034, reason = "I/O error: Connection refused (os error 111)", causedBy = "crates/directory/src/core/dispatch.rs:25", id = "A"

butonic · 2025-12-11T13:47:51Z

devtools/deployments/opencloud_full/.env

+STALWART_DOMAIN=
+


Suggested change

STALWART_DOMAIN=

STALWART_DOMAIN=

# LDAP config to use. Can either be idmldap (the built in IdP) or ldap (when using keycloak).

STALWART_AUTH_DIRECTORY=idmldap

butonic · 2025-12-11T13:49:18Z

services/groupware/DEVELOPER.md

+## Services
+
+### Stalwart
+


add a section about the STALWART_AUTH_DIRECTORY env var

butonic · 2025-12-11T13:52:52Z

services/groupware/pkg/groupware/groupware_api.go

@@ -0,0 +1,19 @@
+package groupware


I think we should drop the groupware_ prefix from files in this package

butonic · 2025-12-11T13:53:56Z

pkg/jmap/jmap_api.go

@@ -0,0 +1,49 @@
+package jmap


I think we should drop the jmap_ prefix from all files in this package

butonic · 2025-12-11T14:46:38Z

devtools/deployments/opencloud_full/opencloud.yml

+      AUTH_BASIC_LDAP_BIND_PASSWORD: "admin"
+      USERS_LDAP_BIND_PASSWORD: "admin"
+      GROUPS_LDAP_BIND_PASSWORD: "admin"
+      IDM_LDAPS_ADDR: 0.0.0.0:9235


I configured everything on .compose.test to prevent collisions with my .opencloud.test deployment. The stalwart URL then needs to be set for the groupware:

Suggested change

IDM_LDAPS_ADDR: 0.0.0.0:9235

IDM_LDAPS_ADDR: 0.0.0.0:9235

GROUPWARE_JMAP_BASE_URL: https://${STALWART_DOMAIN:-stalwart.opencloud.test}

oh and we need to set FRONTEND_GROUPWARE_ENABLED: "true" and enable the mail app in the web config when stalwart is enabled ... but that connot be configured with a simple env var ... 😞 we need to replace the config file then ...

butonic

Ok, I still don't have a menu entry for mail in the web UI even though the config.json contains the mail app. I assume that is a problem with my setup/config. in any case, I can see the mail UI when navigation to {$OC_URL}/mail manually. \o/

I did run into some minor issues when following the docs. If you could address them ~~(and enlighten me on how to get menu to show the groupware apps)~~ I'm happy to merge this. I mostly want others to be able follow the DEVELOPMENT.md and have a working setup. Kudos for that, btw.

The groupware service itself follows our ... boilerplate ... service code ... and implements the JMAP handling. I assume that will have to evolve, but we can merge it and iterate.

Tip

the web repo has a compose file with all the apps enabled. That gave me the final hint to get the menu entries

pbleser-oc added 30 commits December 9, 2025 09:15

Add Stalwart container to the opencloud_full deployment, using the Op…

0435d56

…enLDAP container as a directory for user authentication

WIP: initial implementation of the groupware service

d638fba

add missing routing for /groupware (currently unprotected for testing)

72af257

rename Stalwart fallback admin username from 'admin' to 'mailadmin' s…

6620313

…ince 'admin' exists as a regular user in LDAP and thus won't have access to the administration

add an OIDC Directory to Stalwart, requires exposing Keycloak port 80…

6da208e

…80 directly to access the userinfo endpoint using HTTP since the certificates in traefik are self-signed and end up being rejected by Stalwart with no option to bypass the certificate check

WIP: restructure the Jmap client, and implement the /me/messages Grap…

da9ed5f

…h API endpoint with it

move services/groupware/pkg/jmap to pkg/jmap

4cf4d44

add an auth-api service to make an exemplary implementation of an ext…

4e6053c

…ernal authentication API for third party services such as Stalwart

minor corrections to the Stalwart configuration

02f33bd

upgrade Stalwart to 0.12

a6cdb4e

groupware and jmap improvements and refactoring

71cddaa

auth-api: fix: was missing newly introduced metrics

b3766ab

jwkset: remove debugging printlns

0f47e3a

groupware: removed debugging logs

22e51bd

upgrade Stalwart to 0.12.4

e0ea733

Use password policy overlay in LDAP and configure Stalwart to use it

772a902

fix Stalwart LDAP configuration

5920291

opencloud_full: upgrade Stalwart to 0.12.5, and use the ghcr.io conta…

2412e64

…iner repository to avoid Hub limits

groupware: remove unneeded messages.go that was a remainder from an e…

8df4ef6

…arlier implementation attempt, which also fixes compilation issues due to changes in main

Groupware and jmap: cleanup and API documentation

d544efd

groupware: implement JSON:API's error response format, with a revampe…

2191b1d

…d error handling in jmap and services/groupware

groupware: refactoring the API mechanisms

8be4d11

refactored the Session object, refactored the services/groupware dire…

94c932d

…ctory, and started Swagger documentation implementation

upgrade Stalwart to 0.13.2

1b9249e

groupware: further implementation and improvements

0d2a5e9

groupware: more JMAP operations implementation

ac8d258

groupware: blob streaming (upload and download)

6e4918b

pbleser-oc added 7 commits December 9, 2025 09:15

groupware: upgrade the Stalwart image in devtools from 0.14.0 to 0.14.1

dfda4b3

groupware, auth-api: remove tracing and tracing configuration

d0effed

groupware: fix missing casting to jmap.State after changes in pkg/jmap

8ae4694

groupware: minor: remove network declaration in stalwart.yml

e7d557c

groupware: refactor response objects to take a list of accountIds

25068ad

groupware: add retrieving and adding mailboxIds for drafts and sent i…

42cce92

…f they are missing

pbleser-oc self-assigned this Dec 9, 2025

pbleser-oc added Status:Needs-Review Type:Organizational Type:Maintenance E.g. technical debt, packaging, etc. and removed Type:Organizational labels Dec 9, 2025

pbleser-oc added 4 commits December 9, 2025 09:55

groupware: add description and version annotations for env configurat…

6ff5b1a

…ion properties

groupware: fix failing pkg/jscalendar unit tests

3da6855

groupware: fix failing pkg/jscontact unit tests

dc7df40

groupware: shift some attributes of the Groupware object around, in d…

790a241

…efaults and config sub-structures

pbleser-oc requested a review from butonic December 10, 2025 08:32

pbleser-oc marked this pull request as ready for review December 10, 2025 08:32

butonic reviewed Dec 10, 2025

View reviewed changes

butonic reviewed Dec 11, 2025

View reviewed changes

services/groupware/DEVELOPER.md

## Services

### Stalwart

Copy link

Contributor

butonic Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a section about the STALWART_AUTH_DIRECTORY env var

butonic reviewed Dec 11, 2025

View reviewed changes

pkg/jmap/jmap_api.go

@@ -0,0 +1,49 @@

package jmap

Copy link

Contributor

butonic Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should drop the jmap_ prefix from all files in this package

butonic reviewed Dec 11, 2025

View reviewed changes

butonic requested changes Dec 11, 2025

View reviewed changes

groupware: update to Stalwart 0.15.0

3d8cad1

pbleser-oc force-pushed the groupware branch from a9ef8f3 to 3d8cad1 Compare December 17, 2025 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Groupware #1994

Groupware #1994

Uh oh!

pbleser-oc commented Dec 9, 2025 •

edited

Loading

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025 •

edited

Loading

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \
	ldapsearch -H ldap://localhost -D 'cn=admin,dc=opencloud,dc=eu' \

	IDM_LDAPS_ADDR: 0.0.0.0:9235
	IDM_LDAPS_ADDR: 0.0.0.0:9235
	GROUPWARE_JMAP_BASE_URL: https://${STALWART_DOMAIN:-stalwart.opencloud.test}

Groupware #1994

Are you sure you want to change the base?

Groupware #1994

Uh oh!

Conversation

pbleser-oc commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pbleser-oc commented Dec 9, 2025 •

edited

Loading

butonic Dec 11, 2025 •

edited

Loading

butonic left a comment •

edited

Loading