RAS-1867 Re-implement My Messages to make query more performant by LJBabbage · Pull Request #472 · ONSdigital/ras-secure-message

LJBabbage · 2026-05-18T14:31:01Z

What and why?

This PR improves performance for internal thread retrieval (My Messages), removes unused functionality, rationalises the query logic, and adds coverage for previously untested paths.

Performance

Before

When the My Messages tab was selected, we queried the secure_message table first and performed operations such as determining the latest message before joining to the status table, without index.

This resulted in unnecessary scanning and processing of large datasets before pagination was applied, causing poor performance for users with a high message volume.

Now

The query flow has been reversed for internal users using my_messages:

We query the status table first using actor-based filters
We then join to secure_message using the indexed msg_id
Latest message logic is then applied
LIMIT and OFFSET are used to paginate efficiently at the database level

Additional changes

Consolidated the 4 internal thread SQL queries into a single query builder to avoid future misalignment between count and result queries
Retained the existing pattern of:
- querying total count first
- then retrieving the paginated results
Left external thread retrieval unchanged to minimise risk in this PR (though it could be optimised similarly in future)
Removed redundant pagination code
Removed unused search criteria and associated tests
Extended test coverage for previously untested logic paths

How to test

Deploy this branch and test a variety of thread and filtering scenarios. Suggested checks:

Create messages across 2 different surveys and confirm they appear in the correct inbox
Verify that:

a new respondent message appears in the inbox
the thread is shown as unread/bold
once replied to, it moves into My Messages

Verify messages sent initially via reporting units appear in My Messages
Confirm business reference searching still works
Confirm closed message filtering still works
Verify pagination behaviour
Regression test any additional thread filtering combinations

Jira

LJBabbage · 2026-05-18T14:31:40Z


    for message in paginated_list.items:
        msg = message.serialize(user, body_summary=body_summary)
-        msg["_links"] = {"self": {"href": f"{host_url}{MESSAGE_BY_ID_ENDPOINT}/{msg['msg_id']}"}}


This isn't used so removed

LJBabbage · 2026-05-18T14:33:29Z

-        "self": {"href": f"{host_url}{endpoint}?{string_query_args}&page={message_args.page}"},
-    }
-
-    if paginated_list.has_next:


As above this isn't used

LJBabbage · 2026-05-18T14:36:56Z

            "subject": self.subject,
            "body": self.body[:100] if body_summary else self.body,
            "thread_id": self.thread_id,
-            "case_id": self.case_id,


LJBabbage · 2026-05-18T14:38:21Z


    @staticmethod
    def thread_count_by_survey(request_args, user):
-        """Count users threads for a specific survey"""


merged with results to stop drift in the future and make more maintainable

LJBabbage · 2026-05-18T14:40:44Z

+                self.assertEqual(len(response), 1)

-                self.assertEqual(len(response.items), 1)
+    def test_thread_list_no_message(self):


Not related to this PR, but added line coverage

LJBabbage · 2026-05-19T09:00:33Z

            "from_internal": self.from_internal,
            "sent_date": str(self.sent_at),
            "read_date": str(self.read_at),
-            "_links": "",


Again not needed

LJBabbage · 2026-05-19T09:02:57Z

            )

-        logger.info("Successfully retrieved threads for user", user_uuid=g.user.user_uuid)
-        messages, links = process_paginated_list(result, request.host_url, g.user, message_args, THREAD_LIST_ENDPOINT)


I've split the internal and external respondent process up, so this is done else where

LJBabbage · 2026-05-19T09:04:23Z

@@ -1,8 +0,0 @@
-from behave import given, when


LJBabbage · 2026-05-19T09:04:32Z

@@ -1,8 +0,0 @@
-from behave import given, when


LJBabbage · 2026-05-19T09:08:23Z

  Then the thread count is '0' threads

-
-Scenario: Respondent sends messages to different ce, internal user filters by ce


We don't offer the ability to filter on these criteria https://github.com/ONSdigital/response-operations-ui/blob/main/response_operations_ui/controllers/message_controllers.py#L129 and never have so I removed it in the code and naturally the test here. Frontstage only allows filtering is_closed

LJBabbage · 2026-05-19T09:08:54Z

    | specific user |
    | group        |

-  Scenario Outline: There are 3 conversations between an internal user and respondent each with 2  messages ,


As above we don't have this functionality in rops

LJBabbage · 2026-05-19T09:14:53Z

-        if messages:
-            messages = add_users_and_business_details(messages)
-        return jsonify({"messages": messages, "_links": links})
+        return jsonify({"messages": messages})


_links isn't used at all in rops or frontstage, so it and all code associated with it can be removed. Rops doesn't even mock it in tests, frontstage does, which has now been removed without any effect.

anwilkie · 2026-05-20T15:00:00Z

/deploy wilkia

ras-rm-pr-bot · 2026-05-20T15:00:29Z

Deploying to dev cluster with following parameters:

namespace: wilkia
tag: pr-472
configBranch: main
paramKey: ``
paramValue: ``

matthew-robinson-ons · 2026-05-21T13:48:29Z

+                args = get_args(new_respondent_conversations=True)
+                response = Retriever.retrieve_thread_list(self.user_internal, args)
+                self.assertEqual(len(response), 1)
+                self.assertEqual(response[0]["labels"], ["INBOX", "UNREAD"])


Should this also include the "SENT" label?

Sent is only associated with a ROPS user i believe, the assert is also only proving what I set above is coming back. The irony is that the system only cares about UNREAD (to highlight) and not the others, but I didn't want to unpick it at this juncture

When i tested it SENT is getting attached to a respondent and a Rops user but I wasn't sure how the labels were actually being used. I did think that maybe they weren't all used when they returned to Rops/frontstage so that makes sense if it is just UNREAD. It's working the same on the PR as on main so no issues with it.

Respondent creates a new thread: SENT <respondent_uuid> INBOX GROUP UNREAD GROUP Rops creates a new thread: SENT <rops_user_uuid> INBOX <respondent_uuid> UNREAD <respondent_uuid>

matthew-robinson-ons · 2026-05-21T14:05:18Z

+"""
+
+PAGINATION_SQL = """
+ORDER BY sm.msg_id, sm.id DESC


The my messages tab is displaying the messages in sent_at DESC in the current implementation

matthew-robinson-ons

Tested multiple message scenarios. Labels are consistently displayed as before, attaching to the correct actor. Messages move in and out of my messages as before based on which internal user replied most recently. It's hard to comment on the performance of the queries without having anything to compare it against, we only really have that information in PROD currently.

The pagination object is displaying the messages on the page in a different order, previously it was showing the most recent (last sent) messages first.

RAS-1867 Re-implement My Messages to make query more performant

27f2d46

LJBabbage requested a review from a team as a code owner May 18, 2026 14:31

LJBabbage commented May 18, 2026

View reviewed changes

LJBabbage and others added 2 commits May 19, 2026 09:39

Fix tests and tidy up

8997b74

auto patch increment

58d224a

LJBabbage commented May 19, 2026

View reviewed changes

restore add_users_and_business_details

98fe880

anwilkie approved these changes May 20, 2026

View reviewed changes

matthew-robinson-ons reviewed May 21, 2026

View reviewed changes

matthew-robinson-ons requested changes May 21, 2026

View reviewed changes

fix order

399d9af

		Then the thread count is '0' threads


		Scenario: Respondent sends messages to different ce, internal user filters by ce

Conversation

LJBabbage commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why?

Performance

Additional changes

How to test

Jira

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LJBabbage May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anwilkie commented May 20, 2026

Uh oh!

ras-rm-pr-bot commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthew-robinson-ons left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LJBabbage commented May 18, 2026 •

edited

Loading

LJBabbage May 18, 2026 •

edited

Loading