Fix stale group listing after rename in groups migration workflow

[mwojtyczka]

Changes

• Fix non-monotonic consistency gap between rename_workspace_local_groups and reflect_account_groups_on_workspace in the migrate-groups workflow
• When the groups API returns stale data showing a renamed group under its old name, detect this by comparing group IDs and proceed with account group reflection instead of skipping

Problem

A customer ran the migrate-groups workflow and observed that all steps completed successfully on the first run, but only groups were renamed — account groups were never added to the workspace and permissions were not migrated. Running the workflow a second time completed the remaining steps.

The root cause is a non-monotonic consistency gap between steps 2 and 3 of the workflow. Step 2 (rename_workspace_local_groups) renames groups and waits up to 2 minutes for the API listing to reflect the changes. However, even after this wait confirms the rename, step 3 (reflect_account_groups_on_workspace) makes a fresh API call that can hit a different cache server and see the old group name. When it sees name_in_account still present as a workspace group, it skips the group thinking it already exists — making both step 3 (reflect) and step 4 (apply permissions) no-ops. The existing 2-minute wait in _wait_for_renamed_groups does not protect against this because the groups API is not monotonically consistent: seeing the correct state once does not guarantee subsequent calls will also see it.

Fix

In reflect_account_groups_on_workspace, when a workspace group matching name_in_account is found, compare its ID against migrated_group.id_in_workspace:

• Same ID: the renamed group is appearing with its old name due to stale cache — log a warning and proceed with reflecting the account group
• Different ID: a genuinely different workspace group exists with that name — skip as before

Functionality

• added relevant user documentation
• added new CLI command
• modified existing command: databricks labs ucx ...
• added a new workflow
• modified existing workflow: groups-migration
• added a new table
• modified existing table: ...

Tests

• manually tested
• added unit tests
• added integration tests
• verified on staging environment (screenshot attached)

Verification on staging environment

Tested with 2 groups.

All 4 steps completed successfully in a single run. The rename step took ~6.7 minutes — well beyond the 2-minute consistency wait. And critically, reflect_account_groups_on_workspace succeeded and actually did work (321s, not a quick no-op skip), followed by apply_permissions completing in 11s. This confirms the fix is working — even if the groups API returned stale data, step 3 detected it by comparing group IDs and proceeded correctly.

Logs

rename_workspace_local_groups

10:04:12 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta ...
10:04:13 INFO [d.l.u.workspace_access.groups] Found 1962 Group
10:04:13 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
10:09:30 INFO [d.l.u.workspace_access.groups] Found 612 WorkspaceGroup
10:09:43 INFO [d.l.u.workspace_access.groups] Starting to rename 2 groups for migration...
10:09:55 INFO [d.l.blueprint.parallel][rename_groups_in_the_workspace_0] rename groups in the workspace 2/2, rps: 0.173/sec
10:09:55 INFO [d.l.blueprint.parallel] Finished 'rename groups in the workspace' tasks: 0% results available (0/2). Took 0:00:11.567695
10:09:56 INFO [d.l.blueprint.parallel][waiting_for_renamed_groups_in_the_workspace_0] waiting for renamed groups in the workspace 2/2, rps: 1.760/sec
10:09:56 INFO [d.l.blueprint.parallel] Finished 'waiting for renamed groups in the workspace' tasks: 0% results available (0/2). Took 0:00:01.137327
10:09:56 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ...
10:09:56 INFO [d.l.u.workspace_access.groups] Found 2574 WorkspaceGroup
10:09:56 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName ...
10:09:56 INFO [d.l.u.workspace_access.groups] Found 2574 WorkspaceGroup

reflect_account_groups_on_workspace

10:10:11 INFO [d.l.u.workspace_access.groups] Listing account groups with id,displayName,externalId...
10:10:11 INFO [d.l.u.workspace_access.groups] Found 2527 account groups
10:10:11 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta ...
10:10:11 INFO [d.l.u.workspace_access.groups] Found 1962 Group
10:10:11 INFO [d.l.u.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
10:15:17 INFO [d.l.u.workspace_access.groups] Found 612 WorkspaceGroup
10:15:19 INFO [d.l.u.workspace_access.groups] Starting to reflect 2 account groups into workspace for migration...
10:15:19 INFO [d.l.blueprint.parallel][reflect_account_groups_on_this_workspace_1] reflect account groups on this workspace 2/2, rps: 3.244/sec
10:15:19 INFO [d.l.blueprint.parallel] Finished 'reflect account groups on this workspace' tasks: 100% results available (2/2). Took 0:00:00.618280

apply_permissions

10:15:34 INFO [d.l.u.workspace_access.groups] Migrating permissions for 2 account groups.
10:15:34 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-a (workspace) -> rc-test-group-a (account) starting
10:15:35 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-a (workspace) -> rc-test-group-a (account) progress=11(+11)
10:15:35 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-a (workspace) -> rc-test-group-a (account) finished
10:15:35 INFO [d.l.u.workspace_access.groups] Migrated 11 permissions: db-temp-rc-test-group-a (workspace) -> rc-test-group-a (account)
10:15:35 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-b (workspace) -> rc-test-group-b (account) starting
10:15:36 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-b (workspace) -> rc-test-group-b (account) progress=5(+5)
10:15:37 INFO [d.l.u.workspace_access.groups] Migrating permissions: db-temp-rc-test-group-b (workspace) -> rc-test-group-b (account) finished
10:15:37 INFO [d.l.u.workspace_access.groups] Migrated 5 permissions: db-temp-rc-test-group-b (workspace) -> rc-test-group-b (account)
10:15:37 INFO [d.l.u.workspace_access.groups] Migrated 16 permissions for 2/2 groups successfully.
10:15:37 INFO [d.l.u.workspace_access.workflows] Group permission migration completed successfully.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.79%. Comparing base (d7a1c55) to head (efb6d6c).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4738      +/-   ##
==========================================
- Coverage   87.95%   87.79%   -0.16%     
==========================================
  Files         123      123              
  Lines       17570    17572       +2     
  Branches     3714     3714              
==========================================
- Hits        15453    15428      -25     
- Misses       1425     1456      +31     
+ Partials      692      688       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.