[QA EPIC] OpenSearch Migration — End-to-End QA Validation (Phases 0–3)

[fabrizzio-dotCMS]

Description

QA sub-epic for the ES → OpenSearch migration pipeline under #34164. Covers end-to-end validation of the dual-write pipeline across Phases 0–3, plus acceptance criteria for PR #35632 (startup hardening), PR #35735 (TLS/HTTPS connectivity fix), and PR #35609 (vendor-neutral SearchAPI — Phase 2/3 reads now testable).

Parent epic: #34164 — Migrate Mapping Layer classes
Fix PRs:

• fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632 — Startup validation hardening, haltMigration(), phase-aware index init, thread-safe formatter
• fix(opensearch): apply SSL context to connection manager and add OS_TLS_CERT_REQUIRED #35735 — TLS/HTTPS connectivity fix: SSL context wired correctly, OS_TLS_CERT_REQUIRED property added
• feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609 — Vendor-neutral SearchAPI and phase-aware router: Phase 2/3 read path now migrated

Test plans:

• docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md — base scenarios A–L, F (Phases 0–3, TC-001–TC-035)

Environment Setup

# Start migration stack (ES 7.10 + Kibana + OpenSearch 3.x + Dashboards)
docker compose -f docker/docker-compose-examples/os-migration/docker-compose.yml up -d

Service	URL	Purpose
Elasticsearch 7.10	http://localhost:9200	ES REST API
Kibana	http://localhost:5601	ES index inspection / queries
OpenSearch 3.x	http://localhost:9201	OS REST API
OpenSearch Dashboards	http://localhost:5602	OS index inspection / queries

# dotmarketing-config.properties — HTTP local setup
OS_ENDPOINTS=http://localhost:9201
OS_AUTH_BASIC_USER=admin
OS_AUTH_BASIC_PASSWORD=admin

# Phase flag: 0=ES only  1=dual-write+ES reads  2=dual-write+OS reads  3=OS only (search reads testable)
FEATURE_FLAG_OPEN_SEARCH_PHASE=0

# dotmarketing-config.properties — HTTPS / Kubernetes setup (PR #35735)
DOT_OS_ENDPOINTS=https://opensearch-host:9200
DOT_OS_AUTH_BASIC_USER=admin
DOT_OS_AUTH_BASIC_PASSWORD=secret
# DOT_OS_TLS_CERT_REQUIRED=true  ← uncomment to enforce strict cert validation

Migration Phase Reference

Phase	Writes	Reads	ES Status	OS Write Failures	Testable?
0	ES only	ES only	Active	n/a	Yes — fully
1	ES + OS	ES only	Active	Fire-and-forget (logged)	Yes — fully
2	ES + OS	OS only	Active (fallback)	Fire-and-forget (logged)	Yes — fully (PR #35609)
3	OS only	OS only	Decommissioned	Propagate to caller	Partially — search reads testable (PR #35609); write-path deferred

Scope

1. Zero regression on ES-backed functionality (Phase 0 baseline)
2. Correct dual-write in Phase 1 — every ES write is mirrored to OS; OS failures are fire-and-forget
3. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): haltMigration() auto-resets phase to 0 when OS is unreachable or has the wrong version at startup
4. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): hasEmptyIndices() is phase-aware — empty OS index in Phase 1 does not trigger a false ES reindex
5. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): threadSafeTimestampFormatter prevents corrupted index names under concurrent requests
6. NEW (PR fix(opensearch): apply SSL context to connection manager and add OS_TLS_CERT_REQUIRED #35735): HTTPS connectivity — SSL context applied correctly; OS_TLS_CERT_REQUIRED=false default; strict cert validation opt-in
7. NEW (PR feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609): Phase 2 application-level reads — search()/searchRaw() route to OSSearchAPIImpl; ES fallback on OS failure
8. NEW (PR feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609): SearchAPIImpl routing correctness per phase (0→ES, 1→ES, 2→OS, 3→OS)
9. NEW (PR feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609): Neutral DTOs: ContentSearchResults<T>, ContentSearchResponse, AggregationBucket
10. NEW (PR feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609): Phase 3 search read path — OS is primary, no ES fallback, failures propagate
11. Index lifecycle operations (activate, deactivate, delete, reindex) — including ES-only vs OS-only name variants
12. Content Type and contentlet create/delete sync between ES and OS shadow index
13. Multi-node cluster startup consistency in Phase 1
14. Invalid OS_ENDPOINTS graceful degradation
15. REST API security: DELETE /v1/esindex requires admin authentication
16. Complex query equivalence between Kibana and OpenSearch Dashboards

DB Validation Queries

-- All index rows
SELECT * FROM indicies ORDER BY index_name;
-- OS rows only (os:: prefix = DB artifact, never visible to callers)
SELECT * FROM indicies WHERE index_name LIKE 'os::%';
-- Count per provider
SELECT CASE WHEN index_name LIKE 'os::%' THEN 'OS' ELSE 'ES' END AS provider, COUNT(*) FROM indicies GROUP BY 1;
-- Expected Phase 0: 2 rows (ES only). Expected Phase 1 fresh: 4 rows (2 ES + 2 OS).

Child Tasks

PR #35632 Acceptance & Startup Hardening

• [QA-G1] OpenSearch Migration — PR #35632 Acceptance: Startup Hardening (TC-001–TC-005) #35635 — G1: PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632 Acceptance — Startup Hardening (TC-001–TC-005)
• [QA-G2] OpenSearch Migration — Invalid DOT_OS_ENDPOINTS Values (TC-006–TC-008) #35636 — G2: Invalid DOT_OS_ENDPOINTS Values (TC-006–TC-008)

Cluster & Infrastructure

• [QA-G3] OpenSearch Migration — Multi-Node Cluster Phase 1 (TC-009–TC-011) #35637 — G3: Multi-Node Cluster Phase 1 (TC-009–TC-011)

Content Lifecycle Sync

• [QA-G4] OpenSearch Migration — Content Type Lifecycle Sync (TC-012–TC-014) #35638 — G4: Content Type Lifecycle Sync (TC-012–TC-014)
• [QA-G5] OpenSearch Migration — Contentlet Delete Sync (TC-015) #35639 — G5: Contentlet Delete Sync (TC-015)

Index Lifecycle

• [QA-G6] OpenSearch Migration — Index Delete: ES vs OS Behavior (TC-016–TC-020) #35640 — G6: Index Delete — ES vs OS Behavior (TC-016–TC-020)

Security & Query Validation

• [QA-G7] OpenSearch Migration — REST Security: 401 / 403 on Index Endpoints (TC-021–TC-022) #35641 — G7: REST Security — 401/403 on Index Endpoints (TC-021–TC-022)
• [QA-G8] OpenSearch Migration — Complex Query Equivalence: Kibana vs OS Dashboards (TC-023–TC-025) #35642 — G8: Complex Query Equivalence — Kibana vs OS Dashboards (TC-023–TC-025)

SearchAPI Query Layer Migration (PR #35609 — previously blocked)

• [QA-G9] OpenSearch Migration — Phase 2 Application-Level Read Validation (C-6–C-8) #35751 — G9: Phase 2 Application-Level Read Validation (C-6–C-8)
• [QA-G10] OpenSearch Migration — SearchAPI Phase Routing & Neutral DTOs (TC-026–TC-035) #35752 — G10: SearchAPI Phase Routing & Neutral DTOs (TC-026–TC-035)
• [QA-G11] OpenSearch Migration — Phase 3 End-to-End Content Validation #35753 — G11: Phase 3 Search Read Scenarios (F-3–F-4)

TLS / HTTPS Connectivity (PR #35735)

• HTTPS endpoint connects without PKIX error when OS_TLS_CERT_REQUIRED is not set (default false)
• Startup log shows certRequired=false confirming the fix is active
• OS_TLS_CERT_REQUIRED=true rejects untrusted certs as expected

Acceptance Criteria

• All G1–G11 child tasks pass with documented results
• TLS/HTTPS connectivity validated per PR fix(opensearch): apply SSL context to connection manager and add OS_TLS_CERT_REQUIRED #35735 test plan
• Phase 2 application reads validated via SearchAPIImpl → OS (G9)
• SearchAPIImpl routing correctness confirmed for all 4 phases (G10)
• Phase 3 search read propagation validated (G11)
• DB validation queries pass at each phase transition
• Phase 3 write-path scenarios remain deferred (OS-only bulk indexing not yet validated at scale)
• Any match_phrase_prefix divergence (TC-023) filed as a separate bug

Additional Context

Depends on: #34164
Fix PRs: #35632, #35735, #35609