Skip to content

fix(import): eager magic-table creation for every schema (closes #1615)#1616

Merged
rubenvdlinde merged 1 commit into
developmentfrom
fix/eager-magic-table-creation
May 22, 2026
Merged

fix(import): eager magic-table creation for every schema (closes #1615)#1616
rubenvdlinde merged 1 commit into
developmentfrom
fix/eager-magic-table-creation

Conversation

@rubenvdlinde
Copy link
Copy Markdown
Contributor

Summary

Closes #1615. ImportHandler::importFromApp previously called MagicMapper::ensureTableForRegisterSchema() only from inside the seed-objects loop (line 3187 in the old layout). Schemas without seed data — typically log-style schemas like call_log, job_log, synchronization_log, synchronization_contract_log — therefore never got their oc_openregister_table_{registerId}_{schemaId} table during register import.

The fallback (lazy-create on first saveObject) only fires when a service runs end-to-end. If anything earlier in the chain swallows an error, the table stays missing forever and the dashboard reads zero.

Reproduction (and verified fix)

On a chain-E openconnector instance with feature/i18n-complete-translations + Repair step:

Before fix:

$ for sid in 213 218 219 221 223 224 225 226; do
    docker exec openregister-postgres psql -tAc \
      "SELECT 1 FROM information_schema.tables WHERE table_name='oc_openregister_table_65_$sid'"
  done
source                table_65_213 exists=1
job                   table_65_218 exists=1
mapping               table_65_219 exists=1
synchronization       table_65_221 exists=1
call_log              table_65_223 exists=NO
job_log               table_65_224 exists=NO
synchronization_log   table_65_225 exists=NO
synchronization_contract_log table_65_226 exists=NO

After fix + occ maintenance:repair --include-expensive:

source                table_65_213 exists=1
job                   table_65_218 exists=1
mapping               table_65_219 exists=1
synchronization       table_65_221 exists=1
call_log              table_65_223 exists=1
job_log               table_65_224 exists=1
synchronization_log   table_65_225 exists=1
synchronization_contract_log table_65_226 exists=1

Subsequent CallService::call() then succeeds and the call_log row is queryable via GET /api/objects/openregister/api/objects/openconnector/call_log (after a separate owner-attribution step — see #1617 below).

What changed

autoCreateRegisterIfApplication() (already runs for any x-openregister.type=application descriptor) gains a per-schema loop right after the Register row is created/reconciled and the Configuration's registers[] is synced. The loop iterates the existing $schemas parameter, is idempotent (MagicTableHandler::ensureTableForRegisterSchema short-circuits when the table exists, line 109-112), and per-schema-non-fatal (a failure logs a warning + continues).

The pre-existing seed-objects-loop pre-create (formerly at line 3187, now ~3225) stays as a defensive no-op for back-compat.

Out of scope (filed as follow-ups during this investigation)

  • Honor annotation: immutability + retention enforcement #1614x-openregister-archival annotation is silently stripped by the vocabulary check. Logs aren't immutable + retention rules don't run.
  • New issue I'll file alongside this PR: system-context writes (services running outside an HTTP request) write rows with empty _owner, which then get filtered out of REST list queries even for admin — needs an opt-in "system-owned" flag or auto-attribution to the source's owner.
  • New openconnector descriptor cleanup: call_log.sourceId + event_message.eventId|consumerId|subscriptionId were typed integer (chain-A leftovers); should be string format=uuid. Filing on openconnector side.

Test plan

  • Local: chain-E openconnector instance, 4 missing log tables created on re-import
  • CallService writes a call_log row successfully after the table exists
  • CI: PHP quality gates, unit tests, integration
  • Newman run on a CI ephemeral container produces non-zero call_log / job_log counts (was always zero before)

`ImportHandler::importFromApp` previously called
`MagicMapper::ensureTableForRegisterSchema` only from inside the
seed-objects loop (line 3187, formerly 3174). Schemas with an empty
seed-objects array therefore never got their per-schema magic table
provisioned during register import.

The fallback was lazy: the first write against the schema would trigger
`ensureTableForRegisterSchema` inside `MagicMapper::saveObject`. For
entity-style schemas (source, job, mapping, …) users write to them
directly so the table appears on first use. For **log-style schemas**
(call_log, job_log, synchronization_log, synchronization_contract_log
in ConductionNL/openconnector — and similar in other apps) writes only
happen from internal services. If anything earlier in the chain
swallowed an error (controller validation, find lookup, etc.), the
service never reached saveObject and the table stayed missing forever.

In practice: openconnector dashboards rendered empty even after
thousands of Newman + Playwright invocations because every
`POST /sources/test/{id}` 404'd at the controller (separate bug,
ConductionNL/openconnector#840), AND if it ever reached CallService
the saveObject would have thrown `relation oc_openregister_table_65_223
does not exist`.

## Fix

Provision the magic table for **every** Schema bound to the Register
inside `autoCreateRegisterIfApplication()`, right after the Register
is created / reconciled and its Configuration is synced. The loop
iterates the schemas already in scope, is idempotent (the underlying
helper short-circuits when the table already exists per
MagicTableHandler line 109-112), and non-fatal (any per-schema
failure logs a warning instead of aborting the whole import).

The original seed-objects-loop pre-create at line 3225 stays in place
as a defensive no-op for back-compat.

## Verified end-to-end on a chain-E openconnector instance

  occ maintenance:repair --include-expensive
  → OpenConnector Repair step fires importFromApp
  → autoCreateRegisterIfApplication runs the new loop
  → All 15 openconnector schemas now have magic tables:
       source / job / mapping / synchronization (entity)
       call_log / job_log / synchronization_log /
       synchronization_contract_log (log — previously missing)
       … plus 7 others.
  CallService->call() then succeeds and writes to call_log.
@github-actions
Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ a301acd

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
composer ✅ 162/162
npm ✅ 532/532
PHPUnit ⏭️
Newman ⏭️
Playwright ⏭️

Quality workflow — 2026-05-21 18:41 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde rubenvdlinde merged commit 56277fa into development May 22, 2026
16 of 20 checks passed
@rubenvdlinde rubenvdlinde deleted the fix/eager-magic-table-creation branch May 22, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant