SensorML `documents` array silently dropped — system thumbnails/media links lost

[Sam-Bolling]

Summary

When a system is created or updated via PUT /systems/{id} with a SensorML body containing a documents array, the Go server accepts the request (HTTP 200/201) but silently drops the documents field. Subsequent GET /systems/{id}?resultFormat=sml responses contain identifiers, classifiers, contacts, and position — but no documents array at all.

This breaks any client that depends on SensorML document references (photos, datasheets, manuals, specification links).

Steps to Reproduce

1. POST/PUT a system with SensorML containing a documents array:

{
  "type": "PhysicalSystem",
  "uid": "urn:example:system:test",
  "name": "Test System",
  "documents": [
    {
      "role": "http://dbpedia.org/resource/Photograph",
      "name": "Station Hardware Photo",
      "link": {
        "href": "https://www.example.com/images/station.jpg",
        "type": "image/jpeg"
      }
    },
    {
      "role": "http://www.opengis.net/def/role/OGC/0/specificationDocument",
      "name": "Station Datasheet",
      "link": {
        "href": "https://www.example.com/docs/station_spec.pdf",
        "type": "application/pdf"
      }
    }
  ]
}

2. GET the system back with ?resultFormat=sml
3. Observe: the documents field is completely absent from the response

Actual Result

{
  "type": "PhysicalSystem",
  "uid": "urn:os4csapi:system:ndbc:station:41009:buoyfeed",
  "name": "Buoy 41009 Feed",
  "identifiers": [...],
  "classifiers": [...],
  "contacts": [...],
  "position": {...}
  // ← documents field is MISSING
}

Expected Result

{
  "type": "PhysicalSystem",
  "uid": "urn:os4csapi:system:ndbc:station:41009:buoyfeed",
  "name": "Buoy 41009 Feed",
  "identifiers": [...],
  "classifiers": [...],
  "contacts": [...],
  "position": {...},
  "documents": [
    {
      "role": "http://dbpedia.org/resource/Photograph",
      "name": "Station Hardware Photo",
      "link": {
        "href": "https://www.ndbc.noaa.gov/images/stations/41009.jpg",
        "type": "image/jpeg"
      }
    }
  ]
}

Impact

• CSAPI Explorer (and any SensorML-aware client) uses documents to extract system thumbnail photos, datasheets, and external reference links
• All 37 systems on the Go server are missing their photo thumbnails despite the publisher sending correct SensorML with documents on bootstrap
• This affects every system type (NDBC, CO-OPS, USGS, NWS, ISS, etc.)

Likely Location

The Go server's SensorML model struct probably doesn't include a Documents field, so the JSON unmarshaller silently drops it on ingest. Alternatively the serializer omits it on output.

Relevant SensorML spec reference: OGC SensorML 2.1, AbstractProcess → documentation / DocumentList

Environment

• Server version: connected-systems-go (latest main)
• Deployment: Oracle VM at 129.80.248.53:8282, Caddy route /csapi-go/*
• Confirmed on: 2026-04-17

Related Issues

• ?uid= query parameter is silently ignored — server returns all resources unfiltered #7 — ?uid= filter silently ignored
• Default pagination limit of 10 is unusually low — causes silent result truncation for typical workloads #9 — Default pagination limit of 10 too low

[Sam-Bolling]

Evaluation result — VALIDATED. Real defect, well-isolated. Recommend P2 firm (defensible P1 — silent data loss with 2xx).

Full write-up: docs/research/issue-evaluations/issue-010.md. Spec extracts and live transcripts under docs/research/evidence/issue-010/.

Root cause: the JSON struct-tag literal is wrong

// internal/model/domains/system.go:121 (and 8 other places)
Documentation common_shared.Documents `json:"documentation,omitempty"`
//                                          ^^^^^^^^^^^^^
//                                          should be `documents`

Spec authority

Canonical OGC SensorML JSON encoding DescribedObject.json (in opengeospatial/ogcapi-connected-systems/sensorml/schemas/json/):

"documents": {
  "description": "Additional documentation about the asset",
  "type": "array",
  "items": { "$ref": "Document.json" }
}

The property name is literally documents. PhysicalSystem.json → AbstractPhysicalProcess.json → AbstractProcess.json → DescribedObject.json; no intermediate schema redefines or aliases it. The canonical examples (weather_station_system.json, examples/spec/documents.json) both use "documents".

The issue body's spec citation "OGC SensorML 2.1, AbstractProcess → documentation / DocumentList" refers to the XML element name (<sml:documentation> containing a <DocumentList>). When SensorML was given a JSON encoding the SWG renamed the property to the array-friendly plural documents to match contacts / identifiers / classifiers. This XML-to-JSON naming difference is the most plausible origin of the bug — someone read the older XML schema or English prose and used documentation as the JSON tag.

Live verification on HEAD — definitive isolation

	Body property	HTTP	GET roundtrip preserves it?
T1	"documents": [...] (spec-correct)	201 Created	No — entirely absent
Control	"documentation": [...] (cs-go's tag)	201 Created	Yes — full payload returned

The control proves the underlying types, JSONB persistence, and marshal/unmarshal pipeline are all functional. The defect is isolated entirely to the JSON struct-tag literal. POST/control transcripts in live-test-2026-04-30.txt.

Why data is silently dropped

Go's encoding/json defaults: unknown JSON keys are silently ignored on unmarshal (cs-go does not call DisallowUnknownFields); nil + omitempty is omitted entirely on marshal. So a spec-correct documents array hits the unknown-key branch on POST → never persisted → never serialized back. HTTP 201 returns because everything else in the payload deserialized cleanly; the client gets no signal that part of its payload was discarded.

Scope — broader than just /systems

Inventory of Documents json tags across cs-go (9 fields, 5 files):

internal/model/common_shared/history.go:97          documentation
internal/model/domains/deployment.go:34             documents          ← only correct tag (DB struct)
internal/model/domains/deployment.go:112            documentation
internal/model/domains/procedure.go:32              documentation
internal/model/domains/procedure.go:122             documentation
internal/model/domains/system_event.go:22           documentation
internal/model/domains/system.go:41                 documentation
internal/model/domains/system.go:121                documentation
internal/model/domains/system.go:151                documentation

8 of 9 are wrong. The single correct tag (deployment.go:34) is the GORM DB struct, but the SML output struct on line 112 still uses documentation, so the deployment wire format is also broken. Same root cause; same fix; affects all SensorML-accepting endpoints (/systems, /deployments, /procedures, /system-events). For that reason I am not filing separate follow-up issues — keeping this on #10 with explicit broader scope. The maintainer can split into per-resource tickets if preferred.

Severity

Filed bug with no explicit severity. Recommend P2 firm, defensible P1:

• For P2: localized to one field; types, pipeline, persistence all work; affects optional documentation rather than core data.
• For P1: silent partial-data loss (HTTP 201 misleads the client); affects every spec-correct client; affects every resource type accepting SensorML; documented operational impact ("All 37 systems on the Go server are missing their photo thumbnails"); no client-side workaround that doesn't require the client to deliberately violate the spec.

The "silent loss with 2xx" pattern is the worst flavor of this kind of bug because clients have no signal to retry, log, or alert. Maintainer's call between P1 and P2 — fix path is identical.

Recommended fix

9-line search-and-replace:

- json:"documentation,omitempty"
+ json:"documents,omitempty"

Plus regression coverage: round-trip a documents array through SystemSML / DeploymentSML / ProcedureSML / EventSML structs.

Migration note: data persisted under the wrong tag (test rows from this evaluation, plus any rows from clients that adapted to cs-go's wrong spelling) will be inaccessible after the fix unless a JSONB column rename + value re-key migration is performed. The /deployments DB column is already named documents (per gorm tag); /systems and /procedures gorm tags also say documentation and need column verification.

Strong upstream PR candidate

SomethingCreativeStudios/connected-systems-go should receive this as an upstream PR. The fix is mechanical, the root cause is unambiguous, and the spec evidence is decisive.

Methodology rules added

1. Spec-name vs. encoding-name vs. struct-name vs. column-name are four separate concepts. XML element names ≠ JSON property names, even for conceptually identical schemas. Always verify the encoding-specific name from the canonical encoding schema — not prose, not the XML element name, not a sibling field.
2. Issue's own spec citation can be a misleading translation. This issue's spec reference is technically accurate (it cites the XML element name) but does not establish the JSON encoding name. Always re-derive the encoding name independently.
3. Internal inconsistency is a smell of half-awareness. cs-go has the correct tag in one place and the wrong tag everywhere else — signals partial prior awareness without complete propagation.
4. Control tests with the wrong-but-matching name cheaply isolate root cause. Posting once with documents and once with documentation took 30 seconds and conclusively pinned the bug to the tag literal. Use this for any "field silently dropped" claim.

[Sam-Bolling]

Following up before closing — verification against current upstream/main shows the fix is partial. Posting site-by-site evidence so @SomethingCreativeStudios can decide whether to expand scope or whether the remaining sites are intentionally out of scope.

What is fixed

The documents ↔ documentation tag mismatch is repaired on the SensorML output path for /systems:

# commit c2ab201 — "time range and better 400"
# internal/model/domains/system.go:148  (SystemSensorMLFeature)
- Documentation common_shared.Documents `json:"documentation,omitempty"`
+ Documentation common_shared.Documents `json:"documents,omitempty"`

And the Deployment DB/input struct already carried the correct tag (per commit f716e6c):

// internal/model/domains/deployment.go:34  (Deployment)
Documentation common_shared.Documents `gorm:"type:jsonb" json:"documents,omitempty"`

This is the site that the original live test hit (T1 — POST /systems with "documents":[...] then GET back as SensorML). Once a server with c2ab201 is deployed, that exact reproducer will roundtrip cleanly. Confirmed via static diff; not yet re-run live since the public head endpoint is on a pre-merge SHA at the time of writing.

What is still broken on upstream/main

git grep for the wrong literal across internal/:

internal/model/common_shared/history.go:97       Documentation Documents          `json:"documentation,omitempty"`
internal/model/domains/deployment.go:112         Documentation common_shared.Documents `json:"documentation,omitempty"`   // DeploymentSensorMLFeature
internal/model/domains/procedure.go:32           Documentation common_shared.Documents `gorm:"type:jsonb" json:"documentation,omitempty"`   // Procedure (DB/input)
internal/model/domains/procedure.go:122          Documentation common_shared.Documents `json:"documentation,omitempty"`   // ProcedureSensorMLFeature
internal/model/domains/system.go:41              Documentation common_shared.Documents `gorm:"type:jsonb" json:"documentation,omitempty"`   // System (DB/input)
internal/model/domains/system.go:121             Documentation common_shared.Documents `json:"documentation,omitempty"`   // SystemGeoJSONProperties
internal/model/domains/system_event.go:22        Documentation common_shared.Documents `gorm:"type:jsonb" json:"documentation,omitempty"`   // SystemEvent

Seven remaining sites. Concretely:

Site	Behavioural impact
system.go:41 (System, the input struct)	Headline symptom of this issue is still reproducible: POST /systems with spec-correct "documents":[...] → JSON unmarshaller skips the unknown key → Documentation stays nil → 201 Created → silent data loss on the input path. The c2ab201 fix only repairs the SensorML output path; for SensorML POST/PUT this is the field that decides whether the data is captured.
system.go:121 (SystemGeoJSONProperties)	GET /systems?f=geojson emits documentation, not documents. Nonconformant for any client requesting the GeoJSON encoding.
procedure.go:32 + :122	Same input-loss + output-spelling defect on /procedures.
deployment.go:112 (DeploymentSensorMLFeature)	SensorML output for /deployments still emits documentation. Counterpart to the system.go:148 fix that did land.
system_event.go:22 (SystemEvent)	Same defect on event resources.
common_shared/history.go:97 (Event.Documentation inside history)	History items embed the wrong tag.

The original evaluation §3 enumerated all 9 affected sites; only 2 are now correct.

Why this matters for the headline symptom

The publisher report ("All 37 systems missing their photo thumbnails despite the publisher sending correct SensorML with documents on bootstrap") routes through:

1. Client POSTs "documents":[...] → server unmarshals into System (line 41, still uses documentation tag) → field silently dropped → row persists with no documents.
2. (Even if input had landed) Client GETs SensorML → SystemSensorMLFeature (line 148, fixed) → would now emit documents correctly.

Step 2 is necessary but not sufficient — the input drop at step 1 still strips the data before it can ever round-trip out. So a deployment built from upstream/main today should still reproduce the issue's headline symptom on POST.

Question for @SomethingCreativeStudios

Two ways to read your "Typo in schema" triage note, and they imply different next steps:

• (A) You meant the whole class of typo (SystemSensorMLFeature was the most visible one but the same fix applies across the 9 sites) → then this issue should stay open until the remaining 7 are repaired. Mechanical fix is s/documentation,omitempty/documents,omitempty/g across the 5 affected files; the gotcha is the JSONB column rename (gorm tags on system.go:41, procedure.go:32, system_event.go:22 will rename DB columns, so a migration that copies values from the old documentation column to the new documents column is needed for any deployments that already accumulated data under the old column name).
• (B) You consider only the SensorML output (SystemSensorMLFeature) normative and the other sites out of scope (e.g. GeoJSON output is treated as best-effort, input is treated as accepting either spelling, etc.) → then closing as completed is appropriate, and the remaining sites can be tracked as separate issues if/when they bite.

I'm happy to either (i) leave this issue open as a tracker for case (A) and offer the search-and-replace + migration as a one-shot PR against your repo, or (ii) close as completed for case (B) and not file follow-ups unless someone hits one. Which would you prefer?

Cross-references

• Original evaluation: docs/research/issue-evaluations/issue-010.md (T1 vs control, 9-site static enumeration, spec authority extract).
• Spec authority: opengeospatial/ogcapi-connected-systems → sensorml/schemas/json/DescribedObject.json defines the property as literally documents (not documentation); weather_station_system.json and examples/spec/documents.json both use "documents". The XML element name <sml:documentation> in SensorML 2.1 is the likely source of the typo, since it does not match the JSON encoding.
• Fix-side commit: c2ab201 (SystemSensorMLFeature only).

Holding this issue open pending your call between (A) and (B).