[P3 bug] `null` on `optional:true` numeric field is rejected, while omission is accepted — asymmetric absence

[Sam-Bolling]

Summary

For a numeric Quantity field with optional: true, omitting the field is accepted (HTTP 201) but sending it with null is rejected (HTTP 400 must be a number). Both forms are user intent for "no value here", and the asymmetry is undocumented and confusing.

Reproduction

Datastream with baro_altitude declared as Quantity with "optional": true.

T14: POST observations  body={"result":{"geo_altitude":10500}}
  → HTTP 201 ✓ (omitted is OK)

T15: POST observations  body={"result":{"baro_altitude":null,"geo_altitude":10500}}
  → HTTP 400 {"error":"... result.baro_altitude must be a number"}

Full transcript: docs/research/evidence/issue-004/nan-tests-head-2026-04-30.txt (cases T14, T15).

Source

In internal/api/observation_schema_validation.go, the datarecord branch only checks for field absence to short-circuit on optional: true. When the field is present with null, control reaches the quantity branch with value = nil, the float64 type assertion fails, and the error fires.

Proposed fix

In the datarecord (and vector / dataarray) branch, treat value == nil (a JSON null) as equivalent to absence when the field is optional: true:

fieldVal, present := record[field.Name]
if (!present || fieldVal == nil) {
    if field.Optional != nil && *field.Optional { continue }
    return fmt.Errorf("%s.%s is required by datastream schema", path, field.Name)
}

Priority

P3 — UX / consistency; not a security or data-loss bug, but it forces clients to special-case absent vs explicit-null in their result encoders. Same diagnostic pattern documented in #19 for phenomenonTime vs resultTime: two ways of expressing absence should converge.

Cross-refs

• Parent research issue: Research: NaN handling for numeric observation fields — rejection vs. nilValue support #4
• Sibling asymmetric-absence pattern: decodeObservationPayload silently drops numeric phenomenonTime; should reject symmetrically with resultTime #19

[Sam-Bolling]

Verdict: KEEP — but with significant reframing

The behavioural pattern this issue describes is real and reproduces
verbatim against HEAD 08cf161. However, careful spec analysis
materially changes the verdict relative to the body: I recommend
downgrading severity from P3 bug → P4 diagnostic / documentation
and declining the body's proposed code fix. Detail below.

Live reproduction (5-row matrix on optional + required Quantity)

#	result	Actual
T1	omit optional	201 ✓
T2	optional=null	400 "result.baro_altitude must be a number"
T3	optional=42.5 (control)	201 ✓
T4	omit required	400 "result.geo_altitude is required by datastream schema"
T5	required=null	400 "result.geo_altitude must be a number"

T1 vs T2 reproduces the asymmetry in the body verbatim.

Static — body's diagnosis is precise

internal/api/observation_schema_validation.go:91-95 consults
field.Optional only inside if !exists. JSON null decodes to Go
nil, the key is present, the optional short-circuit is skipped,
recursion hits the leaf quantity branch and isNumber(nil) returns
false. The body's code-behaviour description is exactly correct.

Spec analysis — the part that changes the verdict

I checked the bundled OAS31 (Connected Systems Part 2,
ogcapi-connectedsystems-2.bundled.oas31.yaml):

1. optional is defined (line 521) as: "Specifies if the data for
   this component can be omitted in the datastream". The spec word
   is "omitted" — i.e. literally absent — not "omitted or null".

2. Zero occurrences of nullable and zero occurrences of
   "null" JSON-Schema types in the entire bundled OAS31. Quantity is
   type: number, full stop. Per JSON-Schema-2020-12 semantics,
   null is invalid against type: number — independent of
   optional.

3. SWE Common 3.0's canonical "no value here" carriers are: (a)
   omission of an optional component, or (b) the nilValues
   mechanism's spec-mandated string carriers ("NaN", "Infinity",
   "+Infinity", "-Infinity" — see [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21). JSON null is not in
   the SWE Common toolkit.

So the current server behaviour (reject null on a typed leaf) is
spec-conformant. This contrasts sharply with #21, where the
current behaviour is *non-*conformant (it rejects spec-mandated
nilValues strings). The two issues are surface-similar but their
spec status is opposite.

Why the body's proposed fix is not recommended

Silently accepting null as equivalent to absence on optional: true
fields would:

1. Teach clients a non-canonical encoding, pulling them away from
   spec-correct omission and from the nilValues mechanism that [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21
   is in flight to add.
2. Weaken schema enforcement (a value no spec schema permits would
   silently disappear).
3. Obscure real client serializer bugs (a client that emits null due
   to a serializer issue would no longer get the early-warning 400).
4. Create two competing absence mechanisms in parallel with [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21's
   nilValues addition, leaving four ways to say "no value":
   omission, null, nilValue carrier, and explicit reason — confusing
   client authors.

Recommended remediation (preserves spec conformance, addresses real concern)

The legitimate concern is diagnostic clarity — "must be a number" is misleading for a null value. Recommended fix:

1. On value == nil for any typed leaf branch (all 7 — quantity,
   count, boolean, time, category, text, *range), emit a clearer
   error:

   result.<field> is null; null is not a permitted value
   (use field omission for optional fields, or a declared nilValue
    such as "NaN" / "-Infinity" / "+Infinity" / "Infinity").
   

2. Update API user docs to clarify the asymmetry and direct clients to
   omission / nilValues for absence semantics.

3. Optional v1.1 follow-on (only with evidence of widespread
   client friction): accept null as syntactic sugar behind a
   documented configuration flag, off by default to preserve
   strict OAS31 conformance.

Adjacent diagnostic finding

T5 (required field = null) emits "must be a number" rather than the
clearer "is required" that T4 (omit-required) emits. Same
diagnostic-clarity concern; not a separate defect since the 400 outcome
is correct.

---

Evaluation artefacts (in our fork):

• docs/research/issue-evaluations/issue-022.md
• docs/research/evidence/issue-022/static-analysis-2026-04-30.md
• docs/research/evidence/issue-022/live-test-2026-04-30.md
• docs/research/evidence/issue-022/spec-authority-2026-04-30.md

[Sam-Bolling]

Leaving open — pushback before closing. This is the one case in the 26-issue corpus where our evaluation reached a conclusion that conflicts with the upstream implementation, and I want to surface the spec-conformance concern before this gets buried as resolved.

The implementation that landed upstream

SomethingCreativeStudios/connected-systems-go shipped the body's proposed fix. On upstream/main, internal/api/observation_schema_validation.go datarecord branch:

fieldVal, exists := obj[field.Name]
if !exists {
    if field.Optional != nil && *field.Optional {
        continue
    }
    // ... required-missing branch
}
// null JSON value for an optional field is acceptable (treat as absent).
if fieldVal == nil && field.Optional != nil && *field.Optional {
    continue
}

…with the parallel pattern applied to the vector branch for coordinates. T2 of our original test matrix (optional: true field set to null) now returns 201 instead of 400, exactly as the body proposed.

The maintainer's triage note was "kinda dup of other io related issue", and the triage list pinned this issue to commit a467aba ("Adding Strict Parsing... now unknown fields will cause a bad request with the path and field name"). But a467aba is about DisallowUnknownFields-style strict input parsing across the I/O layer — its diff (17 files, +491/−55) does not touch observation_schema_validation.go. The actual null-on-optional change landed alongside the schema-validation rewrite that addressed #21, not in a467aba. So the "kinda dup" note appears to be thematic grouping under the broader strict-input-parsing umbrella, not a literal commit pin.

Why we are pushing back

Our evaluation's spec analysis (preserved in issue-022.md) concluded the body's framing as a "P3 bug" was incorrect on a strict reading of the OGC SWE Common 3.0 / OAS31 specifications, and that the proposed fix would weaken schema enforcement in three concrete ways:

1. The OAS31 schema does not type-union null

Per spec-authority-2026-04-30.md:

• Quantity is declared as JSON Schema type: number. Full stop. No nullable, no ["number","null"] union, no oneOf with {type: "null"}.
• The bundled OAS31 contains zero occurrences of nullable: true and zero "null" JSON-Schema types.
• The optional flag on a component is documented as data that "can be omitted" (line 521 of the OAS31), not "omitted or null".

Per JSON Schema semantics, null is invalid against type: number regardless of any optional flag at a higher level. The maintainer's fix relaxes this without a corresponding spec change.

2. SWE Common already provides two canonical absence carriers; null is not one of them

SWE Common 3.0's "no value here" toolkit is:

Carrier	Semantic	Encoding
Field omission	"this field is not present in this record"	omit the JSON key entirely
nilValues table	"this field has a value, but the value carries a documented reason for being missing/special"	string carrier ("NaN", "-Infinity", etc., or a custom string declared in the datastream's nilValues table — closed under #21)

Adding JSON null as a third absence carrier creates a competing mechanism with no spec authority. Worse, it competes specifically with the nilValues mechanism that #21 just made canonical. The directional message we are now sending to clients is mixed:

• "Use nilValues to declare custom absence reasons" (per [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21's fix)
• "…or you can just send null and we'll silently accept it" (per [P3 bug] null on optional:true numeric field is rejected, while omission is accepted — asymmetric absence #22's fix)

A client could reasonably interpret this as "null is the easy path; nilValues is over-engineering." That is the opposite of the conformance trajectory the rest of the validator pass is on.

3. Schema enforcement is materially weakened

Pre-fix, a JSON serializer bug that emitted null for an optional: true numeric field would trigger an early-warning 400 — useful diagnostic feedback for client developers. Post-fix, the bug becomes silent: the field disappears from the stored observation with no signal to the client. This is the same class of "silent data loss" defect we just closed under #18 (TimeRange.UnmarshalJSON) and #19 (decodeObservationPayload numeric phenomenonTime). Re-introducing it — narrower in scope, but the same shape — undoes some of the diagnostic-clarity gains from those closures.

What we recommended instead

issue-022.md §"Recommended remediation":

1. Improve the null-typed leaf diagnostic uniformly across all 7 leaf-scalar branches:

   result.<field> is null; null is not a permitted value
   (use field omission for optional fields, or a declared nilValue
    such as "NaN" / "-Infinity" / "+Infinity" / "Infinity").
   

   This gives the body's complainant ("the error message must be a number is wrong for null") a precise, actionable, spec-correct fix.

2. Document the asymmetry in the API user docs: omission = "no value present"; nilValues carrier = "value is missing for a declared reason"; null is not accepted.

3. Optional v1.1 follow-on (only with adoption-data evidence): accept null as syntactic sugar for absence behind a configuration flag, off by default, preserving strict OAS31 conformance for default deployments.

This addresses the user-experience complaint (misleading "must be a number" message) without weakening schema enforcement, without competing with nilValues, and without driving clients away from canonical encoding.

Asking the maintainer to reconsider

@SomethingCreativeStudios — flagging this for your reconsideration. We agree with you on every other Fixed-bucket issue we've closed in this pass (and we've been closing them with detailed verification of your implementations). On this one, our spec analysis suggests the relaxation is going to cost more in conformance than it gains in UX:

• Would you consider gating the null → absent coercion behind a configuration flag, off by default? That preserves the strict OAS31 behavior for spec-aware deployments while keeping the UX-relaxed mode available for clients that need it.
• If not, would you at least consider improving the null-typed leaf diagnostic so a client sending null to a required field (T5 of the test matrix) sees a more informative message than the current "must be a number"? That's a free win regardless of the optional-null call.

We're prepared to keep this issue open on our fork pending your response, and (if you'd find it useful) we can prepare a PR against SomethingCreativeStudios/connected-systems-go implementing either the configuration-flag option or just the diagnostic improvement.

If after consideration you confirm the current behavior is the intended product call, we'll close this as won't-fix-per-upstream-decision and document the disagreement in our evaluation record. We respect that this is your call to make on your codebase.

Cross-references

• Original evaluation (with the spec analysis above): docs/research/issue-evaluations/issue-022.md.
• Static analysis: docs/research/evidence/issue-022/static-analysis-2026-04-30.md.
• Live tests: docs/research/evidence/issue-022/live-test-2026-04-30.md.
• Spec authority: docs/research/evidence/issue-022/spec-authority-2026-04-30.md.
• Adjacent: [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21 (canonical nilValues absence carrier — now competes with null), TimeRange.UnmarshalJSON silently swallows non-string and parse-failed times — Datastream POST stores empty TimeRange without error #18 (TimeRange silent-discard, the same shape of defect we just closed elsewhere).

Status: Open, pending maintainer response.