Skip to content

Commit faa328b

Browse files
Sam-BollingSam-Bolling
committed
docs: update Go Server Integration Report — migration complete
All 10 publishers now dual-publishing. Updated sections: executive summary, resource inventory (37 sys/58 ds/11 proc/58 dep), 4 new behavioral differences, per-publisher code changes, fleet status table, VM deployment layout, git history, lessons learned.
1 parent 62b78a3 commit faa328b

1 file changed

Lines changed: 222 additions & 59 deletions

File tree

‎docs/research/CSAPI_Go_Server_Integration_Report_2026-04-17.md‎

Lines changed: 222 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# CSAPI-Go Server Integration Report
22

33
**Date:** 2026-04-17
4-
**Status:** In Progress
4+
**Status:** Complete
55
**Scope:** Dual-publish the entire OS4CSAPI publisher fleet to the connected-systems-go server
66

77
---
@@ -10,7 +10,7 @@
1010

1111
OS4CSAPI has deployed a second Connected Systems API server — [connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go) — alongside the existing OSH SensorHub. This report documents the integration effort: server architecture, behavioral differences discovered during live testing, workarounds applied, publishers migrated to date, and the plan to complete the remaining fleet.
1212

13-
**Current state:** 2 of 10 publishers dual-publishing. 6 GitHub issues filed against the Go server.
13+
**Final state:** 10 of 10 publishers dual-publishing on the Go server. 37 systems, 58 datastreams, 11 procedures, 58 deployments bootstrapped. All services running as systemd units with observations flowing. 6 GitHub issues filed against the Go server, plus 4 additional behavioral differences discovered during fleet-wide migration.
1414

1515
---
1616

@@ -129,7 +129,39 @@ During live integration testing, we identified 6 differences between connected-s
129129
- **Workaround (Explorer):** `extractSystemId()` helper with `system@link.href` fallback. Pushed as commit `2f0869a`.
130130
- **Workaround (Library):** `parseBaseStream()` in `part2.ts` checks `system@link.href` → `system@id` chain. Filed as [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/166).
131131

132-
### 3.3 Other Go Server Characteristics
132+
### 3.3 Additional Behavioral Differences (Discovered During Fleet Migration)
133+
134+
These were discovered during the full fleet migration and are not yet filed as issues.
135+
136+
#### Unique constraint on datastream `unique_identifier` (global scope)
137+
138+
- **Severity:** P2-Major
139+
- **Problem:** PostgreSQL `idx_datastreams_unique_identifier` enforces global uniqueness across ALL datastreams, not just within a system. Multi-station publishers initially shared one UID template (e.g., `urn:os4csapi:datastream:nws:nwsSurfaceObs:v1`) for all stations — the second station's datastream creation failed.
140+
- **Workaround:** All multi-station publishers now generate per-station UIDs: `urn:os4csapi:datastream:nws:{station_id}:nwsSurfaceObs:v1`.
141+
- **Files changed:** `bootstrap_nws.py`, `bootstrap_ndbc.py`, `bootstrap_coops.py`, `bootstrap_aviation_wx.py`, `bootstrap_usgs_water.py`, `bootstrap_usgs_nims.py`.
142+
143+
#### `?uid=` query parameter ignored
144+
145+
- **Severity:** P2-Major
146+
- **Problem:** `GET /systems?uid=urn:os4csapi:...` returns ALL systems (unfiltered) instead of the matching one. Combined with the default pagination limit of 10, `find_by_uid()` missed resources beyond the first page.
147+
- **Workaround:** `find_by_uid()` now appends `&limit=1000` to all queries and matches client-side.
148+
- **File changed:** `bootstrap_helpers.py`.
149+
150+
#### `/deployments` only returns top-level deployments
151+
152+
- **Severity:** P3-Minor
153+
- **Problem:** `GET /deployments` does not include sub-deployments in results. Sub-deployments are only accessible via `GET /deployments/{parent_id}/subdeployments`.
154+
- **Workaround:** `ensure_deployment()` now searches `deployments/{parent_id}/subdeployments` when `parent_id` is provided.
155+
- **File changed:** `bootstrap_helpers.py`.
156+
157+
#### Strict result schema validation (timestamp required)
158+
159+
- **Severity:** P3-Minor (extends Issue #5)
160+
- **Problem:** Go server validates that observation results contain ALL fields defined in the datastream schema, including `timestamp`. Publishers were popping `timestamp` from results (SensorHub auto-fills it from `phenomenonTime`), causing `result.timestamp is required by datastream schema` errors.
161+
- **Workaround:** All publishers now re-add `timestamp` from `phenomenonTime` when targeting Go server.
162+
- **Files changed:** All 8 publisher files.
163+
164+
### 3.4 Full Behavioral Comparison
133165

134166
| Behavior | SensorHub | connected-systems-go |
135167
|---|---|---|
@@ -138,19 +170,24 @@ During live integration testing, we identified 6 differences between connected-s
138170
| ID format | Short numeric | UUID |
139171
| Auth requirement | Required (HTTP Basic) | None (headers tolerated) |
140172
| Datastream UID on create | Optional (auto-generated) | Effectively required (see #1) |
173+
| Datastream UID scope | Per-system | Global unique constraint |
174+
| `?uid=` filter parameter | Supported | Ignored (returns all) |
175+
| Default pagination limit | 100 | 10 |
176+
| `/deployments` listing | All (flat) | Top-level only |
177+
| `result.timestamp` field | Auto-filled from phenomenonTime | Required in result body |
141178

142179
---
143180

144181
## 4 Code Changes Applied
145182

146183
### 4.1 Shared Infrastructure — `bootstrap_helpers.py`
147184

148-
**`find_by_uid()`** — Updated to check both `items` and `features` keys in collection responses, since the Go server wraps geo-resources in GeoJSON `features` arrays:
149-
150-
```python
151-
# Support both GeoJSON (features) and flat JSON (items) collections
152-
items = result.get("items", []) or result.get("features", [])
153-
```
185+
| Change | Detail | Commit |
186+
|---|---|---|
187+
| GeoJSON support | `find_by_uid()` checks both `items` and `features` keys | `e022ef2` |
188+
| `OSH_BASE_URL` | `get_config()` reads `OSH_BASE_URL` env var as server URL fallback | `906ae33` |
189+
| Pagination fix | `find_by_uid()` appends `&limit=1000` to all queries | `92f584b` |
190+
| Subdeployment search | `ensure_deployment()` searches `deployments/{parent_id}/subdeployments` when `parent_id` set | `3a02268` |
154191

155192
### 4.2 USGS Earthquake Publisher
156193

@@ -186,19 +223,109 @@ items = result.get("items", []) or result.get("features", [])
186223

187224
**Confirmed:** ~170 published, 0 errors per cycle on both servers.
188225

226+
### 4.4 ISS Publisher
227+
228+
**Files created:** `publishers/iss/bootstrap_iss.py` (new)
229+
**Files changed:** `publishers/iss/iss_publisher.py`
230+
231+
| Change | Detail |
232+
|---|---|
233+
| Bootstrap script | Created `bootstrap_iss.py` using `bootstrap_helpers.py` — 2 systems (position + track), 2 datastreams, 2 procedures, 1 deployment |
234+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
235+
| `_is_go_server` flag | Detects Go server from URL |
236+
| Timestamp in result | Re-adds `timestamp` from `phenomenonTime` when missing |
237+
238+
### 4.5 NWS Surface Observations Publisher
239+
240+
**Files changed:** `bootstrap_nws.py`, `nws_publisher.py`
241+
242+
| Change | Detail |
243+
|---|---|
244+
| Per-station datastream UID | `urn:os4csapi:datastream:nws:{station_id}:nwsSurfaceObs:v1` |
245+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
246+
| `_is_go_server` flag | NaN→0.0, timestamp re-added from phenomenonTime |
247+
248+
### 4.6 NDBC Buoy Publisher
249+
250+
**Files changed:** `bootstrap_ndbc.py`, `ndbc_publisher.py`, `ndbc_buoycam_publisher.py`
251+
252+
| Change | Detail |
253+
|---|---|
254+
| Per-station datastream UIDs | `urn:os4csapi:datastream:ndbc:{station_id}:ndbcBuoyObs:v1` and `urn:os4csapi:datastream:ndbc:{station_id}:ndbcBuoycam:v1` |
255+
| `OSH_BASE_URL` override | Both publishers read `OSH_BASE_URL` env var |
256+
| `_is_go_server` flag | NaN→0.0, timestamp re-added from phenomenonTime |
257+
258+
### 4.7 CO-OPS Coastal Observations Publisher
259+
260+
**Files changed:** `bootstrap_coops.py`, `coops_publisher.py`
261+
262+
| Change | Detail |
263+
|---|---|
264+
| Per-station datastream UID | `urn:os4csapi:datastream:coops:{station_id}:coopsCoastalObs:v1` |
265+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
266+
| `_is_go_server` flag | NaN→0.0, timestamp re-added from phenomenonTime |
267+
268+
### 4.8 AviationWeather METAR Publisher
269+
270+
**Files changed:** `bootstrap_aviation_wx.py`, `aviation_wx_publisher.py`
271+
272+
| Change | Detail |
273+
|---|---|
274+
| Per-station datastream UID | `urn:os4csapi:datastream:awx:{icao_id}:metarObs:v1` |
275+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
276+
| `_is_go_server` flag | NaN→0.0, timestamp re-added from phenomenonTime |
277+
278+
### 4.9 USGS Water Publisher
279+
280+
**Files changed:** `bootstrap_usgs_water.py`, `usgs_water_publisher.py`
281+
282+
| Change | Detail |
283+
|---|---|
284+
| Per-station datastream UIDs | `urn:os4csapi:datastream:usgs-water:{nwis_id}:usgsDischarge:v1` and `urn:os4csapi:datastream:usgs-water:{nwis_id}:usgsGageHeight:v1` |
285+
| Station key fix | Fixed `st["id"]` → `nwis_id` (station dict uses `nwisId` key) |
286+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
287+
| `_is_go_server` flag | Timestamp re-added from phenomenonTime |
288+
289+
### 4.10 USGS NIMS Imagery Publisher
290+
291+
**Files changed:** `bootstrap_usgs_nims.py`, `usgs_nims_publisher.py`
292+
293+
| Change | Detail |
294+
|---|---|
295+
| Per-camera datastream UID | `urn:os4csapi:datastream:usgs-nims:{cam_id}:usgsNimsImage:v1` |
296+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
297+
| `_is_go_server` flag | Timestamp re-added from phenomenonTime |
298+
| Companion pattern | NIMS creates datastreams on USGS Water systems (depends on Water bootstrap) |
299+
189300
---
190301

191302
## 5 Go Server Resource Inventory
192303

193-
After bootstrapping USGS EQ and OpenSky on the Go server:
304+
Final resource counts after complete fleet migration:
194305

195-
| Resource Type | Count | Examples |
196-
|---|---|---|
197-
| Systems | 2 | USGS Earthquake Feed, OpenSky ADS-B Feed |
198-
| Datastreams | 2 | earthquakeEvent, adsbState |
199-
| Deployments | 2 | USGS EQ deployment, OpenSky deployment |
200-
| Procedures | 2 | USGS EQ procedure, OpenSky procedure |
201-
| Observations | Growing | ~300 EQ events/cycle, ~170 aircraft states/cycle |
306+
| Resource Type | Count |
307+
|---|---|
308+
| Systems | 37 |
309+
| Datastreams | 58 |
310+
| Procedures | 11 |
311+
| Deployments | 58 |
312+
| Observations | Growing (561+ at first verification) |
313+
314+
### 5.1 Breakdown by Publisher
315+
316+
| Publisher | Systems | Datastreams | Procedures | Deployments |
317+
|---|---|---|---|---|
318+
| NWS | 10 | 10 | 1 | 12 |
319+
| NDBC (obs) | 5 | 5 | 1 | 7 |
320+
| NDBC (buoycam) | — | 5 | 1 | — |
321+
| CO-OPS | 5 | 5 | 1 | 7 |
322+
| AviationWeather | 5 | 5 | 1 | 7 |
323+
| USGS Water | 8 | 16 | 1 | 10 |
324+
| USGS NIMS | — | 8 | 1 | 10 |
325+
| USGS Earthquake | 1 | 1 | 1 | 2 |
326+
| OpenSky ADS-B | 1 | 1 | 1 | 2 |
327+
| ISS | 2 | 2 | 2 | 1 |
328+
| **Total** | **37** | **58** | **11** | **58** |
202329

203330
---
204331

@@ -275,23 +402,43 @@ The established pattern for adding Go server support to any publisher:
275402

276403
## 7 Publisher Fleet Status
277404

278-
| # | Publisher | SensorHub | Go Server | Notes |
279-
|---|-----------|:---------:|:---------:|-------|
280-
| 1 | USGS Earthquake | ✅ Running | ✅ Running | First dual-publish. 300 obs/cycle. |
281-
| 2 | OpenSky ADS-B | ✅ Running | ✅ Running | ~170 obs/cycle. NaN→0.0 workaround. |
282-
| 3 | ISS | ✅ Running | ❌ Not started | 1 system, 2 DS. Low complexity. |
283-
| 4 | NWS | ✅ Running | ❌ Not started | 10 stations. Medium complexity. |
284-
| 5 | NDBC | ✅ Running | ❌ Not started | 5 buoys. Medium complexity. |
285-
| 6 | NDBC BuoyCAM | ✅ Running | ❌ Not started | 5 cameras. Image observations — may need special handling. |
286-
| 7 | CO-OPS | ✅ Running | ❌ Not started | 5 tide stations. Medium complexity. |
287-
| 8 | AviationWeather | ✅ Running | ❌ Not started | 5 METAR stations. Medium complexity. |
288-
| 9 | USGS Water | ✅ Running | ❌ Not started | 8 stations. Medium complexity. |
289-
| 10 | USGS NIMS | ✅ Running | ❌ Not started | 8 cameras. Image observations. |
405+
All 10 publishers are dual-publishing on both SensorHub and the Go server.
406+
407+
| # | Publisher | SH Service | Go Service | Interval | Notes |
408+
|---|-----------|------------|------------|----------|-------|
409+
| 1 | USGS Earthquake | `usgs-eq-publisher` | `usgs-eq-publisher-go` | 60s | 300 obs/cycle |
410+
| 2 | OpenSky ADS-B | `opensky-publisher` | `opensky-publisher-go` | ~300s | ~170 obs/cycle, NaN→0.0 |
411+
| 3 | ISS | `iss-publisher` | `iss-publisher-go` | 30s | CelesTrak TLE fetch may timeout (transient) |
412+
| 4 | NWS | `nws-publisher` | `nws-publisher-go` | 3600s | 10 stations |
413+
| 5 | NDBC | `ndbc-publisher` | `ndbc-publisher-go` | 3600s | 5 buoys |
414+
| 6 | NDBC BuoyCAM | `ndbc-buoycam-publisher` | `ndbc-buoycam-publisher-go` | 900s | 5 cameras, image observations |
415+
| 7 | CO-OPS | `coops-publisher` | `coops-publisher-go` | 360s | 5 tide stations |
416+
| 8 | AviationWeather | `aviation-wx-publisher` | `aviation-wx-publisher-go` | 600s | 5 METAR stations |
417+
| 9 | USGS Water | `usgs-water-publisher` | `usgs-water-publisher-go` | 900s | 8 gages (discharge + gage height) |
418+
| 10 | USGS NIMS | `usgs-nims-publisher` | `usgs-nims-publisher-go` | 900s | 8 cameras, companion datastream pattern |
290419

291420
**Excluded from dual-publish:**
292421
- **UAS Simulator** — FastAPI service, not a publisher in the same sense; separate integration path.
293422
- **Localizer** — Depends on UAS simulator data; separate integration path.
294423

424+
### 7.1 VM Deployment Layout
425+
426+
Each Go publisher has its own directory under `/home/ubuntu/`:
427+
428+
```
429+
/home/ubuntu/nws-publisher-go/
430+
/home/ubuntu/ndbc-publisher-go/ # Also serves ndbc-buoycam-publisher-go
431+
/home/ubuntu/coops-publisher-go/
432+
/home/ubuntu/aviation-wx-publisher-go/
433+
/home/ubuntu/usgs-water-publisher-go/
434+
/home/ubuntu/usgs-nims-publisher-go/
435+
/home/ubuntu/iss-publisher-go/
436+
/home/ubuntu/usgs-eq-publisher/ # Shared dir (no -go suffix)
437+
/home/ubuntu/OSHConnect-Python/ # OpenSky uses main repo clone
438+
```
439+
440+
Each directory contains a copy of the relevant `publishers/` subtree plus `bootstrap_helpers.py`. Staging repo clone at `/tmp/OSHConnect-Python` for updates.
441+
295442
---
296443

297444
## 8 GitHub Issues Filed
@@ -311,11 +458,11 @@ Related library issue: [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-
311458

312459
---
313460

314-
## 9 Git Status
461+
## 9 Git History
315462

316463
### OSHConnect-Python (this repo)
317464

318-
6 unpushed commits (`e022ef2`..`eed2e4d`) on `main`:
465+
All commits pushed to `main`:
319466

320467
| Commit | Description |
321468
|--------|-------------|
@@ -325,43 +472,51 @@ Related library issue: [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-
325472
| `63c5201` | OpenSky: add OSH_BASE_URL override, uid on datastream, fix display URL |
326473
| `d331433` | OpenSky: keep timestamp field for Go server (schema validation requires it) |
327474
| `eed2e4d` | OpenSky: replace NaN strings with 0.0 for Go server (strict JSON validation) |
328-
329-
**Action needed:** Push to origin before continuing.
475+
| `b7e551f` | feat: dual-publish support for all 8 remaining publishers |
476+
| `906ae33` | feat: ISS bootstrap + bootstrap_helpers OSH_BASE_URL fix |
477+
| `e7f792a` | fix: per-station unique datastream UIDs for Go server bootstraps |
478+
| `92f584b` | fix: add limit=1000 to find_by_uid for Go server pagination |
479+
| `3a02268` | fix: search subdeployments under parent, fix USGS Water station key |
480+
| `62b78a3` | fix: ensure result.timestamp present for Go server schema validation |
330481

331482
### ogc-csapi-explorer
332483

333484
Commit `2f0869a` (Go server `@link.href` compat) already pushed.
334485

335486
---
336487

337-
## 10 Next Steps — Remaining Publisher Migration
488+
## 10 Lessons Learned
489+
490+
### 10.1 Go Server Requires Explicit Everything
491+
492+
SensorHub is lenient — auto-generating UIDs, accepting partial results, filling timestamps from envelope fields. The Go server enforces strict PostgreSQL constraints and JSON schema validation. Every field must be explicit.
338493

339-
### Phase 1: Push existing work
340-
1. Push 6 unpushed OSHConnect-Python commits to GitHub
494+
### 10.2 Bootstrap Idempotency Requires Reliable `find_by_uid`
341495

342-
### Phase 2: Migrate remaining publishers (8)
343-
For each publisher, apply the dual-publish pattern (§6):
496+
The `find_by_uid()` pattern (check if exists, skip or create) breaks when the server ignores the `uid` query parameter and pagination hides existing resources. The `&limit=1000` workaround is fragile for large deployments.
344497

345-
| Priority | Publisher | Rationale |
346-
|----------|-----------|-----------|
347-
| 1 | ISS | Simplest (1 system, 2 DS). Quick win to validate pattern. |
348-
| 2 | NWS | 10 stations, well-tested bootstrap. Good stress test. |
349-
| 3 | NDBC | 5 buoys, similar structure to NWS. |
350-
| 4 | CO-OPS | 5 tide stations, similar structure. |
351-
| 5 | AviationWeather | 5 METAR stations, similar structure. |
352-
| 6 | USGS Water | 8 stations, similar structure. |
353-
| 7 | NDBC BuoyCAM | Image observations — may require Go server testing for binary/URL payloads. |
354-
| 8 | USGS NIMS | Image observations — same considerations as BuoyCAM. |
498+
### 10.3 Multi-Station Publishers Need Per-Station UIDs
355499

356-
### Phase 3: Verification
357-
- Confirm all Go server services running with 0 errors
358-
- Run Explorer smoke test against Go server
359-
- Update health-check dashboard to monitor Go server publishers
500+
SensorHub treats datastream UIDs as optional and per-system scoped. The Go server's global `UNIQUE` constraint on `unique_identifier` means every datastream across the entire database must have a distinct UID.
360501

361-
### Phase 4: Documentation
362-
- Update this report with final status
363-
- Update `Publisher_Fleet_Portability_Plan.md` to reference dual-publish capability
364-
- Update `README.md` with Go server instructions
502+
### 10.4 Subdeployment Hierarchy Is Not Transparent
503+
504+
The Go server's `/deployments` endpoint only returns top-level deployments. Bootstraps that create child deployments must search under the parent explicitly.
505+
506+
### 10.5 VM Deployment Workflow
507+
508+
The current file-copy deployment (staging repo → per-publisher directories) is error-prone. Multiple bootstrap failures were caused by stale code in publisher directories. A git-pull-based or symlink-based approach would be more reliable.
509+
510+
---
511+
512+
## 11 Future Work
513+
514+
- **File GitHub issues** for the 4 newly discovered Go server behaviors (§3.3)
515+
- **Explorer smoke test** against Go server to verify map visualization
516+
- **Consolidate VM deployment** — replace per-directory file copies with symlinks or a deploy script
517+
- **UAS Simulator + Localizer** — evaluate Go server integration path
518+
- **Monitoring** — add health-check dashboard for Go publisher services
519+
- **CelesTrak resilience** — add retry/fallback for ISS TLE fetch timeouts
365520

366521
---
367522

@@ -370,7 +525,7 @@ For each publisher, apply the dual-publish pattern (§6):
370525
Services running on `129.80.248.53` as of 2026-04-17:
371526

372527
```
373-
# SensorHub publishers (12)
528+
# SensorHub publishers (10)
374529
iss-publisher.service
375530
nws-publisher.service
376531
ndbc-publisher.service
@@ -382,9 +537,17 @@ usgs-eq-publisher.service
382537
usgs-water-publisher.service
383538
usgs-nims-publisher.service
384539
385-
# Go server publishers (2)
386-
usgs-eq-publisher-go.service
540+
# Go server publishers (10)
541+
iss-publisher-go.service
542+
nws-publisher-go.service
543+
ndbc-publisher-go.service
544+
ndbc-buoycam-publisher-go.service
545+
coops-publisher-go.service
546+
aviation-wx-publisher-go.service
387547
opensky-publisher-go.service
548+
usgs-eq-publisher-go.service
549+
usgs-water-publisher-go.service
550+
usgs-nims-publisher-go.service
388551
389552
# Other
390553
simulator.service

0 commit comments

Comments
 (0)