You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/research/CSAPI_Go_Server_Integration_Report_2026-04-17.md
+82-9Lines changed: 82 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@
10
10
11
11
OS4CSAPI has deployed a second Connected Systems API server — [connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go) — alongside the existing OSH SensorHub. This report documents the integration effort: server architecture, behavioral differences discovered during live testing, workarounds applied, publishers migrated to date, and the plan to complete the remaining fleet.
12
12
13
-
**Final state:** 10 of 10 publishers dual-publishing on the Go server. 37 systems, 58 datastreams, 11 procedures, 58 deployments bootstrapped. All services running as systemd units with observations flowing. 6 GitHub issues filed against the Go server, plus 4 additional behavioral differences discovered during fleet-wide migration.
13
+
**Final state:** 10 of 10 publishers dual-publishing on the Go server. 37 systems, 58 datastreams, 11 procedures, 58 deployments bootstrapped. All services running as systemd units with observations flowing. 10 GitHub issues filed against the Go server covering bugs, behavioral differences, and missing features. BuoyCAM image URL fix deployed (absolute URLs via `BUOYCAM_CACHE_BASE_URL` env var).
14
14
15
15
---
16
16
@@ -57,7 +57,7 @@ The [ogc-csapi-explorer](https://github.com/OS4CSAPI/ogc-csapi-explorer) demo ap
57
57
58
58
## 3 Behavioral Differences Discovered
59
59
60
-
During live integration testing, we identified 6 differences between connected-systems-go and OSH SensorHub. These have been filed as GitHub issues on [OS4CSAPI/connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go/issues).
60
+
During live integration testing, we identified 10+ differences between connected-systems-go and OSH SensorHub. These have been filed as GitHub issues on [OS4CSAPI/connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go/issues).
61
61
62
62
### 3.1 Bugs (P1–P2)
63
63
@@ -131,24 +131,35 @@ During live integration testing, we identified 6 differences between connected-s
131
131
132
132
### 3.3 Additional Behavioral Differences (Discovered During Fleet Migration)
133
133
134
-
These were discovered during the full fleet migration and are not yet filed as issues.
134
+
These were discovered during the full fleet migration. Key items have since been filed as GitHub issues.
135
135
136
136
#### Unique constraint on datastream `unique_identifier` (global scope)
137
137
138
138
-**Severity:** P2-Major
139
139
-**Problem:** PostgreSQL `idx_datastreams_unique_identifier` enforces global uniqueness across ALL datastreams, not just within a system. Multi-station publishers initially shared one UID template (e.g., `urn:os4csapi:datastream:nws:nwsSurfaceObs:v1`) for all stations — the second station's datastream creation failed.
140
140
-**Workaround:** All multi-station publishers now generate per-station UIDs: `urn:os4csapi:datastream:nws:{station_id}:nwsSurfaceObs:v1`.
-**Problem:**`GET /systems?uid=urn:os4csapi:...` returns ALL systems (unfiltered) instead of the matching one. Combined with the default pagination limit of 10, `find_by_uid()` missed resources beyond the first page.
147
149
-**Workaround:**`find_by_uid()` now appends `&limit=1000` to all queries and matches client-side.
-**Problem:** Go server defaults to `limit=10` for all collection endpoints. With 37 systems, 58 datastreams, and 58 deployments, default queries miss the majority of resources. SensorHub defaults to 100.
157
+
-**Workaround:** All client code appends explicit `&limit=100` or `&limit=1000` to queries.
158
+
-**Impact:** Affects Explorer UI (map shows only 10 of 37 systems), bootstrap scripts, and library consumers.
159
+
150
160
#### `/deployments` only returns top-level deployments
-**Problem:**`GET /deployments` does not include sub-deployments in results. Sub-deployments are only accessible via `GET /deployments/{parent_id}/subdeployments`.
154
165
-**Workaround:**`ensure_deployment()` now searches `deployments/{parent_id}/subdeployments` when `parent_id` is provided.
@@ -160,6 +171,16 @@ These were discovered during the full fleet migration and are not yet filed as i
160
171
-**Problem:** Go server validates that observation results contain ALL fields defined in the datastream schema, including `timestamp`. Publishers were popping `timestamp` from results (SensorHub auto-fills it from `phenomenonTime`), causing `result.timestamp is required by datastream schema` errors.
161
172
-**Workaround:** All publishers now re-add `timestamp` from `phenomenonTime` when targeting Go server.
162
173
-**Files changed:** All 8 publisher files.
174
+
-**Note:** Extension of Issue #5. Not filed as a separate issue.
-**Problem:** The Go server silently drops the `documents` array from SensorML on ingest or output. Querying `GET /systems?resultFormat=sml` returns `identifiers`, `classifiers`, `contacts`, `position` — but NO `documents`. Publishers send `documents` with photo URLs, thumbnails, and documentation links, but they are lost.
181
+
-**Impact:** System thumbnails broken in the Explorer (`extractSmlMedia()` reads `sml.documents`). Any media links, datasheets, or documentation URLs attached to systems are inaccessible.
182
+
-**Workaround:** None available (requires Go server code fix). Explorer shows blank thumbnails for all Go server systems.
183
+
-**Verification:**`GET /systems?resultFormat=sml` on Go server — response has no `documents` field. Same query on SensorHub returns full `documents` array with photo URLs.
163
184
164
185
### 3.4 Full Behavioral Comparison
165
186
@@ -171,10 +192,11 @@ These were discovered during the full fleet migration and are not yet filed as i
@@ -254,6 +276,7 @@ These were discovered during the full fleet migration and are not yet filed as i
254
276
| Per-station datastream UIDs |`urn:os4csapi:datastream:ndbc:{station_id}:ndbcBuoyObs:v1` and `urn:os4csapi:datastream:ndbc:{station_id}:ndbcBuoycam:v1`|
255
277
|`OSH_BASE_URL` override | Both publishers read `OSH_BASE_URL` env var |
256
278
|`_is_go_server` flag | NaN→0.0, timestamp re-added from phenomenonTime |
279
+
|`BUOYCAM_CACHE_BASE_URL` fix | Added `Environment=BUOYCAM_CACHE_BASE_URL=https://129-80-248-53.sslip.io/buoycam` to both `ndbc-buoycam-publisher.service` and `ndbc-buoycam-publisher-go.service`. Without this, `image_cache.py` produced relative paths (e.g., `/41009/2026/04/17/...jpg`) instead of absolute URLs. State file cleared and services restarted to force re-publish all 5 stations with correct URLs. |
257
280
258
281
### 4.7 CO-OPS Coastal Observations Publisher
259
282
@@ -411,7 +434,7 @@ All 10 publishers are dual-publishing on both SensorHub and the Go server.
411
434
| 3 | ISS |`iss-publisher`|`iss-publisher-go`| 30s | CelesTrak TLE fetch may timeout (transient) |
@@ -439,6 +462,29 @@ Each Go publisher has its own directory under `/home/ubuntu/`:
439
462
440
463
Each directory contains a copy of the relevant `publishers/` subtree plus `bootstrap_helpers.py`. Staging repo clone at `/tmp/OSHConnect-Python` for updates.
441
464
465
+
### 7.2 BuoyCAM Image Cache
466
+
467
+
BuoyCAM images are cached on disk at `/var/www/buoycam/` and served by Caddy:
468
+
469
+
```
470
+
# Caddy config
471
+
handle_path /buoycam/* {
472
+
root * /var/www/buoycam
473
+
header Access-Control-Allow-Origin *
474
+
file_server browse
475
+
}
476
+
```
477
+
478
+
The `image_cache.py` module builds image URLs using:
Without `BUOYCAM_CACHE_BASE_URL` set, the publisher produced relative paths like `/41009/2026/04/17/20260417T150121Z.jpg` — which broke in the Explorer since images need absolute URLs. Fixed by adding `Environment=BUOYCAM_CACHE_BASE_URL=https://129-80-248-53.sslip.io/buoycam` to both BuoyCAM systemd services. State files cleared and services restarted to force re-publish all 5 stations with correct absolute URLs.
|[#8](https://github.com/OS4CSAPI/connected-systems-go/issues/8)| bug | Subdeployments hidden from top-level `/deployments` listing |
504
+
|[#9](https://github.com/OS4CSAPI/connected-systems-go/issues/9)| bug | Default pagination limit of 10 too low — most resources hidden |
505
+
|[#10](https://github.com/OS4CSAPI/connected-systems-go/issues/10)| bug | SensorML `documents` array silently dropped — system thumbnails/media links lost |
456
506
457
-
Related library issue: [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/166) — Library parsers need `@link.href` fallback.
507
+
Related library issues:
508
+
-[ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/166) — Library parsers need `@link.href` fallback.
509
+
-[ogc-client-CSAPI_2#167](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/167) — `buildQueryString()` should not apply a default `limit` parameter.
458
510
459
511
---
460
512
@@ -507,16 +559,22 @@ The Go server's `/deployments` endpoint only returns top-level deployments. Boot
507
559
508
560
The current file-copy deployment (staging repo → per-publisher directories) is error-prone. Multiple bootstrap failures were caused by stale code in publisher directories. A git-pull-based or symlink-based approach would be more reliable.
509
561
562
+
### 10.6 Environment Variables Must Be Explicitly Set in Systemd
563
+
564
+
The BuoyCAM image URL issue was caused by a missing `BUOYCAM_CACHE_BASE_URL` environment variable in both publisher systemd services. The `image_cache.py` module defaults to `""` for the base URL, producing relative paths that work in development but break when the images are served from a different origin. All publisher-specific env vars must be audited when deploying new services.
565
+
510
566
---
511
567
512
568
## 11 Future Work
513
569
514
-
-**File GitHub issues** for the 4 newly discovered Go server behaviors (§3.3)
570
+
-~~**File GitHub issues** for the 4 newly discovered Go server behaviors (§3.3)~~ — Done: #7, #8, #9 filed; #10 filed for new `documents` issue
571
+
-**Go server SensorML `documents` support** — Issue #10 tracks this. System thumbnails and media links will remain broken until the Go server preserves the `documents` array on ingest/output. No client-side workaround available.
515
572
-**Explorer smoke test** against Go server to verify map visualization
516
573
-**Consolidate VM deployment** — replace per-directory file copies with symlinks or a deploy script
517
574
-**UAS Simulator + Localizer** — evaluate Go server integration path
518
575
-**Monitoring** — add health-check dashboard for Go publisher services
519
576
-**CelesTrak resilience** — add retry/fallback for ISS TLE fetch timeouts
577
+
-**Audit publisher env vars** — ensure all required environment variables (e.g., `BUOYCAM_CACHE_BASE_URL`) are set in every systemd service file
0 commit comments