Skip to content

Commit f34287d

Browse files
Sam-BollingSam-Bolling
committed
docs: add CSAPI-Go server integration research report
1 parent eed2e4d commit f34287d

1 file changed

Lines changed: 391 additions & 0 deletions

File tree

‎docs/research/CSAPI_Go_Server_Integration_Report_2026-04-17.md‎

Lines changed: 391 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,391 @@
1+
# CSAPI-Go Server Integration Report
2+
3+
**Date:** 2026-04-17
4+
**Status:** In Progress
5+
**Scope:** Dual-publish the entire OS4CSAPI publisher fleet to the connected-systems-go server
6+
7+
---
8+
9+
## 1 Executive Summary
10+
11+
OS4CSAPI has deployed a second Connected Systems API server — [connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go) — alongside the existing OSH SensorHub. This report documents the integration effort: server architecture, behavioral differences discovered during live testing, workarounds applied, publishers migrated to date, and the plan to complete the remaining fleet.
12+
13+
**Current state:** 2 of 10 publishers dual-publishing. 6 GitHub issues filed against the Go server.
14+
15+
---
16+
17+
## 2 Server Architecture
18+
19+
### 2.1 Deployment Topology
20+
21+
| Component | SensorHub | connected-systems-go |
22+
|---|---|---|
23+
| **URL** | `https://129-80-248-53.sslip.io/sensorhub/api` | `https://129-80-248-53.sslip.io/csapi-go/` |
24+
| **Host** | Oracle Cloud VM (same host) | Oracle Cloud VM (same host) |
25+
| **Port** | 8181 (behind Caddy) | 8282 (behind Caddy) |
26+
| **Runtime** | Java / OSH SensorHub | Go binary / Docker |
27+
| **Storage** | H2 embedded DB | PostgreSQL + PostGIS |
28+
| **Auth** | HTTP Basic (`os4csapi` / `ogc134mm`) | None (tolerates auth headers) |
29+
| **IDs** | Short numeric (e.g., `0520`) | UUIDs (e.g., `a1b2c3d4-...`) |
30+
| **Reverse proxy** | Caddy route `/sensorhub/*` | Caddy route `/csapi-go/*` |
31+
| **Explorer preset** | "OSH SensorHub" | "CSAPI-Go" |
32+
33+
### 2.2 Docker Deployment
34+
35+
The Go server runs as a Docker container on the Oracle VM:
36+
37+
```
38+
docker run -d \
39+
--name csapi-go \
40+
--restart unless-stopped \
41+
-p 8282:8282 \
42+
-e DATABASE_URL=postgres://... \
43+
connected-systems-go:latest
44+
```
45+
46+
Caddy reverse-proxies `/csapi-go/*` to `localhost:8282`.
47+
48+
### 2.3 Explorer Integration
49+
50+
The [ogc-csapi-explorer](https://github.com/OS4CSAPI/ogc-csapi-explorer) demo app includes:
51+
52+
- **CF Pages proxy:** `demo/functions/api/csapi-go/[[path]].ts` → forwards to Go server
53+
- **Vite dev proxy:** `demo/vite.config.ts` → local dev routing
54+
- **Server selector:** "CSAPI-Go" option on the connection page (no auth required)
55+
56+
---
57+
58+
## 3 Behavioral Differences Discovered
59+
60+
During live integration testing, we identified 6 differences between connected-systems-go and OSH SensorHub. These have been filed as GitHub issues on [OS4CSAPI/connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go/issues).
61+
62+
### 3.1 Bugs (P1–P2)
63+
64+
#### Issue #1 — Datastream `uid` empty-string unique constraint violation
65+
66+
- **GitHub:** [OS4CSAPI/connected-systems-go#1](https://github.com/OS4CSAPI/connected-systems-go/issues/1)
67+
- **Severity:** P1-Critical
68+
- **Problem:** POST a datastream without `uid` → server stores `uid = ""`. Second POST (also without `uid`) fails with a PostgreSQL unique constraint violation.
69+
- **Workaround:** Always provide an explicit `uid` on datastream creation.
70+
- **Code change:** All `ensure_datastream()` calls now include `"uid": "urn:os4csapi:datastream:..."`.
71+
72+
#### Issue #2 — DELETE does not cascade FK constraints
73+
74+
- **GitHub:** [OS4CSAPI/connected-systems-go#2](https://github.com/OS4CSAPI/connected-systems-go/issues/2)
75+
- **Severity:** P1-Critical
76+
- **Problem:** DELETE on a parent resource (deployment, system) fails with a raw PostgreSQL FK violation instead of cascading or returning a structured `409` error.
77+
- **Workaround:** Delete child resources bottom-up (observations → datastreams → systems → deployments).
78+
- **Impact:** Bootstrap `--clean` operations must be carefully ordered.
79+
80+
### 3.2 Research Spikes (P3–P4)
81+
82+
#### Issue #3 — Time fields must be ISO 8601 strings
83+
84+
- **GitHub:** [OS4CSAPI/connected-systems-go#3](https://github.com/OS4CSAPI/connected-systems-go/issues/3)
85+
- **Severity:** P3-Minor
86+
- **Comparison:** SensorHub accepts both numeric epoch values and ISO strings. Go server rejects numerics.
87+
- **Workaround:** Publishers detect `"csapi-go"` in the base URL and coerce time fields to strings.
88+
- **Code change (USGS EQ):**
89+
```python
90+
self._coerce_time_to_str = "csapi-go" in self._base_url
91+
# In _post_observation():
92+
if self._coerce_time_to_str:
93+
for key in ("eventTime", "updatedTime"):
94+
if key in r and not isinstance(r[key], str):
95+
r[key] = str(r[key])
96+
```
97+
98+
#### Issue #4 — Rejects `"NaN"` strings for numeric fields
99+
100+
- **GitHub:** [OS4CSAPI/connected-systems-go#4](https://github.com/OS4CSAPI/connected-systems-go/issues/4)
101+
- **Severity:** P3-Minor
102+
- **Comparison:** SensorHub accepts `"NaN"` as a Gson-compatible token. Go server rejects it for schema-declared numeric fields.
103+
- **Workaround:** Publishers detect the Go server and replace `"NaN"` with `0.0`.
104+
- **Code change (OpenSky):**
105+
```python
106+
self._is_go_server = "csapi-go" in self._base_url
107+
# In _post_observation():
108+
if self._is_go_server:
109+
for key, val in r.items():
110+
if val == "NaN":
111+
r[key] = 0.0
112+
```
113+
- **Trade-off:** `0.0` is semantically different from "no data" — for altitude, 0 means sea level.
114+
115+
#### Issue #5 — Strict schema validation (all declared fields required)
116+
117+
- **GitHub:** [OS4CSAPI/connected-systems-go#5](https://github.com/OS4CSAPI/connected-systems-go/issues/5)
118+
- **Severity:** P3-Minor
119+
- **Comparison:** SensorHub accepts observations with a subset of schema-declared fields. Go server requires ALL fields present.
120+
- **Workaround:** Publishers include every declared field in every observation (e.g., duplicating `resultTime` as `result.timestamp`).
121+
- **Code change (OpenSky):** Keep `timestamp` field in the observation result even though it duplicates `resultTime`.
122+
123+
#### Issue #6 — Returns `@link` objects only, not flat `@id` strings
124+
125+
- **GitHub:** [OS4CSAPI/connected-systems-go#6](https://github.com/OS4CSAPI/connected-systems-go/issues/6)
126+
- **Severity:** P4-Informational
127+
- **Comparison:** SensorHub returns `"system@id": "0520"`. Go server returns `"system@link": { "href": "systems/<uuid>" }`.
128+
- **Impact:** Broke the Explorer's map view (0 datastreams shown) and the library's parent-system resolution.
129+
- **Workaround (Explorer):** `extractSystemId()` helper with `system@link.href` fallback. Pushed as commit `2f0869a`.
130+
- **Workaround (Library):** `parseBaseStream()` in `part2.ts` checks `system@link.href` → `system@id` chain. Filed as [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/166).
131+
132+
### 3.3 Other Go Server Characteristics
133+
134+
| Behavior | SensorHub | connected-systems-go |
135+
|---|---|---|
136+
| Collection key for geo-resources | `items` | `features` (GeoJSON) |
137+
| Collection key for datastreams | `items` | `items` |
138+
| ID format | Short numeric | UUID |
139+
| Auth requirement | Required (HTTP Basic) | None (headers tolerated) |
140+
| Datastream UID on create | Optional (auto-generated) | Effectively required (see #1) |
141+
142+
---
143+
144+
## 4 Code Changes Applied
145+
146+
### 4.1 Shared Infrastructure — `bootstrap_helpers.py`
147+
148+
**`find_by_uid()`** — Updated to check both `items` and `features` keys in collection responses, since the Go server wraps geo-resources in GeoJSON `features` arrays:
149+
150+
```python
151+
# Support both GeoJSON (features) and flat JSON (items) collections
152+
items = result.get("items", []) or result.get("features", [])
153+
```
154+
155+
### 4.2 USGS Earthquake Publisher
156+
157+
**Files changed:** `bootstrap_usgs_eq.py`, `usgs_eq_publisher.py`
158+
159+
| Change | Detail |
160+
|---|---|
161+
| `uid` on datastream | Added to bootstrap: `"uid": "urn:os4csapi:datastream:usgs-eq-feed:earthquakeEvent:v1"` |
162+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var to target Go server |
163+
| Time coercion | `_coerce_time_to_str` flag — converts `eventTime`/`updatedTime` to strings when targeting Go |
164+
165+
**Systemd services:**
166+
- `usgs-eq-publisher.service` → SensorHub
167+
- `usgs-eq-publisher-go.service` → Go server (env: `OSH_BASE_URL=https://129-80-248-53.sslip.io/csapi-go`)
168+
169+
**Confirmed:** 300 published, 0 errors per cycle on both servers.
170+
171+
### 4.3 OpenSky ADS-B Publisher
172+
173+
**Files changed:** `bootstrap_opensky.py`, `opensky_publisher.py`
174+
175+
| Change | Detail |
176+
|---|---|
177+
| `uid` on datastream | Added to bootstrap: `"uid": "urn:os4csapi:datastream:opensky-feed:adsbState:v1"` |
178+
| `OSH_BASE_URL` override | Publisher reads `OSH_BASE_URL` env var |
179+
| `_is_go_server` flag | Detects Go server from URL |
180+
| NaN → 0.0 | Replaces `"NaN"` strings with `0.0` for numeric fields |
181+
| Keep `timestamp` field | Retains the field in observation results (Go requires all schema fields) |
182+
183+
**Systemd services:**
184+
- `opensky-publisher.service` → SensorHub
185+
- `opensky-publisher-go.service` → Go server
186+
187+
**Confirmed:** ~170 published, 0 errors per cycle on both servers.
188+
189+
---
190+
191+
## 5 Go Server Resource Inventory
192+
193+
After bootstrapping USGS EQ and OpenSky on the Go server:
194+
195+
| Resource Type | Count | Examples |
196+
|---|---|---|
197+
| Systems | 2 | USGS Earthquake Feed, OpenSky ADS-B Feed |
198+
| Datastreams | 2 | earthquakeEvent, adsbState |
199+
| Deployments | 2 | USGS EQ deployment, OpenSky deployment |
200+
| Procedures | 2 | USGS EQ procedure, OpenSky procedure |
201+
| Observations | Growing | ~300 EQ events/cycle, ~170 aircraft states/cycle |
202+
203+
---
204+
205+
## 6 Dual-Publish Pattern
206+
207+
The established pattern for adding Go server support to any publisher:
208+
209+
### 6.1 Bootstrap Script Changes
210+
211+
1. Add explicit `uid` to all `ensure_datastream()` calls:
212+
```python
213+
ensure_datastream(base_url, auth, system_id, {
214+
"uid": "urn:os4csapi:datastream:<publisher>:<output>:v1",
215+
"outputName": "<output>",
216+
...
217+
})
218+
```
219+
220+
### 6.2 Publisher Code Changes
221+
222+
1. **`OSH_BASE_URL` override** — read from env to target a different server:
223+
```python
224+
self._base_url = os.environ.get(
225+
"OSH_BASE_URL",
226+
f"https://{self.osh_address}/{self.osh_root}/api",
227+
)
228+
```
229+
230+
2. **Go server detection** — set a flag for conditional behavior:
231+
```python
232+
self._is_go_server = "csapi-go" in self._base_url
233+
```
234+
235+
3. **Time coercion** — convert numeric time fields to strings:
236+
```python
237+
if self._is_go_server:
238+
for key in TIME_FIELDS:
239+
if key in result and not isinstance(result[key], str):
240+
result[key] = str(result[key])
241+
```
242+
243+
4. **NaN replacement** — substitute `0.0` for `"NaN"` strings:
244+
```python
245+
if self._is_go_server:
246+
for key, val in result.items():
247+
if val == "NaN":
248+
result[key] = 0.0
249+
```
250+
251+
5. **Include all schema fields** — ensure every declared field is present in every observation.
252+
253+
### 6.3 VM Deployment
254+
255+
1. Run bootstrap against Go server:
256+
```bash
257+
OSH_BASE_URL="https://129-80-248-53.sslip.io/csapi-go" \
258+
OSH_USER=dummy OSH_PASS=dummy \
259+
python -m publishers.<name>.bootstrap_<name>
260+
```
261+
262+
2. Create systemd service (copy existing, change env):
263+
```ini
264+
[Service]
265+
Environment="OSH_ADDRESS=129-80-248-53.sslip.io"
266+
Environment="OSH_BASE_URL=https://129-80-248-53.sslip.io/csapi-go"
267+
Environment="OSH_USER=dummy"
268+
Environment="OSH_PASS=dummy"
269+
ExecStart=/usr/bin/python3 -m publishers.<name>.<name>_publisher
270+
```
271+
272+
3. Enable and start: `sudo systemctl enable --now <name>-publisher-go.service`
273+
274+
---
275+
276+
## 7 Publisher Fleet Status
277+
278+
| # | Publisher | SensorHub | Go Server | Notes |
279+
|---|-----------|:---------:|:---------:|-------|
280+
| 1 | USGS Earthquake | ✅ Running | ✅ Running | First dual-publish. 300 obs/cycle. |
281+
| 2 | OpenSky ADS-B | ✅ Running | ✅ Running | ~170 obs/cycle. NaN→0.0 workaround. |
282+
| 3 | ISS | ✅ Running | ❌ Not started | 1 system, 2 DS. Low complexity. |
283+
| 4 | NWS | ✅ Running | ❌ Not started | 10 stations. Medium complexity. |
284+
| 5 | NDBC | ✅ Running | ❌ Not started | 5 buoys. Medium complexity. |
285+
| 6 | NDBC BuoyCAM | ✅ Running | ❌ Not started | 5 cameras. Image observations — may need special handling. |
286+
| 7 | CO-OPS | ✅ Running | ❌ Not started | 5 tide stations. Medium complexity. |
287+
| 8 | AviationWeather | ✅ Running | ❌ Not started | 5 METAR stations. Medium complexity. |
288+
| 9 | USGS Water | ✅ Running | ❌ Not started | 8 stations. Medium complexity. |
289+
| 10 | USGS NIMS | ✅ Running | ❌ Not started | 8 cameras. Image observations. |
290+
291+
**Excluded from dual-publish:**
292+
- **UAS Simulator** — FastAPI service, not a publisher in the same sense; separate integration path.
293+
- **Localizer** — Depends on UAS simulator data; separate integration path.
294+
295+
---
296+
297+
## 8 GitHub Issues Filed
298+
299+
All filed on [OS4CSAPI/connected-systems-go](https://github.com/OS4CSAPI/connected-systems-go/issues):
300+
301+
| # | Label | Title |
302+
|---|-------|-------|
303+
| [#1](https://github.com/OS4CSAPI/connected-systems-go/issues/1) | bug | Datastream creation without explicit `uid` stores empty string, violates unique constraint |
304+
| [#2](https://github.com/OS4CSAPI/connected-systems-go/issues/2) | bug | DELETE on parent resources fails with raw PostgreSQL FK constraint error |
305+
| [#3](https://github.com/OS4CSAPI/connected-systems-go/issues/3) | enhancement | Research: Time field encoding — strict ISO 8601 vs. numeric timestamps |
306+
| [#4](https://github.com/OS4CSAPI/connected-systems-go/issues/4) | enhancement | Research: NaN handling for numeric observation fields |
307+
| [#5](https://github.com/OS4CSAPI/connected-systems-go/issues/5) | enhancement | Research: Strict schema validation — requiring ALL declared fields |
308+
| [#6](https://github.com/OS4CSAPI/connected-systems-go/issues/6) | enhancement | Research: Cross-resource references — `@link` objects only vs. flat `@id` strings |
309+
310+
Related library issue: [ogc-client-CSAPI_2#166](https://github.com/OS4CSAPI/ogc-client-CSAPI_2/issues/166) — Library parsers need `@link.href` fallback.
311+
312+
---
313+
314+
## 9 Git Status
315+
316+
### OSHConnect-Python (this repo)
317+
318+
6 unpushed commits (`e022ef2`..`eed2e4d`) on `main`:
319+
320+
| Commit | Description |
321+
|--------|-------------|
322+
| `e022ef2` | Go CSAPI compat: uid on datastreams, str Time fields, GeoJSON features lookup, OSH_BASE_URL override |
323+
| `cfa4395` | Fix display URL to use actual base_url |
324+
| `1cda1d5` | Dual-publish compat: coerce Time fields to strings only for Go server |
325+
| `63c5201` | OpenSky: add OSH_BASE_URL override, uid on datastream, fix display URL |
326+
| `d331433` | OpenSky: keep timestamp field for Go server (schema validation requires it) |
327+
| `eed2e4d` | OpenSky: replace NaN strings with 0.0 for Go server (strict JSON validation) |
328+
329+
**Action needed:** Push to origin before continuing.
330+
331+
### ogc-csapi-explorer
332+
333+
Commit `2f0869a` (Go server `@link.href` compat) already pushed.
334+
335+
---
336+
337+
## 10 Next Steps — Remaining Publisher Migration
338+
339+
### Phase 1: Push existing work
340+
1. Push 6 unpushed OSHConnect-Python commits to GitHub
341+
342+
### Phase 2: Migrate remaining publishers (8)
343+
For each publisher, apply the dual-publish pattern (§6):
344+
345+
| Priority | Publisher | Rationale |
346+
|----------|-----------|-----------|
347+
| 1 | ISS | Simplest (1 system, 2 DS). Quick win to validate pattern. |
348+
| 2 | NWS | 10 stations, well-tested bootstrap. Good stress test. |
349+
| 3 | NDBC | 5 buoys, similar structure to NWS. |
350+
| 4 | CO-OPS | 5 tide stations, similar structure. |
351+
| 5 | AviationWeather | 5 METAR stations, similar structure. |
352+
| 6 | USGS Water | 8 stations, similar structure. |
353+
| 7 | NDBC BuoyCAM | Image observations — may require Go server testing for binary/URL payloads. |
354+
| 8 | USGS NIMS | Image observations — same considerations as BuoyCAM. |
355+
356+
### Phase 3: Verification
357+
- Confirm all Go server services running with 0 errors
358+
- Run Explorer smoke test against Go server
359+
- Update health-check dashboard to monitor Go server publishers
360+
361+
### Phase 4: Documentation
362+
- Update this report with final status
363+
- Update `Publisher_Fleet_Portability_Plan.md` to reference dual-publish capability
364+
- Update `README.md` with Go server instructions
365+
366+
---
367+
368+
## Appendix A — VM Service Inventory
369+
370+
Services running on `129.80.248.53` as of 2026-04-17:
371+
372+
```
373+
# SensorHub publishers (12)
374+
iss-publisher.service
375+
nws-publisher.service
376+
ndbc-publisher.service
377+
ndbc-buoycam-publisher.service
378+
coops-publisher.service
379+
aviation-wx-publisher.service
380+
opensky-publisher.service
381+
usgs-eq-publisher.service
382+
usgs-water-publisher.service
383+
usgs-nims-publisher.service
384+
385+
# Go server publishers (2)
386+
usgs-eq-publisher-go.service
387+
opensky-publisher-go.service
388+
389+
# Other
390+
simulator.service
391+
```

0 commit comments

Comments
 (0)