Skip to content

Commit 0facebc

Browse files
Sam-BollingSam-Bolling
committed
Merge upstream/main into integrate/upstream-merge-2026-05-09
Brings in 21 upstream commits (Botts-Innovative-Research/OSHConnect-Python) Highlights: - AnyComponent over SerializeAsAny (7e6ac05) - prevents silent SWE field loss - Time comparison + deserialization fix (ea73e22) - VectorSchema serialization alias fix (5541ccd) - SWE Common 3.0 conformance updates (97cd5e2, 0d6ef64, 5a4a970) - Discovery: more reliable subresource attachment (04bee27) - CSAPI Part 3 pub/sub topics + event bus refactor (95bdb6b, 406bd26) - SQLite-backed DataStore (2e15c29) - MkDocs migration + Docker DevX improvements Conflict resolution: README.md - kept fork-focused content, appended upstream MkDocs section since it documents new mkdocs.yml and docs/markdown/ files coming in with the merge.
2 parents d8c4058 + 5bd72d7 commit 0facebc

51 files changed

Lines changed: 5048 additions & 506 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/docs_pages.yaml‎

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,16 @@
11
name: Docs2Pages
22
on: [ push, pull_request, workflow_dispatch ]
3-
permissions:
4-
contents: write
3+
4+
permissions: {}
55

66
jobs:
7-
build-docs:
7+
build:
88
runs-on: ubuntu-latest
9+
concurrency:
10+
group: docs-build-${{ github.ref }}
11+
cancel-in-progress: true
12+
permissions:
13+
contents: read
914
steps:
1015
- name: Checkout
1116
uses: actions/checkout@v5
@@ -19,15 +24,30 @@ jobs:
1924
- name: Install dependencies
2025
run: uv sync --all-extras
2126

22-
- name: Sphinx build
23-
run: |
24-
uv run sphinx-build -b html docs/source docs/build/html
27+
- name: Build MkDocs site
28+
run: uv run mkdocs build --strict
2529

26-
- name: Deploy documentation
27-
uses: peaceiris/actions-gh-pages@v4
30+
- name: Upload Pages artifact
2831
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
32+
uses: actions/upload-pages-artifact@v3
2933
with:
30-
publish_branch: gh-pages
31-
github_token: ${{ secrets.GITHUB_TOKEN }}
32-
publish_dir: ./docs/build/html
33-
force_orphan: true
34+
path: ./docs/build/html
35+
36+
deploy:
37+
needs: build
38+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
39+
runs-on: ubuntu-latest
40+
permissions:
41+
pages: write
42+
id-token: write
43+
environment:
44+
name: github-pages
45+
url: ${{ steps.deployment.outputs.page_url }}
46+
# Per GitHub Pages guidance: don't cancel an in-flight deploy.
47+
concurrency:
48+
group: pages
49+
cancel-in-progress: false
50+
steps:
51+
- name: Deploy to GitHub Pages
52+
id: deployment
53+
uses: actions/deploy-pages@v4

‎README.md‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,4 +129,52 @@ API Documentation: [https://botts-innovative-research.github.io/OSHConnect-Pytho
129129

130130
## License
131131

132-
See [LICENSE](LICENSE).
132+
See [LICENSE](LICENSE).
133+
134+
## Generating the Docs
135+
136+
The documentation is built with [MkDocs](https://www.mkdocs.org/) using the
137+
Material theme, [mkdocstrings](https://mkdocstrings.github.io/) for
138+
auto-generated API reference from the source, and
139+
[mermaid](https://mermaid.js.org/) for architecture diagrams. Markdown sources
140+
live under `docs/markdown/`.
141+
142+
Install dev dependencies (including MkDocs and plugins):
143+
144+
```bash
145+
uv sync
146+
```
147+
148+
Build the HTML docs:
149+
150+
```bash
151+
uv run mkdocs build
152+
```
153+
154+
The output will be in `docs/build/html/`. Open `docs/build/html/index.html` in
155+
a browser to view locally.
156+
157+
For a live-reloading preview while editing:
158+
159+
```bash
160+
uv run mkdocs serve
161+
```
162+
163+
Then visit http://127.0.0.1:8000.
164+
165+
To match what CI publishes (warnings become errors — useful when you've
166+
touched docstrings):
167+
168+
```bash
169+
uv run mkdocs build --strict
170+
```
171+
172+
CI builds the site on every push and deploys `main` to GitHub Pages via
173+
`.github/workflows/docs_pages.yaml`.
174+
175+
The legacy Sphinx setup under `docs/source/` is kept temporarily for
176+
reference and builds to a separate output directory:
177+
178+
```bash
179+
uv run sphinx-build -b html docs/source docs/build/sphinx
180+
```

‎docker-compose.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# =============================================================================
2+
# OSHConnect-Python — Local PyPI Server
3+
#
4+
# A lightweight pypiserver for publishing dev builds of oshconnect so that
5+
# downstream projects (OCSASim, etc.) can `pip install` normally instead of
6+
# pointing at raw wheel paths.
7+
#
8+
# Usage:
9+
# docker compose up -d # start the local PyPI
10+
# ./scripts/publish-local.sh # build wheel + upload to local PyPI
11+
# docker compose down -v # tear down + remove packages volume
12+
#
13+
# Consumer side (e.g. OCSASim):
14+
# pip install --index-url http://localhost:8090/simple/ oshconnect
15+
# uv pip install --index-url http://localhost:8090/simple/ oshconnect
16+
# =============================================================================
17+
18+
services:
19+
pypi:
20+
image: pypiserver/pypiserver:latest
21+
container_name: local-pypi
22+
command: run -a . -P . -o
23+
ports:
24+
- "8090:8080"
25+
volumes:
26+
- pypi-packages:/data/packages
27+
healthcheck:
28+
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:8080/"]
29+
interval: 5s
30+
timeout: 3s
31+
start_period: 5s
32+
retries: 3
33+
34+
volumes:
35+
pypi-packages:

‎docs/markdown/api.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# API Reference
2+
3+
All public symbols are re-exported from the top-level package and can be
4+
imported directly:
5+
6+
```python
7+
from oshconnect import OSHConnect, Node, Datastream, TimePeriod, ObservationFormat
8+
```
9+
10+
Lower-level CS API utilities are available from the `oshconnect.csapi4py`
11+
sub-package:
12+
13+
```python
14+
from oshconnect.csapi4py import APIResourceTypes, MQTTCommClient, ConnectedSystemsRequestBuilder
15+
```
16+
17+
---
18+
19+
## Core Application
20+
21+
::: oshconnect.oshconnectapi
22+
23+
---
24+
25+
## Streamable Resources
26+
27+
The primary objects for interacting with systems, datastreams, and control
28+
streams on an OSH node. Includes `Node`, `System`, `Datastream`,
29+
`ControlStream`, and supporting enums.
30+
31+
::: oshconnect.streamableresource
32+
33+
---
34+
35+
## Resource Data Models
36+
37+
Pydantic models that represent CS API resources returned from or sent to an
38+
OSH server.
39+
40+
::: oshconnect.resource_datamodels
41+
42+
---
43+
44+
## SWE Schema Components
45+
46+
Builder classes for constructing datastream and command schemas using SWE
47+
Common data types.
48+
49+
::: oshconnect.swe_components
50+
51+
::: oshconnect.schema_datamodels
52+
53+
---
54+
55+
## Event System
56+
57+
Pub/sub event bus for in-process notifications. Implement `IEventListener`
58+
to receive events.
59+
60+
::: oshconnect.eventbus
61+
62+
::: oshconnect.events.core
63+
64+
::: oshconnect.events.builder
65+
66+
---
67+
68+
## Time Management
69+
70+
::: oshconnect.timemanagement
71+
72+
---
73+
74+
## CS API Integration (`csapi4py`)
75+
76+
### Constants and Enums
77+
78+
::: oshconnect.csapi4py.constants
79+
80+
### Request Builder
81+
82+
::: oshconnect.csapi4py.con_sys_api
83+
84+
### API Helper
85+
86+
::: oshconnect.csapi4py.default_api_helpers
87+
88+
### MQTT Client
89+
90+
::: oshconnect.csapi4py.mqtt

‎docs/markdown/architecture.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Architecture
2+
3+
OSHConnect is structured around a small number of long-lived objects that mirror
4+
the resource hierarchy of the OGC API – Connected Systems specification.
5+
6+
## Object hierarchy
7+
8+
```mermaid
9+
graph TD
10+
OSHConnect[OSHConnect<br/><i>application entry point</i>]
11+
Node[Node<br/><i>connection to one OSH server</i>]
12+
APIHelper[APIHelper<br/><i>CS API HTTP requests</i>]
13+
Session[SessionManager<br/><i>OSHClientSession instances</i>]
14+
MQTT[MQTTCommClient<br/><i>paho-mqtt wrapper</i>]
15+
System[System<br/><i>sensor system</i>]
16+
Datastream[Datastream<br/><i>output channel — observations</i>]
17+
ControlStream[ControlStream<br/><i>input channel — commands &amp; status</i>]
18+
19+
OSHConnect --> Node
20+
Node --> APIHelper
21+
Node --> Session
22+
Node --> MQTT
23+
Node --> System
24+
System --> Datastream
25+
System --> ControlStream
26+
```
27+
28+
## Key abstractions
29+
30+
- **`OSHConnect`** (`oshconnectapi.py`) — top-level class. Owns nodes and
31+
provides `discover_systems()`, `discover_datastreams()`,
32+
`save_config()` / `load_config()`, and `create_and_insert_system()`.
33+
- **`Node`** (`streamableresource.py`) — wraps a server connection. Drives
34+
discovery via `APIHelper` and owns the `MQTTCommClient`. All HTTP resource
35+
creation goes through here.
36+
- **`StreamableResource`** (`streamableresource.py`) — abstract base for
37+
`System`, `Datastream`, and `ControlStream`. Manages MQTT
38+
subscriptions/publications, WebSocket connections, and the inbound /
39+
outbound message deques. Connection modes: `PUSH`, `PULL`, `BIDIRECTIONAL`.
40+
- **`Datastream` / `ControlStream`** (`streamableresource.py`) — concrete
41+
streamable resources. Datastreams publish observations; ControlStreams
42+
publish commands and receive status updates. Both follow CS API Part 3
43+
topic conventions (`:data`, `:status`, `:commands`).
44+
- **`resource_datamodels.py`** — Pydantic models for the CS API resource types
45+
(`SystemResource`, `DatastreamResource`, `ControlStreamResource`,
46+
`ObservationResource`). These map directly to API request and response
47+
bodies.
48+
- **`swe_components.py`** — Pydantic models for SWE Common schema components
49+
(`DataRecordSchema`, `QuantitySchema`, `VectorSchema`, etc.). Used to define
50+
observation and command schemas when creating new datastreams.
51+
- **`csapi4py/`** — sub-package that handles the CS API specifics: URL
52+
construction (`endpoints.py`), request building (`con_sys_api.py`), enums
53+
(`constants.py`), and MQTT topic conventions (`mqtt.py`).
54+
- **`EventHandler`** (`eventbus.py`) — singleton pub/sub bus. Listeners
55+
subscribe to event types (e.g. `NEW_OBSERVATION`) and topic strings; events
56+
are dispatched asynchronously through an internal queue.
57+
- **`timemanagement.py`** — `TimeInstant` (epoch / ISO-8601), `TimePeriod`,
58+
`TemporalModes` (`REAL_TIME`, `ARCHIVE`, `BATCH`), and `TimeUtils`
59+
conversions.
60+
61+
## Typical data flow
62+
63+
```mermaid
64+
sequenceDiagram
65+
autonumber
66+
participant App as OSHConnect
67+
participant N as Node
68+
participant H as APIHelper
69+
participant S as Server
70+
participant DS as Datastream
71+
72+
App->>N: add_node()
73+
App->>N: discover_systems()
74+
N->>H: retrieve_resource(SYSTEM)
75+
H->>S: HTTP GET /systems
76+
S-->>H: JSON
77+
H-->>N: System objects
78+
App->>DS: discover_datastreams()
79+
DS->>DS: initialize() — open MQTT/WebSocket
80+
DS->>DS: start() — begin streaming
81+
S-->>DS: observations → _inbound_deque
82+
Note over App,DS: To insert: resource.insert_self() →<br/>APIHelper.create_resource() → POST →<br/>server returns Location header with new ID
83+
```
84+
85+
## Dependencies
86+
87+
- **pydantic** — all resource and schema models. Bumping the minimum requires
88+
confirming pre-built wheels exist for all supported Python versions
89+
(3.12 – 3.14).
90+
- **shapely** — geometry handling for spatial resources.
91+
- **paho-mqtt** — MQTT streaming for CS API Part 3.
92+
- **websockets** / **aiohttp** — WebSocket and async HTTP streaming.
93+
- **requests** — synchronous HTTP for discovery and resource creation.

‎docs/markdown/index.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# OSHConnect-Python
2+
3+
OSHConnect-Python is the Python member of the OSHConnect family of application
4+
libraries. It provides a simple, straightforward way to interact with
5+
OpenSensorHub (or any other OGC API – Connected Systems server).
6+
7+
It supports Parts 1, 2, and 3 (Pub/Sub) of the OGC Connected Systems API,
8+
including:
9+
10+
- System, Datastream, and ControlStream discovery and management
11+
- Real-time MQTT streaming using CS API Part 3 `:data` topic conventions
12+
- Resource event topic subscriptions (CloudEvents lifecycle notifications)
13+
- Batch retrieval and archival stream playback
14+
- Configuration persistence (JSON save/load)
15+
- SWE Common schema builders for defining datastream and command schemas
16+
17+
All major classes and utilities are importable directly from `oshconnect`.
18+
Lower-level CS API utilities are available from `oshconnect.csapi4py`.
19+
20+
## Where to next
21+
22+
- [Architecture](architecture.md) — object hierarchy, data flow, and key abstractions
23+
- [Tutorial](tutorial.md) — common workflows for connecting, discovering, streaming, and inserting resources
24+
- [API Reference](api.md) — auto-generated reference for every public symbol

0 commit comments

Comments
 (0)