codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2) by estebanzimanyi · Pull Request #3 · MobilityDB/MobilityKafka

estebanzimanyi · 2026-05-21T10:15:34Z

Add a generated, tier-aware Java facade over the full MEOS public API surface — Kafka Streams mirror of MobilityFlink #5.

What is generated

Identical shape to MobilityFlink #5 — same generators, same tier classification, same catalog source. Differs only in:

Package: org.mobilitydb.kafka.meos (vs org.mobilitydb.flink.meos)
Module: kafka-streams-app/ (vs flink-processor/)

Layer	Java class shape	Files	Methods
OO-classified	`MeosOps<Class>` — one per MEOS object-model class	50	751
Free fns	`MeosOpsFree<Header>` — one per public MEOS header for fns not assigned to any OO class	6	1,346
Shared runtime	`MeosOpsRuntime` (singleton `MEOS_AVAILABLE`)	1	—
Total		57	2,097 (77.7% of JMEOS PR #19's 2,699-method surface)

Tier breakdown: 804 stateless · 797 bounded-state · 161 windowed · 140 cross-stream · 195 io-meta.

Each emitted method forwards verbatim to functions.GeneratedFunctions.<name>(...) after probing MeosOpsRuntime.MEOS_AVAILABLE. Method Javadocs carry tier markers — see MobilityFlink #5 for the full tier vocabulary table.

Kafka Streams wiring per tier

Tier	Kafka Streams pattern
`stateless` / `bounded-state`	`mapValues(...)` / `filter(...)` — MEOS handle is per-event state
`windowed`	`groupByKey().windowedBy(TimeWindows...).aggregate(...)`
`cross-stream`	`KStream-KStream` join with windowed state store
`io-meta`	Serde / format helpers
`sequence-only` (14 fns, deferred)	Excluded — inherently non-streamable

Coexistence with `berlinmod.MEOSBridge`

Same coexistence pattern as MobilityFlink #5: MEOSBridge keeps the per-BerlinMOD-query intent; MeosOps* exposes the raw MEOS surface tier-by-tier. Both share the same MEOS_AVAILABLE discipline (via MeosOpsRuntime) and functions.GeneratedFunctions delegation.

Stacking

This PR stacks on feat/jmeos-bridge-swap. Additive-only: 57 new Java files + 5 files under tools/codegen/. No existing file is touched.

Base-branch compile state

Same situation as MobilityFlink — the recent bundled-jar refresh (c28fd83, JMEOS PR #19's jmeos-core jar) brought in the 2,699-method functions.GeneratedFunctions surface but did not include JMEOS PR #18's utils.spatial.* wrappers that the parallel session's berlinmod.MEOSBridge imports. Recipe to refresh with the union jar is in MobilityFlink PR #5's body. This PR's own diff is green in isolation against the refreshed jar.

Regeneration

See kafka-streams-app/tools/codegen/README.md (and the parallel flink-processor/tools/codegen/README.md on MobilityFlink #5 — same recipe).

…eams (27/27) All 27 cells of the BerlinMOD-9 × 3-form parity matrix on MobilityKafka, matching MobilityFlink MobilityDB#3's coverage on the Kafka-Streams runtime. Continuous form (9): per-event emission via record-by-record dispatch. Windowed form (9): STREAM_TIME punctuator at WINDOW_SIZE_MILLIS (10_000 ms) emitting closed windows. Snapshot form (9): STREAM_TIME punctuator at SNAPSHOT_TICK_MILLIS (5_000 ms) emitting per-tick state. State patterns: - Stateless filter (Q2-c, Q3-c, Q8-c) - Single keyed flag (Q1-c, Q1-s, Q4-c) - Single-key state (Q2-s, Q9-s) - Per-vehicle accumulator (Q6-c, Q6-s) - Per-(vehicle, POI) keyed state (Q7-c, Q7-s) - Cross-vehicle shared via selectKey(0) (Q5-c, Q3-s, Q5-s, Q7-s, Q8-s) - Paired shared via selectKey(0) (Q9-c, Q4-s, Q9-s) - winStart-keyed comma-separated vehicleId set (Q1-w, Q3-w, Q8-w) - winStart-keyed encoded last-known per vehicle (Q2-w, Q5-w, Q9-w) - winStart-keyed per-vehicle accumulator (Q6-w) - winStart-keyed per-vehicle entries log (Q4-w, Q7-w) Output counts (TopologyTestDriver, 21-event sorted-by-event-time corpus + 2 sentinel records at t=15001 and t=20001): Q | continuous | windowed | snapshot ---|------------|----------|--------- Q1 | 3 | 2 | 13 Q2 | 7 | 2 | 4 Q3 | 21 | 2 | 9 Q4 | 1 | 2 | 5 Q5 | 19 | 2 | 4 Q6 | 21 | 6 | 13 Q7 | 3 | 6 | 13 Q8 | 21 | 2 | 9 Q9 | 13 | 2 | 4 All 9 windowed cells emit 2 lines per Q (Q1/Q2/Q3/Q4/Q5/Q8/Q9) or 6 lines per Q (Q6/Q7 — one per vehicle × 2 windows for per-vehicle outputs). Continuous and snapshot count differences vs MobilityFlink reflect Kafka Streams' STREAM_TIME punctuator semantics: fires on stream-time advance with state-at-fire-moment, multi-interval jumps coalesced; vs Flink's bounded source flushing all keyed timers with final state at +infty. Dual-sentinel pattern in LocalTest steps the punctuator through the desired tick boundaries. Bump-isolation: zero JMEOS calls, zero MEOS-C calls, zero PyMEOS dependency. Pure Kafka Streams + Jackson + Java. Spatial predicates use Haversine / SegmentDistance / point-in-box, marked TODO(meos) for JMEOS-bridge migration.

… 1.4 MEOSBridge Mirrors the MobilityFlink bridge-swap PR. Introduces MEOSBridge as the runtime spatial-predicate surface for all 27 BerlinMOD-9 × 3-form Kafka Streams cells. The bridge calls MEOS via JMEOS 1.4 (geog_dwithin over WGS84 geographies) when libmeos is loadable and falls back to the pure-Java Haversine / SegmentDistance utilities when it is not. - New kafka-streams-app/src/main/java/berlinmod/MEOSBridge.java with the dwithinMetres / dwithinSegmentMetres / distanceMetres surface; fail-soft static init flips MEOS_AVAILABLE to false on UnsatisfiedLinkError. - All 27 Q<N>{Continuous,Snapshot,Windowed}Processor classes rewritten to call MEOSBridge instead of Haversine / SegmentDistance directly. - JMEOS.jar (478 305 bytes, JMEOS#15 regen artefact) brought into kafka-streams-app/jar/ and declared as a system-path dependency in pom.xml; jnr-ffi added explicitly since system-path jars do not bring transitive dependencies. - BerlinMODQ1LocalTest sets mobilitykafka.meos.enabled=false at main() entry so the TopologyTestDriver run stays green without libmeos.so on the runtime path. - target/ build artefacts gitignored. Build: mvn clean package -DskipTests green. Verify: BerlinMODQ1LocalTest finishes clean, all 27 cells emit the expected output.

…s + extended types + utils.spatial) Updates the bundled `kafka-streams-app/jar/JMEOS.jar` to a combined build of JMEOS PR #19 (regen against MEOS-API meos-idl.json, 2,699 methods including extended types) AND PR #18 (utils.spatial.Haversine + utils.spatial.PointToSegment wrappers that MEOSBridge.java imports). Surface delta vs the previous bundled jar: - public static methods: 2 699 (was 1 685) - utils.spatial.Haversine.distance(lon1, lat1, lon2, lat2) → double - utils.spatial.PointToSegment.distance(pLon, pLat, s1Lon, s1Lat, s2Lon, s2Lat) → double - tnpoint_ methods: 50 - tcbuffer / tpose / trgeo: now exposed - sha: a5895c9b94… size: 1,210,863 B Unblocks the MEOSBridge.java import path (line 116) — previously the jar shipped PR #19's GeneratedFunctions but not PR #18's utils.spatial, so base-branch mvn compile was RED. Both PRs now coalesced into a single jar. Unblocks codegen/kafka-meos-ops wedge stacked on this branch.

… surface (Kafka mirror of MobilityFlink #5) Mirror of MobilityFlink codegen/flink-meos-ops (PR #5) for the Kafka Streams binding. Same generators, same tier classification, same catalog source — differs only in package path (`org.mobilitydb.kafka.meos`) and module layout (`kafka-streams-app/` vs `flink-processor/`). Adds 50 `MeosOps<Class>` + 6 `MeosOpsFree<Header>` + 1 shared `MeosOpsRuntime` = 57 Java classes, 2,097 methods (77.7% of JMEOS PR #19's 2,699-method surface). Stacks on feat/jmeos-bridge-swap; additive-only; touches no existing file. See MobilityFlink #5 for the full tier vocabulary, regeneration recipe, and coexistence design with `berlinmod.MEOSBridge`.

estebanzimanyi · 2026-05-21T11:03:04Z

Coordination confirmation: rebased onto the post-union-jar refresh (6676bbb on Flink / fa70867 on Kafka — the JMEOS PR #19 + PR #18 union jar with utils.spatial.{Haversine,PointToSegment}).

Local verification:

$ mvn -q -DskipTests compile
# Flink: 123 .class files total, of which 57 MeosOps*
# Kafka:  94 .class files total, of which 57 MeosOps*

Full module now compiles green — the codegen wedge sits on top of MEOSBridge.java's utils.spatial.* imports without any inherited compile-red. mergeStateStatus = CLEAN. Ready for review/merge whenever.

Coordination item resolved. Thanks for the union-jar refresh.

estebanzimanyi added 4 commits May 21, 2026 08:38

estebanzimanyi force-pushed the codegen/kafka-meos-ops branch from a1d7fbf to 9ac326f Compare May 21, 2026 11:02

estebanzimanyi mentioned this pull request May 21, 2026

feat(wirings): Kafka Streams DSL tier wirings + capstone demo (stacks on #3; Kafka mirror of Flink #6→#10) #4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2)#3

codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2)#3
estebanzimanyi wants to merge 4 commits into
MobilityDB:mainfrom
estebanzimanyi:codegen/kafka-meos-ops

estebanzimanyi commented May 21, 2026

Uh oh!

estebanzimanyi commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

estebanzimanyi commented May 21, 2026

What is generated

Kafka Streams wiring per tier

Coexistence with berlinmod.MEOSBridge

Stacking

Base-branch compile state

Regeneration

Uh oh!

estebanzimanyi commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Coexistence with `berlinmod.MEOSBridge`