feat(berlinmod): route the streaming-form parity matrix through JMEOS 1.4 MEOSBridge#2
Draft
estebanzimanyi wants to merge 3 commits into
Draft
Conversation
…eams (27/27) All 27 cells of the BerlinMOD-9 × 3-form parity matrix on MobilityKafka, matching MobilityFlink MobilityDB#3's coverage on the Kafka-Streams runtime. Continuous form (9): per-event emission via record-by-record dispatch. Windowed form (9): STREAM_TIME punctuator at WINDOW_SIZE_MILLIS (10_000 ms) emitting closed windows. Snapshot form (9): STREAM_TIME punctuator at SNAPSHOT_TICK_MILLIS (5_000 ms) emitting per-tick state. State patterns: - Stateless filter (Q2-c, Q3-c, Q8-c) - Single keyed flag (Q1-c, Q1-s, Q4-c) - Single-key state (Q2-s, Q9-s) - Per-vehicle accumulator (Q6-c, Q6-s) - Per-(vehicle, POI) keyed state (Q7-c, Q7-s) - Cross-vehicle shared via selectKey(0) (Q5-c, Q3-s, Q5-s, Q7-s, Q8-s) - Paired shared via selectKey(0) (Q9-c, Q4-s, Q9-s) - winStart-keyed comma-separated vehicleId set (Q1-w, Q3-w, Q8-w) - winStart-keyed encoded last-known per vehicle (Q2-w, Q5-w, Q9-w) - winStart-keyed per-vehicle accumulator (Q6-w) - winStart-keyed per-vehicle entries log (Q4-w, Q7-w) Output counts (TopologyTestDriver, 21-event sorted-by-event-time corpus + 2 sentinel records at t=15001 and t=20001): Q | continuous | windowed | snapshot ---|------------|----------|--------- Q1 | 3 | 2 | 13 Q2 | 7 | 2 | 4 Q3 | 21 | 2 | 9 Q4 | 1 | 2 | 5 Q5 | 19 | 2 | 4 Q6 | 21 | 6 | 13 Q7 | 3 | 6 | 13 Q8 | 21 | 2 | 9 Q9 | 13 | 2 | 4 All 9 windowed cells emit 2 lines per Q (Q1/Q2/Q3/Q4/Q5/Q8/Q9) or 6 lines per Q (Q6/Q7 — one per vehicle × 2 windows for per-vehicle outputs). Continuous and snapshot count differences vs MobilityFlink reflect Kafka Streams' STREAM_TIME punctuator semantics: fires on stream-time advance with state-at-fire-moment, multi-interval jumps coalesced; vs Flink's bounded source flushing all keyed timers with final state at +infty. Dual-sentinel pattern in LocalTest steps the punctuator through the desired tick boundaries. Bump-isolation: zero JMEOS calls, zero MEOS-C calls, zero PyMEOS dependency. Pure Kafka Streams + Jackson + Java. Spatial predicates use Haversine / SegmentDistance / point-in-box, marked TODO(meos) for JMEOS-bridge migration.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
12a7d6d to
a429f08
Compare
… 1.4 MEOSBridge
Mirrors the MobilityFlink bridge-swap PR. Introduces MEOSBridge as the
runtime spatial-predicate surface for all 27 BerlinMOD-9 × 3-form Kafka
Streams cells. The bridge calls MEOS via JMEOS 1.4 (geog_dwithin over
WGS84 geographies) when libmeos is loadable and falls back to the
pure-Java Haversine / SegmentDistance utilities when it is not.
- New kafka-streams-app/src/main/java/berlinmod/MEOSBridge.java
with the dwithinMetres / dwithinSegmentMetres / distanceMetres
surface; fail-soft static init flips MEOS_AVAILABLE to false on
UnsatisfiedLinkError.
- All 27 Q<N>{Continuous,Snapshot,Windowed}Processor classes rewritten
to call MEOSBridge instead of Haversine / SegmentDistance directly.
- JMEOS.jar (478 305 bytes, JMEOS#15 regen artefact) brought into
kafka-streams-app/jar/ and declared as a system-path dependency in
pom.xml; jnr-ffi added explicitly since system-path jars do not bring
transitive dependencies.
- BerlinMODQ1LocalTest sets mobilitykafka.meos.enabled=false at main()
entry so the TopologyTestDriver run stays green without libmeos.so
on the runtime path.
- target/ build artefacts gitignored.
Build: mvn clean package -DskipTests green.
Verify: BerlinMODQ1LocalTest finishes clean, all 27 cells emit the
expected output.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
a429f08 to
61af309
Compare
Member
Author
|
Coordination note from a sibling session — the JMEOS 1.4 bump unblock just landed. JMEOS PR #15's MEOS 1.4 bump intent is now landed as JMEOS PR #19 (open, MERGEABLE, CI green for PR #19 (
Stream-side consumers should base on PR #19's fork branch (or JMEOS PR #15 is now commented as "superseded by #19"; it stays open as a paper trail of the structural-drift diagnosis. No bump-wait blocks anything. Immediate action# Clone + base on the regen branch
git clone https://github.com/estebanzimanyi/JMEOS.git
cd JMEOS
git checkout feat/regen-extended-types-meos-idl
# Or stack on the wrappers branch for geodesic Haversine/PointToSegment too
git remote add wrappers https://github.com/estebanzimanyi/JMEOS.git
git fetch wrappers feat/spatial-haversine
git merge wrappers/feat/spatial-haversine # bring in PR #18's wrappersAfter that the full surface is available:
Stream-side platforms (MobilityFlink / MobilityKafka / MobilityNebula) can wire to this directly per the BerlinMOD-9 × 3 forms × 3 platforms parity contract — no blocker remains on the FFI side. Cross-verification this session ran todayA 50-class / 751-method codegen wedge ( |
…s + extended types + utils.spatial) Updates the bundled `kafka-streams-app/jar/JMEOS.jar` to a combined build of JMEOS PR #19 (regen against MEOS-API meos-idl.json, 2,699 methods including extended types) AND PR #18 (utils.spatial.Haversine + utils.spatial.PointToSegment wrappers that MEOSBridge.java imports). Surface delta vs the previous bundled jar: - public static methods: 2 699 (was 1 685) - utils.spatial.Haversine.distance(lon1, lat1, lon2, lat2) → double - utils.spatial.PointToSegment.distance(pLon, pLat, s1Lon, s1Lat, s2Lon, s2Lat) → double - tnpoint_ methods: 50 - tcbuffer / tpose / trgeo: now exposed - sha: a5895c9b94… size: 1,210,863 B Unblocks the MEOSBridge.java import path (line 116) — previously the jar shipped PR #19's GeneratedFunctions but not PR #18's utils.spatial, so base-branch mvn compile was RED. Both PRs now coalesced into a single jar. Unblocks codegen/kafka-meos-ops wedge stacked on this branch.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
c28fd83 to
fa70867
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirrors MobilityFlink#4. Introduces
MEOSBridgeas the runtime spatial-predicate surface for all 27 BerlinMOD-9 × 3-form Kafka-Streams cells. The bridge calls MEOS via JMEOS#15's 1.4 regen (geog_dwithinover WGS84 geographies) when libmeos is loadable and falls back to pure-JavaHaversine/SegmentDistancewhen it is not. TheBerlinMODQ1LocalTestTopologyTestDriver run setsmobilitykafka.meos.enabled=falseatmain()entry so it stays green without libmeos.so on the runtime path.What lands
kafka-streams-app/src/main/java/berlinmod/MEOSBridge.java— single point of contact between the BerlinMOD scaffold and MEOS. SurfacesdwithinMetres/dwithinSegmentMetres/distanceMetres. Static init catchesUnsatisfiedLinkErrorand flipsMEOS_AVAILABLEto false on libmeos absence.Q<N>{Continuous,Snapshot,Windowed}Processorclasses now callMEOSBridgeinstead ofHaversine/SegmentDistancedirectly.kafka-streams-app/jar/and declared as asystem-scope dep inpom.xml; jnr-ffi added explicitly since system-path jars do not transitively pull dependencies.BerlinMODQ1LocalTestsetsmobilitykafka.meos.enabled=falseatmain()entry so the TopologyTestDriver run is green without libmeos.so.target/build artefacts gitignored.Verified
mvn -q clean package -DskipTests— green on JDK 21.BerlinMODQ1LocalTest— clean exit. All 27 cells emit the expected per-(Q, form) outputs (Q1-cont 3 lines, Q3-snap 6 lines, Q5/Q9 pair distances, Q6 cumulative-distance series, Q8 segment-distance booleans).Dependency
This PR depends on the JMEOS.jar artefact produced by JMEOS#15's regen branch (same dep as MobilityFlink#4). The sequencing is:
feat/berlinmod-q1-scaffold(PR feat(berlinmod): full BerlinMOD-9 × 3-form parity matrix on Kafka Streams (27/27 cells) #1).Stacked on: #1
Sibling: MobilityFlink#4