feat(berlinmod): route the streaming-form parity matrix through JMEOS 1.4 MEOSBridge#4
Draft
estebanzimanyi wants to merge 3 commits into
Draft
Conversation
…matrix on MobilityFlink
All nine BerlinMOD reference queries × three streaming forms each
(continuous, windowed, snapshot) on MobilityFlink — the complete 27-cell
stream-layers parity-matrix row, locally verified end-to-end with no
external dependencies (no Kafka, no Docker, no MEOS native lib, no
JMEOS call).
Queries:
Q1 which vehicles have appeared in the stream
Q2 where is vehicle X at time T
Q3 which vehicles within d of P at time T
Q4 which vehicles entered region R, and when
Q5 pairs of vehicles meeting near point P
Q6 cumulative distance per vehicle
Q7 first passage of vehicles through POIs
Q8 vehicles close to a road segment
Q9 distance between vehicles X and Y at time T
Each query has three form classes (Q<N>{Continuous,Windowed,Snapshot}Function)
and a companion BerlinMODQ<N>LocalTest driver running the three forms
through a Flink mini-cluster against a hardcoded synthetic corpus.
Spatial predicates today are pure Java — Haversine distance for
point-to-point (Q3, Q5, Q6, Q9), point-in-box for region containment
(Q4), and a planar-projection point-to-line-segment distance (Q8). Each
spatial call site is marked TODO(meos) for migration to the JMEOS
bridge of the corresponding MEOS operator once the in-flight MEOS 1.4
bump signals settled (Q3 edwithin_tgeo_geo; Q4 STBox eintersects; Q5
NAD / edwithin_tgeo_tgeo; Q6 trajectory length; Q7 edwithin_tgeo_geo;
Q8 distance(tgeompoint, geometry(LINESTRING)); Q9 tdistance). Q1 and
Q2 have no spatial predicate.
State patterns exercised:
- keyed simple flag (Q1)
- keyed last-known position (Q2, Q8)
- keyed transition + entry log (Q4)
- keyed accumulator (Q6)
- keyed first-passage map (Q7)
- shared key-by-constant state (Q9 pair-wise, Q5 multi-pair MapState)
Verified output counts (see PR description for the exact-line excerpts):
Q | continuous | windowed | snapshot
---|------------|----------|---------
Q1 | 3 | 2 | 9
Q2 | 7 | 2 | 3
Q3 | 21 | 2 | 6
Q4 | 4 | 5 | 9
Q5 | 14 | 2 | 3 (only pair (100,200) qualifies for our P + radii)
Q6 | 21 | 6 | 9 (drift corpus; v100=601m, v200=300m, v300=1205m)
Q7 | 3 | 6 | 9 (3 (vehicle, POI) first-passages; intra-window scope)
Q8 | 21 | 2 | 6 (same shape as Q3 with segment-distance)
Q9 | 7 | 2 | 3 (X=100, Y=200; distance 4124m = ~4.1km)
Build verification: mvn clean package green; all nine LocalTests run to
completion (Flink mini-cluster, parallelism=1) producing exactly the
expected output shapes.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
574b6f1 to
0f9f70d
Compare
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
0f9f70d to
a77ad8d
Compare
… 1.4 MEOSBridge
Introduce MEOSBridge as the runtime spatial-predicate surface for all
BerlinMOD-9 × 3-form streaming cells. The bridge calls into MEOS via
JMEOS 1.4 (geog_dwithin over WGS84 geographies) when libmeos is loadable
and falls back to the pure-Java Haversine / SegmentDistance utilities
when it is not — the fallback path is what the BerlinMODQ*LocalTest
mini-cluster drivers exercise (system property mobilityflink.meos.enabled=false).
- New berlinmod/MEOSBridge.java with the dwithinMetres /
dwithinSegmentMetres / distanceMetres surface and a fail-soft
static init that flips MEOS_AVAILABLE to false on UnsatisfiedLinkError.
- All BerlinMOD-9 × 3-form spatial predicates rewritten to call
MEOSBridge instead of Haversine / SegmentDistance directly. 27 cells,
one bridge call surface, identical predicate semantics.
- JMEOS.jar updated to the JMEOS#15 regen branch artefact (478 305
bytes); this is the JMEOS 1.4 regen build that exposes geog_dwithin /
geom_in / geom_to_geog / edwithin_tgeo_geo / nad_tgeo_geo / tpoint_length.
- aisdata/Main.java and aisdata/TrajectoryWindowFunction.java adapted
to the JMEOS 1.4 meos_initialize() / meos_initialize_timezone()
split (the old two-arg meos_initialize(String, error_handler_fn)
signature is gone in JMEOS#15).
- All nine BerlinMODQ*LocalTest mini-cluster drivers set
mobilityflink.meos.enabled=false at main() entry so they remain
green-CI without libmeos.so on the runtime path.
- target/ build artefacts gitignored.
The README's spatial-predicate paragraph is updated to describe the
MEOSBridge route as the production path; the TODO(meos) markers across
the BerlinMOD cells are gone.
Build: mvn clean package -DskipTests green.
Verify: BerlinMODQ{1,3,5,8}LocalTest all finish with FINISHED state
on the mini-cluster fallback path.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
a77ad8d to
cf22d65
Compare
Member
Author
|
Coordination note from a sibling session — the JMEOS 1.4 bump unblock just landed. JMEOS PR #15's MEOS 1.4 bump intent is now landed as JMEOS PR #19 (open, MERGEABLE, CI green for PR #19 (
Stream-side consumers should base on PR #19's fork branch (or JMEOS PR #15 is now commented as "superseded by #19"; it stays open as a paper trail of the structural-drift diagnosis. No bump-wait blocks anything. Immediate action# Clone + base on the regen branch
git clone https://github.com/estebanzimanyi/JMEOS.git
cd JMEOS
git checkout feat/regen-extended-types-meos-idl
# Or stack on the wrappers branch for geodesic Haversine/PointToSegment too
git remote add wrappers https://github.com/estebanzimanyi/JMEOS.git
git fetch wrappers feat/spatial-haversine
git merge wrappers/feat/spatial-haversine # bring in PR #18's wrappersAfter that the full surface is available:
Stream-side platforms (MobilityFlink / MobilityKafka / MobilityNebula) can wire to this directly per the BerlinMOD-9 × 3 forms × 3 platforms parity contract — no blocker remains on the FFI side. Cross-verification this session ran todayA 50-class / 751-method codegen wedge ( |
…s + extended types + utils.spatial)
Updates the bundled `flink-processor/jar/JMEOS.jar` to a combined build
of JMEOS PR #19 (regen against MEOS-API meos-idl.json, 2,699 methods
including extended types) AND PR #18 (utils.spatial.Haversine +
utils.spatial.PointToSegment wrappers that MEOSBridge.java imports).
Surface delta vs the previous bundled jar:
- public static methods: 2 699 (was 1 685)
- utils.spatial.Haversine.distance(lon1, lat1, lon2, lat2) → double
- utils.spatial.PointToSegment.distance(pLon, pLat, s1Lon, s1Lat, s2Lon, s2Lat) → double
- tnpoint_ methods: 50
- tcbuffer / tpose / trgeo: now exposed
- sha: a5895c9b94… size: 1,210,863 B
Unblocks the MEOSBridge.java import path (line 116) — previously the
jar shipped PR #19's GeneratedFunctions but not PR #18's utils.spatial,
so base-branch mvn compile was RED. Both PRs now coalesced into a
single jar built by:
mvn -pl codegen,jmeos-core compile -Dmaven.test.skip=true
cd jmeos-core/target/classes && jar cf JMEOS.jar .
Unblocks codegen/flink-meos-ops wedge stacked on this branch.
estebanzimanyi
force-pushed
the
feat/jmeos-bridge-swap
branch
from
0a57c07 to
6676bbb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces
MEOSBridgeas the runtime spatial-predicate surface for all BerlinMOD-9 × 3-form streaming cells. The bridge calls into MEOS via JMEOS#15's 1.4 regen (geog_dwithinover WGS84 geographies) when libmeos is loadable, and falls back to the pure-JavaHaversine/SegmentDistanceutilities when it is not. The mini-cluster local-test drivers (BerlinMODQ*LocalTest) setmobilityflink.meos.enabled=falseatmain()entry so they stay green without libmeos.so on the runtime path.What lands
flink-processor/src/main/java/berlinmod/MEOSBridge.java— the single point of contact between the BerlinMOD scaffold and MEOS. SurfacesdwithinMetres,dwithinSegmentMetres,distanceMetres. Static init catchesUnsatisfiedLinkErrorand flipsMEOS_AVAILABLEto false on libmeos absence.MEOSBridgeinstead ofHaversine/SegmentDistancedirectly. The pure-Java classes remain as the fallback implementation, not as the primary route.geog_dwithin/geom_in/geom_to_geog/edwithin_tgeo_geo/nad_tgeo_geo/tpoint_length.meos_initialize(String, error_handler_fn)is gone in the regen, replaced by no-argmeos_initialize()+ separatemeos_initialize_timezone(String)/meos_initialize_error_handler(Pointer)entry points.aisdata/Main.javaandaisdata/TrajectoryWindowFunction.javaupdated accordingly.mobilityflink.meos.enabled=falseatmain()entry so the mini-cluster runs are green without libmeos.so present.target/build artefacts gitignored.Verified
mvn -q clean package -DskipTests— green on JDK 21.BerlinMODQ1LocalTest— FINISHED.BerlinMODQ3LocalTest— FINISHED (spatial-radius predicate).BerlinMODQ5LocalTest— FINISHED (pair-meeting predicate).BerlinMODQ8LocalTest— FINISHED (segment-distance predicate).All four exercise the
MEOSBridgefallback path. The MEOS-backed path is the same code with-Dmobilityflink.meos.enabled=true(default) and a loadable libmeos.so on the runtime classpath.Dependency
This PR depends on the JMEOS.jar artefact produced by JMEOS#15's regen branch. The PR sequencing is:
feat/berlinmod-q3-scaffold(PR feat(berlinmod): scaffold the full BerlinMOD-9 streaming-form parity matrix on MobilityFlink (27/27 cells) #3).Sibling work: MobilityKafka gets the same bridge swap shape against the Kafka-Streams Processor API in a separate stacked PR.
Out of scope
MEOSBridge.distanceMetres's internal implementation with a directgeog_distanceJMEOS call once that symbol is exposed (the predicate path is MEOS-backed; only the rarely-called raw-distance path stays Haversine for now).Stacked on: #3