feat(restheart-mongo): real Java line coverage via JaCoCo overlay

Replaces the prior API-route-surface "coverage" (counting fired
routes / curated route table) with actual JaCoCo line coverage
of the RESTHeart 9.x JVM under traffic.

Architecture:
  - `Dockerfile.coverage` is a multi-stage build: stage 1 (alpine)
    fetches JaCoCo 0.8.13 (jacocoagent.jar + jacococli.jar), stage
    2 layers them into the upstream restheart image (which is
    distroless — no shell, no curl, so jars must be pulled in a
    builder stage and COPY'd over).
  - `docker-compose.coverage.yml` is an OVERLAY: applied via `-f
    docker-compose.yml -f docker-compose.coverage.yml`. It sets
    JAVA_TOOL_OPTIONS=-javaagent:.../jacocoagent.jar=output=tcpserver,...
    so JaCoCo attaches at JVM start and listens on port 6300.
    The base `Dockerfile` and `docker-compose.yml` are untouched,
    so keploy/integrations and keploy/enterprise CI lanes consume
    the base compose and pay zero JaCoCo cost (the agent rewrites
    bytecode at class-load, adding ~5-10% per-call overhead that
    would slow record/replay).
  - `flow.sh::restheart_report_coverage` shells into a one-off
    coverage container to dump execution data via JaCoCo TCP and
    render an XML report against /opt/restheart/restheart.jar.
    When called against the base image (no overlay) it prints
    "INFO: ... uninstrumented" and exits 0 so enterprise lanes'
    `flow.sh coverage || true` informational calls keep working.

Also fixes a pre-existing config bug in the base
docker-compose.yml's RHO env var: the override syntax uses ';'
as a key->value separator (the upstream image's default
RHO uses ';'); the prior YAML-folded version used ',' which
RESTHeart parsed as part of the connection-string value, leading
RESTHeart to ignore the override and bind /http-listener/host to
its localhost default — making the HTTP listener unreachable
from the host port mapping. The base compose now uses ';' AND
explicitly overrides /http-listener/host -> "0.0.0.0".

Removed:
  - `restheart_list_routes` (curated route table denominator).
  - `restheart_list_recorded_routes` (keploy-tests / fired-routes
    reader).
  - The legacy route-surface `restheart_report_coverage` body.
  - `list-routes` subcommand.

Validated locally: helper produced `coverage=52.3` to
GITHUB_OUTPUT against a clean stack (1663/3182 lines covered
in restheart.jar; INSTRUCTION coverage 50.8%).

Signed-off-by: Akash Kumar <meakash7902@gmail.com>

Author: AkashKumar7902

.github/workflows/restheart-mongo.yml

@@ -194,4 +194,4 @@ jobs:
 
             Threshold: PR may not drop coverage by more than **${{ env.COVERAGE_THRESHOLD }}pp**. Override per-repo via the `RESTHEART_COVERAGE_THRESHOLD` actions variable.
 
-            Coverage measures the RESTHeart 9.x REST surface (`/{db}/{coll}` CRUD + `_aggrs/{name}` + `_size` + `_meta` + `_indexes` + `_streams/{name}` + `/graphql` + `/graphql/{appname}` + `/{db}/{coll}.files` + `/acl` + `/users` + `/tokens` + sessions/transactions + `/ic` + `/csv` + metrics + OAuth) that `flow.sh::restheart_record_traffic` exercises against the running backend. Reports are attached as artifacts on each job ("coverage-build" / "coverage-release").
+            Coverage is **Java line coverage** (JaCoCo 0.8.13) of the RESTHeart 9.x JVM under traffic — the bytecode `flow.sh::restheart_record_traffic` actually executes (REST CRUD + GraphQL + ACL + users + sessions/transactions + metrics + …). Instrumentation lives in a separate `Dockerfile.coverage` + `docker-compose.coverage.yml` overlay; the base `docker-compose.yml` consumed by keploy/integrations and keploy/enterprise CI lanes runs uninstrumented and pays zero JaCoCo cost. JaCoCo execution dumps + XML reports are attached as artifacts on each job (`coverage-build` / `coverage-release`).

.github/workflows/scripts/run-and-measure.sh

@@ -1,72 +1,43 @@
 #!/usr/bin/env bash
 #
-# run-and-measure.sh — bring restheart-mongo up via the sample's
-# compose, run flow.sh bootstrap + record-traffic with the
-# per-call audit log enabled, run flow.sh coverage, and emit
-# `coverage=PCT` onto $GITHUB_OUTPUT for the downstream
-# coverage-gate job.
+# run-and-measure.sh — bring restheart-mongo up under the
+# coverage overlay (JaCoCo agent attached via JAVA_TOOL_OPTIONS),
+# run flow.sh bootstrap + record-traffic, dump JaCoCo execution
+# data over the agent's TCP server, render a Java line-coverage
+# report, and emit `coverage=PCT` onto $GITHUB_OUTPUT for the
+# downstream coverage-gate job.
 #
-# Called from .github/workflows/restheart-mongo.yml's
-# build-coverage and release-coverage jobs (one per ref under
-# comparison). Both jobs source the same script so the
-# measurement is identical across refs — any drift in the
-# numerator definition would otherwise produce a misleading
-# delta.
+# Coverage isolation contract:
+#   * Base `Dockerfile` and `docker-compose.yml` are untouched.
+#   * The overlay `Dockerfile.coverage` + `docker-compose.coverage.yml`
+#     attach JaCoCo and expose its TCP server. ONLY this script
+#     applies the overlay; keploy/integrations and keploy/enterprise
+#     CI lanes consume the base compose and pay zero JVM-instrument
+#     cost (jacocoagent adds ~5-10% per-call overhead).
 #
-# Inputs (all from the workflow env):
-#   RESTHEART_FIRED_ROUTES_FILE — per-call audit log path; passed
-#                                 through to flow.sh so its
-#                                 record-traffic loop logs each
-#                                 (METHOD, URL) pair, and so its
-#                                 coverage subcommand uses that
-#                                 file as the standalone
-#                                 numerator.
-#   RESTHEART_PHASE             — label spliced into the project
-#                                 name so build vs. release runs
-#                                 don't collide on volume names
-#                                 (compose project naming inside
-#                                 the GH runner is per-job
-#                                 anyway, but RESTHEART_PHASE
-#                                 shows up in the test fixtures
-#                                 and is useful for diffing logs).
-#   GITHUB_OUTPUT               — standard GH Actions sink for
-#                                 step outputs.
+# Inputs (from the workflow env):
+#   RESTHEART_PHASE   — label for log diffing.
+#   GITHUB_OUTPUT     — standard GH Actions sink for step outputs.
 set -Eeuo pipefail
 
-# Compose-substituted variables. Defaults match the sample's
-# docker-compose.yml so a local invocation of this script (no
-# overrides) reproduces what CI runs.
 export RESTHEART_APP_CONTAINER="${RESTHEART_APP_CONTAINER:-restheart_app}"
 export RESTHEART_MONGO_CONTAINER="${RESTHEART_MONGO_CONTAINER:-restheart_mongo}"
 export RESTHEART_APP_PORT="${RESTHEART_APP_PORT:-8080}"
 export RESTHEART_MONGO_IP="${RESTHEART_MONGO_IP:-172.36.0.10}"
 export RESTHEART_NETWORK_SUBNET="${RESTHEART_NETWORK_SUBNET:-172.36.0.0/24}"
-
-# RESTHeart 9.x ships with admin/secret as the default
-# bootstrapped principal. flow.sh reads this header for every
-# call, so exporting it here keeps the standalone CI run aligned
-# with the keploy lanes (which pass the same value through).
 export RESTHEART_ADMIN_AUTH="${RESTHEART_ADMIN_AUTH:-Basic YWRtaW46c2VjcmV0}"
 
-: "${RESTHEART_FIRED_ROUTES_FILE:?RESTHEART_FIRED_ROUTES_FILE must be set by the workflow}"
+mkdir -p coverage
+chmod 777 coverage
+sudo rm -rf coverage/jacoco.exec coverage/report.xml coverage/coverage_report.txt 2>/dev/null \
+    || rm -rf coverage/jacoco.exec coverage/report.xml coverage/coverage_report.txt 2>/dev/null \
+    || true
 
-# Reset audit log for this run; otherwise a prior run's entries
-# would inflate the numerator on a re-trigger.
-: >"$RESTHEART_FIRED_ROUTES_FILE"
+COMPOSE=(docker compose -f docker-compose.yml -f docker-compose.coverage.yml)
 
-# Single-phase bootstrap: RESTHeart embeds its own admin
-# principal at first boot, so there's no separate "seed admin
-# user" stage the way doccano needs. compose up → wait for app
-# port → flow.sh bootstrap (PUTs the db + record-traffic's
-# collections) → flow.sh record-traffic → flow.sh coverage.
-docker compose up -d
+"${COMPOSE[@]}" up -d --build
 
-# Wait for the backend to start serving. Per the sample's
-# restheart_wait_for_app, both 200 AND 401 are success signals
-# — RESTHeart returns 401 on `/` until you authenticate, but
-# 401 still proves the HTTP listener and the auth filter are
-# both up. Anything before that (000 / connection refused) is
-# pre-listen.
+# Both 200 and 401 are success signals.
 for i in $(seq 1 120); do
     code=$(curl -sS -o /dev/null -w '%{http_code}' \
         "http://127.0.0.1:${RESTHEART_APP_PORT}/" 2>/dev/null || echo "")
@@ -80,33 +51,28 @@ if [ "$code" != "200" ] && [ "$code" != "401" ]; then
     docker logs "${RESTHEART_APP_CONTAINER}" --tail 200 2>&1 || true
     echo "----- mongo container logs -----"
     docker logs "${RESTHEART_MONGO_CONTAINER}" --tail 100 2>&1 || true
-    echo "----- docker compose ps -----"
-    docker compose ps || true
-    docker compose down -v --remove-orphans || true
+    "${COMPOSE[@]}" down -v --remove-orphans || true
     exit 1
 fi
 
 bash flow.sh bootstrap 240
-
-# Drive traffic. flow.sh::restheart_record_traffic gates on
-# restheart_wait_for_app internally, so this won't fire curls
-# at a half-booted backend.
 bash flow.sh record-traffic
 
-# Coverage report — uses RESTHEART_FIRED_ROUTES_FILE as numerator
-# since no keploy/test-set-* tree exists in the standalone case.
+# JaCoCo TCP-dump + report (no JVM stop needed).
 COVERAGE_REPORT_FILE="$PWD/coverage_report.txt" bash flow.sh coverage
 
-# Pull the percentage out of the report's `Covered N/M (XX.X%)`
-# line. Anchored on the parenthesised form so a future change to
-# the report's prose doesn't break the parse.
+if [ ! -f coverage_report.txt ]; then
+    echo "::error::flow.sh coverage produced no coverage_report.txt"
+    exit 1
+fi
+
 pct=$(grep -oE '\([0-9]+\.[0-9]+%\)' coverage_report.txt | head -1 | tr -d '()%')
 if [ -z "$pct" ]; then
     echo "::error::Could not parse coverage percentage from coverage_report.txt"
     cat coverage_report.txt || true
     exit 1
 fi
 echo "coverage=${pct}" >>"$GITHUB_OUTPUT"
-echo "coverage: ${pct}% (audit log: $RESTHEART_FIRED_ROUTES_FILE)"
+echo "coverage: ${pct}% (Java line coverage via JaCoCo)"
 
-docker compose down -v --remove-orphans
+"${COMPOSE[@]}" down -v --remove-orphans

restheart-mongo/.gitignore

@@ -0,0 +1,2 @@
+coverage/
+coverage_report.txt

restheart-mongo/Dockerfile.coverage

@@ -0,0 +1,43 @@
+# Coverage overlay image for restheart-mongo.
+#
+# Adds the JaCoCo agent (jacocoagent.jar) and CLI (jacococli.jar)
+# alongside the upstream restheart 9.2.1 image. The agent is
+# attached at JVM start via JAVA_TOOL_OPTIONS (set in
+# docker-compose.coverage.yml) so we don't have to rewrite the
+# upstream entrypoint, which is `java -jar restheart.jar` with
+# specific JVM flags.
+#
+# The agent runs in `tcpserver` mode so the workflow can dump
+# coverage data on demand without restarting the JVM —
+# important for distroless-style upstream images that don't
+# ship a shell.
+#
+# IMPORTANT: this image is only consumed by docker-compose.coverage.yml.
+# The base Dockerfile and docker-compose.yml stay uninstrumented so
+# enterprise's keploy compat lane pays no JVM-instrumentation cost
+# (jacocoagent adds ~5-10% per-call overhead through bytecode
+# rewriting, which would slow record/replay measurably).
+
+# Stage 1: pull JaCoCo zip in an alpine builder. The upstream
+# restheart image is distroless (no shell, no curl/unzip), so we
+# can't fetch JaCoCo from inside it.
+FROM alpine:3.19 AS jacoco-fetch
+ARG JACOCO_VERSION=0.8.13
+RUN apk add --no-cache curl ca-certificates unzip \
+    && curl -fsSL "https://repo1.maven.org/maven2/org/jacoco/jacoco/${JACOCO_VERSION}/jacoco-${JACOCO_VERSION}.zip" -o /tmp/jacoco.zip \
+    && mkdir -p /tmp/jacoco \
+    && unzip -j /tmp/jacoco.zip lib/jacocoagent.jar lib/jacococli.jar -d /tmp/jacoco
+
+# Stage 2: layer JaCoCo into the upstream image. We can't `RUN`
+# anything because the base image has no shell — only COPY and
+# WORKDIR work. COPY --chown sets ownership at copy time so the
+# distroless user (uid 65532) can read the agent.
+FROM softinstigate/restheart:9.2.1
+COPY --from=jacoco-fetch --chown=65532:65532 /tmp/jacoco/jacocoagent.jar /opt/jacoco/jacocoagent.jar
+COPY --from=jacoco-fetch --chown=65532:65532 /tmp/jacoco/jacococli.jar /opt/jacoco/jacococli.jar
+
+# Pre-create /coverage as an empty WORKDIR so docker has a
+# mountpoint for the bind-mount in docker-compose.coverage.yml.
+# WORKDIR doesn't require a shell.
+WORKDIR /coverage
+WORKDIR /opt/restheart

restheart-mongo/docker-compose.coverage.yml

@@ -0,0 +1,31 @@
+# Coverage overlay — applied with:
+#
+#   docker compose -f docker-compose.yml -f docker-compose.coverage.yml up -d --build
+#
+# Used ONLY by the standalone .github/workflows/restheart-mongo.yml
+# CI workflow. Keploy CI lanes (enterprise, integrations) ignore
+# this file and run the base compose unchanged, so they pay zero
+# JaCoCo-instrumentation cost.
+services:
+  restheart:
+    build:
+      context: .
+      dockerfile: Dockerfile.coverage
+    image: ${RESTHEART_COVERAGE_IMAGE:-restheart-mongo:local-coverage}
+    environment:
+      # Attach the JaCoCo agent in TCP server mode. The upstream
+      # entrypoint is `java ... -jar restheart.jar`; JAVA_TOOL_OPTIONS
+      # is read by the JVM and prepended to all java args, so the
+      # `-javaagent` flag arms before restheart.jar starts loading
+      # classes.
+      #
+      # output=tcpserver: the agent listens on port 6300 inside the
+      # container and dumps coverage data over TCP on demand. No
+      # need to stop the JVM to read coverage — the workflow
+      # connects to 6300, dumps, and the report is generated
+      # post-hoc by jacococli.
+      JAVA_TOOL_OPTIONS: "-javaagent:/opt/jacoco/jacocoagent.jar=output=tcpserver,address=0.0.0.0,port=6300,sessionid=keploy,append=false"
+    ports:
+      - "${RESTHEART_JACOCO_PORT:-6300}:6300"
+    volumes:
+      - ./coverage:/coverage

restheart-mongo/docker-compose.yml

@@ -12,9 +12,14 @@ services:
     ports:
       - "${RESTHEART_APP_PORT:-8080}:8080"
     environment:
-      RHO: >
-        /mclient/connection-string->"mongodb://${RESTHEART_MONGO_IP:-172.36.0.10}:27017",
-        /core/log-level->"INFO"
+      # RHO is RESTHeart's runtime config-override syntax:
+      #   key->value  pairs separated by ';'
+      # We override the default mongo URL (which the upstream image
+      # points at host.docker.internal — irrelevant in compose) AND
+      # explicitly bind /http-listener/host to 0.0.0.0; without that
+      # second override the upstream image binds to localhost and is
+      # unreachable from the host port mapping.
+      RHO: '/mclient/connection-string->"mongodb://${RESTHEART_MONGO_IP:-172.36.0.10}:27017";/http-listener/host->"0.0.0.0";/core/log-level->"INFO"'
     depends_on:
       mongo:
         condition: service_healthy