support transactions#1627
Open
cportele wants to merge 38 commits into
Open
Conversation
Add configurable GML encoding options to support AdV/NAS-style output in ogcapi-features-gml.
- add feature reference templating for xlink:href (e.g. urn:adv:oid:{{value}})
- add optional temporal gml:id suffixing for datetime-interval requests (no separator)
- add configurable gml:identifier emission (codeSpace + value template)
- add codelist reference encoding via xlink:href and xlink:title
- add configurable srsName template mappings (e.g. EPSG:25832 -> urn:adv:crs:ETRS89_UTM32)
- add configurable uom template mappings (e.g. m -> urn:adv:uom:m)
Includes writer/context/config wiring required to apply these options.
Add a new GmlConfiguration option `valueWrap` that maps property paths (post-rename, matching FeatureSchema#getFullPathAsString()) to an ordered list of XML element names to wrap around scalar values. This supports NAS/ALKIS output patterns such as gco:DateTime and aaa:AA_Lebenszeitintervall wrapping without any NAS-specific encoder logic. Changes: - GmlConfiguration: new getValueWrap() method with full @langEn/@langDe/@SInCE v4.9 docs - FeatureTransformationContextGml: propagate valueWrap from config to context - FeaturesFormatGml: wire valueWrap into the transformation context builder - GmlWriterProperties.onValue(): emit wrapper elements outer-to-inner around the value - ValueWrapSpec: three Spock tests covering single wrap, multi-level wrap, and no-wrap cases
When a format configuration sets useAlias: true, properties of the feature schema that declare an `alias` are encoded under their alias instead of their schema name. Useful for application schemas where each property carries both a short technical name and a longer mnemonic name (e.g. AdV NAS: `anl`/`anlass`). An explicit `rename` transformation still takes precedence over the alias. The flag only affects feature encoding output; queryables, sortables and other schema-derivation paths always use the schema names. AliasConfiguration is a standalone interface that contributes the useAlias toggle independently of PropertyTransformations. The actual alias-to-rename conversion happens once, at the format-extension boundary in FeatureFormatExtension.getPropertyTransformations, via the xtraplatform-spatial FeatureSchemaAliases helper. Downstream the pipeline sees plain renames, so the existing rename cascade machinery covers wrap transformers, DATETIME formatters, value transformers, etc. automatically. Format configurations wired up: GML, GeoJSON, CityJSON, SfFlat (CSV, FlatGeobuf, Tiles inherit it). JSON-FG composes on top of GeoJSON and inherits the alias behavior without declaring its own option. Includes a NAS-oriented example in GmlConfiguration's class-level JavaDoc demonstrating useAlias alongside the other GML options typically configured for the AAA-NAS profile.
Property elements are now placed in the namespace of their containing object type (declared via objectTypeNamespaces), matching the XML Schema convention elementFormDefault="qualified". This removes the need to put namespace prefixes into the provider schema's `alias` field, which leaked into other encodings (e.g. GeoJSON saw `gmd:processStep` as a property name). Explicit `prefix:name` in the schema name or alias still takes precedence over the inherited namespace.
Adds GML as a supported request content type for POST/PUT on the CRUD
building block and refactors validation so each FeatureFormatExtension owns
its own validation against a format-agnostic ValidatorContext. Resurrects
the two live smoke specs against Groovy 4.
CRUD endpoint and command handler:
* EndpointCrud now passes the request MediaType through to the command
handler (replacing the boolean jsonFg flag) and forwards the Link
header list for downstream profile parsing.
* CommandHandlerCrud / CommandHandlerCrudImpl: replace the boolean jsonFg
field with a MediaType contentType and a linkHeaders list. Drop the
in-handler JSON-Schema validation pipeline (and its codelist/schema
cache wiring); validation is dispatched to the resolved
FeatureFormatExtension.validate(content, ValidatorContext). Build the
ValidatorContext from request material (api/collection/media type,
request context, profiles parsed from Link headers, default
schema-validation profiles) and pass it in. SchemaCacheCrud is moved
out of the CRUD module.
features-core domain:
* DecoderContext now exposes a jakarta MediaType (was ApiMediaType) -
the decoder only needs the parsed media type, not the API-level
wrapper.
* FeatureFormatExtension grows a default validate(content,
ValidatorContext) that formats override.
* New ValidatorContext immutable: apiData, collectionId, mediaType,
type (RETURNABLES | RECEIVABLES), requestContext, declaredProfiles
(parsed from Link headers), defaultProfiles. Format-agnostic so
GeoJSON and GML share the same call shape.
GeoJSON format:
* Adds xtraplatform-jsonschema as a provided dependency so the schema
cache types are visible at compile time.
* Pulls the JSON-Schema-driven validation up from CommandHandlerCrudImpl
into FeaturesFormatGeoJson.validate: resolves the right validation
profile (returnables/receivables x geojson/jsonfg) from declared
profiles, derives a schema via the appropriate cache, and rejects
invalid bodies with IllegalArgumentException ("feature mutation is
rejected").
* Splits the relocated schema cache by scope: new
ReceivablesJsonSchemaCache (renamed from SchemaCacheCrud) and new
ReturnablesJsonSchemaCache.
GML format:
* FeaturesFormatGml.validate implements XSD validation: composes a
Schema from all schemaLocations on the GmlConfiguration and runs the
body through a Validator. Missing/unparseable schemas log a warning
and skip validation rather than failing the request.
* Switches the decoder factory from FeatureTokenDecoderGmlForSql to
FeatureTokenDecoderGml (the SQL-targeted variant is no longer
appropriate for the CRUD input path).
* Drops featureCollectionElementName / featureMemberElementName from
the input profile - bare-feature only on input; collection-level
names are encoder concerns.
* toInputProfile becomes package-private so the new
ToInputProfileSpec.groovy can lock the GmlConfiguration ->
input-profile mapping (list-to-map conversions for srsName/uom and
per-entry direction reversal for variableObjectElementNames).
* FeatureTransformationContextGml gains a rootElementWritten flag so
endDocument() is a no-op on a stream that never opened a root. A
single-feature item GET on a missing id previously left the wstx
writer with no root and surfaced 500; with this change the stream
completes cleanly and failIfNoFeatures turns the empty result into
a 404.
* GmlConfiguration JavaDoc gains a section listing which options are
honoured on the decoder side and which remain output-only, with the
rationale (no input counterpart, or permissive decoder).
Live smoke specs (ogcapi-crud):
* Drops the unmaintained, Groovy-2-era http-builder 0.7 testProvided
dep from ogcapi-crud/build.gradle.
* Rewrites TransactionalRESTApiSpec (GeoJSON) and adds
TransactionalGmlRESTApiSpec (GML) on java.net.http.HttpClient so
both work under Groovy 4. Env-var driven (SUT_URL, SUT_PATH,
SUT_COLLECTION, SUT_BODY_FILE; optional SUT_ID, SUT_CONTENT_CRS,
SUT_FRESH_ID), self-adapts to the collection's
supportsNonAutogeneratedResourceIds (POST vs PUT-fresh-id mode), and
cleans up the fresh-id resource after every iteration and in
cleanupSpec. Covers create->201, PUT-replace->2xx, PATCH with wrong
content type->415, multi-feature body->400, create->GET round-trip;
the GML spec also covers single-feature mixed-CRS->400.
New ogcapi-transactions building block (draft) exposing a service-level POST /transactions endpoint that executes Insert, Replace, Update, and Delete actions against a FeatureProvider as either an atomic multi-action transaction or an independent-per-action batch. Conforms to OGC API Features Part 11: Atomic and Batch Transactions conformance classes - except asynchronous transactions; both modes are individually toggleable in TransactionsConfiguration. - WfsTransactionParser: streaming parser for WFS 2.0 wfs:Transaction XML payloads. wfs:Insert children are bundled per consecutive-same- collection group into one TxInsert with pre-buffered payloads exposed via items(); wfs:Replace buffers one feature, wfs:Update and wfs:Delete parse metadata only. Filters are restricted to fes:ResourceId/@Rid; other predicates are a hard error. - JsonTransactionParser: parser for the OGC API JSON transaction action format, mirroring the same TxAction model. - TransactionExecutorImpl: * Atomic mode opens one Session per FeatureProvider, runs every action through it, and commits at the end; any action failure rolls the whole transaction back. * Batch mode opens a fresh Session per action — each commits or rolls back independently, and the response carries per-action status. * Insert path accumulates decoded per-feature FeatureTokenSources up to INSERT_BATCH_SIZE=100 and calls the multi-source Session.createFeatures overload so the SQL session can fold consecutive same-shape main INSERTs into one multi-row INSERT. * Update is GET-merge-write: fetch the current feature as GeoJSON, apply an RFC 7396 JSON Merge Patch, and write the full merged document through Session.updateFeature(..., partial=true). Above BULK_GET_THRESHOLD=16 target ids, per-target GETs are replaced by a single FEATURES query with an IN-filter served from an in-memory map. * Same-transaction chaining (an Update targeting an id touched earlier in the same atomic transaction) is rejected up front because the GET goes through the provider's query pool at READ COMMITTED and cannot see the session's uncommitted writes. A same-connection read would lift this; future work. - Domain model: Transaction, TxAction sealed hierarchy (TxInsert, TxReplace, TxUpdate, TxDelete), TxActionType, TxSemantic, ActionResult, ActionStatus, ExecutionResult. - Tests: WfsTransactionParserSpec and JsonTransactionParserSpec for the two parsers, TransactionalWfsRESTApiSpec for end-to-end behaviour, plus ALKIS NAS sample features under test resources. Picks up the multi-action SQL mutation Session added in xtraplatform-spatial.
Parse-time feature identity (gml:id for GML, the id member for GeoJSON) and the 1-based position within wfs:Insert / items[] are now captured in a new InsertItem and surfaced as Iterator<InsertItem> from TxInsert.items(). When session.createFeatures rejects a batch, the executor builds a FAILED ActionResult that lists every candidate feature id and index in the failing batch. EndpointTransactions now returns the OGC API Features Part 11 Transaction Response shape on both success and atomic failure, with an exceptions[] array of RFC 7807 problem objects. Each entry carries type, title, status, detail and the Part 11 extension members collectionId, action, actionId, featureIds, featureIndexes. Atomic failures previously responded with a flat problem+json envelope whose detail string referenced only the collection — callers could neither parse it as a Transaction Response nor identify the offending feature.
Vue 3's template parser rejected the generated transactions.html.vue with "Element is missing end tag" because the docs processor inlined <ul>/<li> tags from the JavaDoc without closing </li>. Switch to the <p><code>...</code> + markdown-list pattern used by GltfBuildingBlock, which renders as proper markdown lists in the output.
Extract Prefer header parsing from EndpointTransactions into a
package-private PreferHeader helper. Recognised tokens:
Prefer: respond-async -> 501 Not Implemented (no body parse)
Prefer: return=representation -> full body (default)
Prefer: return=minimal -> summary only, per-action arrays omitted
Prefer: return=none -> 204 No Content on success
(full body still returned on failure
so exceptions can be reported)
Every non-501 response now echoes Preference-Applied: return=<value>.
Declare the header on the OpenAPI operation by introducing
HeaderPreferTransaction (mirrors HeaderPreferFeature for CRUD) and
wiring getHeaders(...) into EndpointTransactions.computeDefinition.
The schema lists the four atomic Prefer values with
return=representation as the default.
Bug fix in TransactionExecutorImpl.runDelete: only count features as
deleted when the underlying SQL DELETE actually matched a row.
Previously every rid in the filter was reported as deleted even if no
feature existed, so totalDeleted and deleteResults overstated the
result and made delete responses unreliable for clients.
Tests:
- PreferHeaderSpec: 24 cases covering parseReturn fallback, mixed-case,
multi-token / multi-header forms, unknown values, and
containsPreferToken substring rejection.
- TransactionalWfsRESTApiSpec: four new phases exercise Prefer over
wfs:Replace and assert both response body shape and the
Preference-Applied header. Existing phase 3 updated to lock the now
working wfs:Update path (200 + totalUpdated == 1).
- Make httpClient @shared so cleanupSpec can reliably delete leftover
test features.
* PreferHeader: parse handling=strict|lenient (default lenient).
* TransactionParser: new validateEnvelope hook; EndpointTransactions
buffers the request body and calls it before parsing when strict.
IllegalArgumentException from the hook maps to 400.
* JsonTransactionParser: validate the envelope against a bundled JSON
Schema derived from the draft OGC API Features Part 11 schemas; the external GeoJSON ref and cql2.yaml ref are
replaced with {type: object} at bundle time (per-feature schemas
are checked separately by the format validator below), links are not supported.
* TransactionExecutor: thread the strict flag through execute().
TransactionExecutorImpl validates each insert item and replace
payload via FeatureFormatExtension.validate before any provider
write. Atomic transactions abort on the first invalid feature
(existing rollback fires). Batch transactions skip invalid items
and continue.
* ActionResult.failedFeatureErrors: one validation message per
rejected feature, parallel to failedFeatureIds/Indexes.
EndpointTransactions.renderBody emits one exceptions[] entry per
rejected feature carrying that feature's own message in detail.
* HeaderPreferTransaction: declare handling=strict|lenient in the
Prefer header schema and document them.
* TransactionalWfsRESTApiSpec: new phases 3e-3i drive strict mode
end-to-end (WFS-valid happy path, JSON envelope accept/reject,
JSON insert with all items invalid, JSON insert with one valid
plus two invalid items).
Rename TransactionalWfsRESTApiSpec to TransactionalRESTApiSpec — the
spec already exercises both wfs:Transaction XML and ogc-tx+json
envelopes — and add five JSON-envelope phases against the running NAS
API:
* phase 5 (Req 22): insert with Content-Crs = storage CRS, coords
round-trip unchanged.
* phase 6 (Req 22 contrast): insert with Content-Crs = CRS84 and
CRS84 coords; storage-CRS GET-back bbox matches the reference
feature within sub-meter reprojection error.
* phase 7 (Req 23): same body as phase 6 but no Content-Crs header;
result matches phase 6, proving the default is CRS84.
* phase 8 (Req 24B): Content-Crs declaring an unsupported CRS is
rejected with 4xx, and a follow-up GET 404s.
* phase 9 (Req 27): GET surfaces the primary-geometry property as the
top-level "geometry" member, not as a properties entry. Reads the
JSON Schema to find the property name so it survives renames.
New helpers: postJson (controllable Content-Crs), bbox / approx
comparison from GeoJSON coordinates, jsonInsertEnvelope, and
fetchReceivableFeatureInCrs. Phase 8's rejected id is included in the
cleanup delete filter (idempotent if absent); phase 4 asserts an exact
deleted count, so a regression that lets phase 8 insert would fail
loudly there too.
Pull request-header parsing into ApiHeader classes: - HeaderContentCrsTransaction.parse() handles header parsing, default-CRS lookup, and CRS-supported validation. - HeaderPreferTransaction absorbs the former PreferHeader helpers and exposes PreferReturn / PreferHandling enums with static parseReturn / parseHandling / containsToken methods. PreferHeader[.Spec] removed. Split the endpoint into a JAX-RS shell plus a CommandHandlerTransactions / Impl that owns transaction execution. The transaction body is no longer buffered eagerly; envelope validation moves out of the TransactionParser interface (default no-op removed) into the JSON handler path. Derive the OpenAPI request schema for application/ogc-tx+json from the bundled transaction-envelope.json instead of an empty ObjectSchema(): - top-level allOf becomes the content schema - $defs entries lift into referencedSchemas with an ogc-tx- prefix - #/$defs/X refs are rewritten to #/components/schemas/ogc-tx-X - JSON-Schema-2020-12 type arrays are normalised for OpenAPI 3.0 (type: [X, null] -> nullable: true; otherwise oneOf of single types) EndpointTransactionsEnvelopeSchemaSpec locks the lift / rewrite / cache behaviour. Envelope schema tidy-up: DRY the action-metadata block via a $defs reference and tighten the feature shape (type / properties / geometry required). Bump @SInCE v4.5 -> v4.8 on TransactionsConfiguration option docs. Modernise TransactionExecutorImpl#runAction to a Java switch expression.
Per RFC 7240, servers should echo the preferences that were both understood and honored. The /transactions handler was only emitting `return=…` and silently dropping the `handling` preference, so clients had no way to confirm that strict per-feature validation was actually applied. Now `Preference-Applied` carries `handling=strict` alongside the return preference whenever strict was applied; lenient (the default) is omitted.
FeaturesFormatGml.validate() was a silent no-op for several overlapping reasons: - The JAXP SchemaFactory was given `http://schemas.opengis.net/...` URLs that fail to resolve. - The top-level StreamSource carried only a bare filename, so relative imports inside the loaded XSDs could not resolve to a base URL. - The validator received the inline XML as a systemId, so JAXP tried to parse it as a URL. - TransactionExecutorImpl built the ValidatorContext with the WFS-form collection id (mixed case) instead of the canonical lowercase id, so the lookup of GmlConfiguration on the FeatureTypeConfiguration failed. Fixes: - Install an LSResourceResolver on the SchemaFactory that upgrades any http:// systemId to https://, and pre-upgrade configured top-level schema URLs the same way. - Pass the full URL as the StreamSource systemId so relative imports resolve against it. - Wrap content in a StringReader so it is parsed as inline XML. - Cache the compiled Schema per (api hashCode, collectionId) à la JsonSchemaCache; each call only creates a fresh Validator. - Canonicalize the collection id inside TransactionExecutorImpl.buildValidatorContext.
Per-feature JSON validation re-serialized the cached JsonSchemaDocument to a String, re-parsed it to a JsonNode, and re-compiled a networknt JsonSchema. This dominated runtime for large /transactions bodies in strict mode, even though the underlying JsonSchemaDocument is already cached via JsonSchemaCache. - Add a CompiledJsonSchema interface in ogcapi-foundation/domain as an opaque, thread-safe handle. - Extend SchemaValidator with compile(String) and validate(CompiledJsonSchema, String). SchemaValidatorImpl wraps networknt's JsonSchema in a private impl and uses a static ObjectMapper instead of allocating one per call. - FeaturesFormatGeoJson caches CompiledJsonSchema per (apiData.hashCode(), collectionId, validator type, jsonFg flag), mirroring the JsonSchemaCache structure. Each feature now only parses its own JSON body and runs the validator.
Move the generic RFC 7240 Prefer-header parsing into HeaderPrefer (handling=… and return=… enums, parseHandling, parseReturn, containsToken, parseParameterised) so all endpoints share one implementation. The engine collapses ambiguous mutually-exclusive values to the caller-supplied fallback per RFC 7240 §2. Drop the ad-hoc Endpoint.strictHandling regex helper and migrate the six callers in CRUD, Styles, Search, Resources, and Transactions.
Move the shared schema-building, header metadata, and request-side parser into a new abstract HeaderContentCrs in the crs module's domain package. The three concrete classes (Features for read, Crud and Transaction for write) now keep only the per-building-block bits: id, description, applicability, configuration, and spec maturity. Rename the existing crs-module class to HeaderContentCrsFeatures to match the sibling naming pattern and free up the HeaderContentCrs name for the abstract base.
…ingBlock Extend @scopeEn/@scopeDe with three paragraphs covering the request headers the endpoint honours and the Part 11 Transaction Response shape returned on both success and atomic failure. These were previously only documented per-header in the OpenAPI output and per-condition in the spec, leaving config authors without a single place to learn what the endpoint accepts and returns.
Collaborator
|
Surge PR preview deployment succeeded. View it at https://ldproxy-ldproxy-pr-1627.surge.sh |
Both the JSON and WFS transaction parsers now merge consecutive identity-free inserts targeting the same collection into one TxInsert. This lets the executor's INSERT_BATCH_SIZE path cover all features from a run of same-type wfs:Insert siblings (or same-collection JSON insert actions) in one Session.createFeatures call rather than N individual calls. Coalescing is skipped when an action carries id/title/description (JSON) or handle (WFS) so that per-action ActionResult mapping is preserved in the response for callers that set those fields.
FeaturesFormatGml.getFeatureDecoder() rebuilt SchemaMapping per feature via SchemaMapping.of(featureSchema). For schemas with many concat branches this dominated the pre-SQL phase of wfs:Transaction inserts (ImmutableSchemaMapping init's getParentSchemasBySourcePath walks the branches with streams), inflating per-feature time by ~100-150x for the heaviest types versus narrow schemas. Add a schemaMappingCache keyed by (apiData.hashCode(), collectionId), mirroring the existing schemaCache for the XSD validation Schema. The hash-based key invalidates automatically when entity config reloads.
The executor now dispatches featureProvider.changes().handle(...) after each successful transaction (atomic post-commit, batch post-loop), mirroring CommandHandlerCrudImpl. Successful actions are aggregated per (collection, mapped action) into one FeatureChange per group with the union of feature ids, new bounding boxes, and new intervals — so a batch of N same-collection inserts produces one CREATE event, not N. Without this, itemCount, spatial/temporal extent, and lastModified on the collection stayed stale after a transaction. The new TransactionalRESTApiSpec phases A0/A1/A2 round-trip the metadata end-to-end against a live ldproxy.
Collaborator
Author
|
I added more updates after trying to load ALKIS Grundriss data from Bonn (~3 million features, ~25 minutes) and Wiesbaden (~2 million features, ~15 minutes) via transactions. After these fixes, all transactions complete successfully. The auth-check of the body has been disabled for the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request checklist
Changes introduced by this PR
Closes #1623.
Depends on ldproxy/xtraplatform-spatial#512 and #1622.
Performance test with inserting 10 000 AX_Flurstueck features:
handling=strictapplication/ogc-tx+jsonwith GeoJSONapplication/xml(wfs:Transaction with GML)