Merge Hongwei's PR OpenBankProject#2815 — adopt his dynamic-entity + resource-docs Lift cleanup

Adopts upstream PR OpenBankProject#2815 (Hongwei) into the local fork's develop. His PR
was opened against `aa21d891d` on upstream — before all of my recent work
landed on `origin/develop` — so we did the dynamic-entity Phase 2 work in
parallel. His implementation supersedes mine because:

  1. **Transaction atomicity.** He extracted `withBusinessDBTransaction`
     from `ResourceDocMiddleware` into `RequestScopeConnection`, and his
     `Http4sDynamicEntity` wraps POST/PUT/DELETE in it. My Phase 2 missed
     this — every DB op auto-committed, no request-scoped atomicity. Real
     bug, undetected by tests because each handler does a single DB op.
  2. **Cleaner Lift unregistration.** `OBPAPIDynamicEntity.routes = Nil`
     + `LiftRules.statelessDispatch.append` commented out + the Lift
     OBPEndpoint handlers in `APIMethodsDynamicEntity` commented (with a
     `/* DISABLED … */` block per the repo's convention). My version
     kept the Lift registration as a dormant fallback, which is safer
     during migration but messier as an end state.
  3. **Characterization tests.** `DynamicEntityFilterAndBankAccessTest`
     (275 lines) locks in GET-all `req.params` filtering (PARAM_LOCALE
     exclusion) + bank-level public/community access — both edge cases I
     hadn't covered. Written *before* his migration as a regression gate.
  4. **Resource-docs cleanup.** Three follow-up commits (aabebc0,
     35988b0, ca461f6) comment out the `ResourceDocs140..600` Lift
     dispatch handlers and restore minimal stubs for the entries
     `Http4sResourceDocs` still references. Independent of dynamic-entity
     but on the same Lift-removal track.

Conflict resolution (3 files, all "take theirs" with one Phase-3a exception):

  - `Http4sDynamicEntity.scala` (add/add) — accepted his 358-line file
    verbatim (vs my 488-line version).
  - `ErrorResponseConverter.scala` — accepted his `JsonResponseException`
    case (extracts `(message, code)` → `OBPErrorResponse` envelope). My
    `1e1047da1` `liftResponseToHttp4s` pass-through approach is superseded;
    his envelope-extraction works for every test that asserts on
    `(code, message)` (which is all of them today). Worth a follow-up to
    swap to `liftResponseToHttp4s` if a future test asserts on Lift-side
    response headers, but not blocking.
  - `Http4sApp.scala` — kept his `dynamicEntityRoutes` declaration
    (`Http4sDynamicEntity.wrappedRoutesDynamicEntity` — his entry-point
    name) **and** my `dynamicEndpointRoutes` declaration (Phase-3a
    dynamic-endpoint adapter, unrelated to his PR). De-duplicated the
    `.orElse(dynamicEntityRoutes.run(req))` wire-in that auto-merge
    inserted twice — kept it once at his position (right after v121).

Surviving from my fork (not affected by his PR):

  - `8dfdae9ae` — DirectLogin cleanup (dead `dlServe` block removal).
  - `ca4237e73` — MakerChecker TTL race resolution + stress regression
    guard.
  - `97c74e09e` — dynamic-endpoint Phase 3a (scoped Lift-Req shim).
  - `4c262e420` — migration-doc update for dynamic-entity (still accurate
    — the doc says Phase 2 is done; the *implementation* is now his).

Effectively superseded by his (no functional loss):

  - `ee05e701a` — my Phase 2 Http4sDynamicEntity (his version replaces it).
  - `1e1047da1` — my JsonResponseException case in ErrorResponseConverter
    (his version replaces it).

Verified: 23 suites / 151 tests green —

  ForceErrorValidationTest (the 5-fail regression — now green),
  JsonSchemaValidationTest (the 1-fail regression — now green),
  AuthenticationTypeValidationTest (same interceptor mechanism),
  v6 DynamicEntityFilterAndBankAccessTest (Hongwei's new tests),
  v6 DynamicEntityAccessFlagsTest, v4 DynamicendPointsTest,
  DynamicIntegrationTest, DynamicResourceDocTest, DynamicEndpointHelperTest,
  DynamicMessageDocTest, code.util.DynamicUtilTest,
  v4 MakerCheckerTransactionRequestTest (incl. my stress regression guard).

Once Hongwei's PR is also merged into upstream and origin/develop pulls
that in, the resulting state matches this merge exactly — this lets the
fork stay green while the upstream PR completes its review.

Author: constantine2nd

obp-api/src/main/scala/bootstrap/liftweb/Boot.scala

@@ -37,7 +37,7 @@ import code.accountattribute.MappedAccountAttribute
 import code.accountholders.MapperAccountHolders
 import code.actorsystem.ObpActorSystem
 import code.api.Constant._
-import code.api.ResourceDocs1_4_0.ResourceDocs300.{ResourceDocs310, ResourceDocs400, ResourceDocs500, ResourceDocs510, ResourceDocs600}
+//import code.api.ResourceDocs1_4_0.ResourceDocs300.{ResourceDocs310, ResourceDocs400, ResourceDocs500, ResourceDocs510, ResourceDocs600}
 import code.api.ResourceDocs1_4_0._
 import code.api._
 import code.api.attributedefinition.AttributeDefinition

obp-api/src/main/scala/code/api/ResourceDocs1_4_0/ResourceDocs140.scala

@@ -70,6 +70,11 @@ trait APIMethodsDynamicEntity {
       }
     }
 
+    /* DISABLED — DynamicEntity runtime CRUD migrated to code.api.dynamic.entity.Http4sDynamicEntity
+       (wired into Http4sApp.baseServices).  These Lift OBPEndpoint handlers are no longer registered
+       (OBPAPIDynamicEntity.routes = Nil and dynamic-entity removed from LiftRules.statelessDispatch).
+       Retained, commented out, for historical reference per the repo's revert-and-comment convention.
+
     lazy val genericEndpoint: OBPEndpoint = {
       case EntityName(bankId, entityName, id, isPersonalEntity) JsonGet req => { cc =>
         val listName = StringHelpers.snakify(entityName).replaceFirst("[-_]*$", "_list")
@@ -513,6 +518,8 @@ trait APIMethodsDynamicEntity {
         }
       }
     }
+    */
+    // end DISABLED handlers (migrated to Http4sDynamicEntity)
   }
 }
 

obp-api/src/main/scala/code/api/ResourceDocs1_4_0/ResourceDocsAPIMethods.scala

@@ -50,26 +50,29 @@ object OBPAPIDynamicEntity extends OBPRestHelper with MdcLoggable with Versioned
   // if old version ResourceDoc objects have the same name endpoint with new version, omit old version ResourceDoc.
   def allResourceDocs = collectResourceDocs(ImplementationsDynamicEntity.resourceDocs)
 
-  val routes : List[OBPEndpoint] = List(ImplementationsDynamicEntity.publicEndpoint, ImplementationsDynamicEntity.communityEndpoint, ImplementationsDynamicEntity.genericEndpoint)
+  // Runtime CRUD migrated to code.api.dynamic.entity.Http4sDynamicEntity (wired into
+  // Http4sApp.baseServices). routes reduced to Nil — the Lift OBPEndpoint handlers are no
+  // longer registered with Lift. This object is retained only as an accessor for
+  // allResourceDocs / routes referenced by ResourceDocsAPIMethods.getResourceDocsList.
+  val routes : List[OBPEndpoint] = Nil
+  // val routes : List[OBPEndpoint] = List(ImplementationsDynamicEntity.publicEndpoint, ImplementationsDynamicEntity.communityEndpoint, ImplementationsDynamicEntity.genericEndpoint)
+
+  // routes.map(endpoint => oauthServe(apiPrefix{endpoint}, None))  // no Lift dispatch registration — served by Http4sDynamicEntity
 
-  routes.map(endpoint => oauthServe(apiPrefix{endpoint}, None))
-  
   logger.info(s"version $version has been run! There are ${routes.length} routes.")
-  // specified response for OPTIONS request.
-  private val corsResponse: Box[LiftResponse] = Full{
-    val corsHeaders = List(
-      "Access-Control-Allow-Origin" -> "*",
-      "Access-Control-Allow-Methods" -> "GET, POST, OPTIONS, PUT, PATCH, DELETE",
-      "Access-Control-Allow-Headers" -> "*",
-      "Access-Control-Allow-Credentials" -> "true",
-      "Access-Control-Max-Age" -> "1728000" //Tell client that this pre-flight info is valid for 20 days
-    )
-    PlainTextResponse("", corsHeaders, HttpStatus.SC_NO_CONTENT)
-  }
-  /*
-   * process OPTIONS http request, just return no content and status is 204
-   */
-  this.serve({
-    case req if req.requestType.method == "OPTIONS" => corsResponse
-  })
+
+  // OPTIONS / CORS is handled by Http4sApp.corsHandler — the Lift OPTIONS serve below is disabled.
+  // private val corsResponse: Box[LiftResponse] = Full{
+  //   val corsHeaders = List(
+  //     "Access-Control-Allow-Origin" -> "*",
+  //     "Access-Control-Allow-Methods" -> "GET, POST, OPTIONS, PUT, PATCH, DELETE",
+  //     "Access-Control-Allow-Headers" -> "*",
+  //     "Access-Control-Allow-Credentials" -> "true",
+  //     "Access-Control-Max-Age" -> "1728000" //Tell client that this pre-flight info is valid for 20 days
+  //   )
+  //   PlainTextResponse("", corsHeaders, HttpStatus.SC_NO_CONTENT)
+  // }
+  // this.serve({
+  //   case req if req.requestType.method == "OPTIONS" => corsResponse
+  // })
 }

obp-api/src/main/scala/code/api/dynamic/entity/APIMethodsDynamicEntity.scala

@@ -2875,7 +2875,10 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{
         case ApiVersion.v5_1_0 => LiftRules.statelessDispatch.append(v5_1_0.OBPAPI5_1_0)
         case ApiVersion.v6_0_0 => LiftRules.statelessDispatch.append(v6_0_0.OBPAPI6_0_0)
         case ApiVersion.`dynamic-endpoint` => LiftRules.statelessDispatch.append(OBPAPIDynamicEndpoint)
-        case ApiVersion.`dynamic-entity` => LiftRules.statelessDispatch.append(OBPAPIDynamicEntity)
+        // dynamic-entity endpoints migrated to Http4sDynamicEntity (wired into Http4sApp.baseServices).
+        // Keep the case label with an empty body so ApiVersion.`dynamic-entity` does NOT fall through
+        // to the ScannedApiVersion branch below (which would re-append it via ScannedApis).
+        case ApiVersion.`dynamic-entity` => // LiftRules.statelessDispatch.append(OBPAPIDynamicEntity)
         case version: ScannedApiVersion => 
           ScannedApis.versionMapScannedApis.get(version).foreach(api => LiftRules.statelessDispatch.append(api))
         case _ => logger.info(s"There is no ${version.toString}")

obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala

@@ -2,6 +2,8 @@ package code.api.util.http4s
 
 import cats.effect._
 import code.api.APIFailureNewStyle
+import code.api.JsonResponseException
+import code.api.util.APIUtil.JsonResponseExtractor
 import code.api.util.ErrorMessages._
 import code.api.util.CallContext
 import net.liftweb.common.{Failure => LiftFailure}
@@ -73,30 +75,40 @@ object ErrorResponseConverter {
 
   /**
    * Convert any error to http4s Response[IO].
-   *
-   * `JsonResponseException` carries a fully-formed Lift `JsonResponse` and is the
-   * mechanism `APIUtil.anonymousAccess` (line 3517) and the after-authenticate
-   * interceptor chain (Force-Error, JSON-schema validation, etc. — `APIUtil.scala:5064`
-   * `afterAuthenticateInterceptors`) use to short-circuit a request with a synthesized
-   * response from inside a `Future`. The Lift bridge handles it with
-   * `case JsonResponseException(jsonResponse) => jsonResponse` in
-   * `Http4sLiftWebBridge.runLiftDispatch`; without an equivalent case here,
-   * EndpointHelpers-based handlers fall through to `unknownErrorToResponse` and
-   * return 500 where production has always returned the synthesized status (e.g.
-   * 400/403/444 for Force-Error scenarios). Reuse `liftResponseToHttp4s` so the
-   * conversion matches the bridge byte-for-byte.
    */
   def toHttp4sResponse(error: Throwable, callContext: CallContext): IO[Response[IO]] = {
     error match {
       case e: APIFailureNewStyle => apiFailureToResponse(e, callContext)
-      case e: code.api.JsonResponseException => Http4sLiftWebBridge.liftResponseToHttp4s(e.jsonResponse)
+      case JsonResponseException(jsonResponse) =>
+        // Force-Error / JSON-schema validation (APIUtil.afterAuthenticateInterceptResult, applied
+        // inside the auth/session-context chain) and dynamic-resource-doc permission errors
+        // (NewStyle) are raised as JsonResponseException carrying a Lift JsonResponse. Lift's
+        // OBPRestHelper catches these and returns the embedded response; mirror that here using
+        // the JsonResponse's own status code (no OBP-prefix remapping).
+        jsonResponse match {
+          case JsonResponseExtractor(message, code) => jsonErrorResponse(code, message, callContext)
+          case _                                    => unknownErrorToResponse(error, callContext)
+        }
       case _ =>
         tryExtractApiFailureFromExceptionMessage(error) match {
           case Some(apiFailure) => apiFailureToResponse(apiFailure, callContext)
           case None => unknownErrorToResponse(error, callContext)
         }
     }
   }
+
+  /** Build a JSON error response using the supplied status code verbatim (used for
+   *  JsonResponseException, whose embedded JsonResponse already carries the final code). */
+  private def jsonErrorResponse(code: Int, message: String, callContext: CallContext): IO[Response[IO]] = {
+    val errorJson = OBPErrorResponse(code, message)
+    val status = org.http4s.Status.fromInt(code).getOrElse(org.http4s.Status.BadRequest)
+    IO.pure(
+      Response[IO](status)
+        .withEntity(toJsonString(errorJson))
+        .withContentType(jsonContentType)
+        .putHeaders(org.http4s.Header.Raw(CIString("Correlation-Id"), callContext.correlationId))
+    )
+  }
 
   /** Old-style versions keep raw 400 codes — they never promote to 403/401/etc.
    *  Mirrors the same set used in ResourceDocMiddleware.authenticate.

obp-api/src/main/scala/code/api/dynamic/entity/OBPAPIDynamicEntity.scala

@@ -74,10 +74,9 @@ object Http4sApp {
   private val v510Routes: HttpRoutes[IO] = gate(ApiVersion.v5_1_0, code.api.v5_1_0.Http4s510.wrappedRoutesV510Services)
   private val v600Routes: HttpRoutes[IO] = gate(ApiVersion.v6_0_0, code.api.v6_0_0.Http4s600.wrappedRoutesV600Services)
   private val v700Routes: HttpRoutes[IO] = gate(ApiVersion.v7_0_0, code.api.v7_0_0.Http4s700.wrappedRoutesV700Services)
-  // Dynamic-entity data plane (operator-created entities under /obp/dynamic-entity/*).
-  // Native http4s replacement for the Lift OBPAPIDynamicEntity dispatch; gated by the
-  // same api_disabled_versions / api_enabled_versions machinery as the versioned routes.
-  private val dynamicEntityRoutes: HttpRoutes[IO] = gate(ApiVersion.`dynamic-entity`, code.api.dynamic.entity.Http4sDynamicEntity.routes)
+  // DynamicEntity runtime CRUD (/obp/dynamic-entity/*) — native http4s, replaces the Lift
+  // OBPAPIDynamicEntity dispatch. Hongwei's PR #2815 implementation.
+  private val dynamicEntityRoutes: HttpRoutes[IO] = gate(ApiVersion.`dynamic-entity`, code.api.dynamic.entity.Http4sDynamicEntity.wrappedRoutesDynamicEntity)
   // Dynamic-endpoint data plane (operator-created endpoints under /obp/dynamic-endpoint/*).
   // Phase-3a adapter: scoped Lift-Req shim that dispatches OBPAPIDynamicEndpoint directly,
   // keeping the runtime Scala codegen (which still emits Lift OBPEndpoint) unchanged.
@@ -135,9 +134,9 @@ object Http4sApp {
         .orElse(v140Routes.run(req))
         .orElse(v130Routes.run(req))
         .orElse(v121Routes.run(req))
+        .orElse(dynamicEntityRoutes.run(req))
         .orElse(code.api.DirectLoginRoutes.routes.run(req))
         .orElse(code.api.AliveCheckRoutes.routes.run(req))
-        .orElse(dynamicEntityRoutes.run(req))
         .orElse(dynamicEndpointRoutes.run(req))
         .orElse(Http4sLiftWebBridge.routes.run(req))
     }

obp-api/src/main/scala/code/api/util/APIUtil.scala

@@ -1,12 +1,14 @@
 package code.api.util.http4s
 
-import cats.effect.{IO, IOLocal}
+import cats.effect.{Deferred, IO, IOLocal, Outcome}
 import cats.effect.unsafe.IORuntime
 import com.alibaba.ttl.TransmittableThreadLocal
 import net.liftweb.common.{Box, Full}
 import net.liftweb.db.ConnectionManager
 import net.liftweb.util.ConnectionIdentifier
+import org.http4s.Response
 
+import code.api.util.APIUtil
 import code.util.Helper.MdcLoggable
 import java.lang.reflect.{InvocationHandler, Method, Proxy => JProxy}
 import java.sql.Connection
@@ -173,6 +175,51 @@ object RequestScopeConnection extends MdcLoggable {
       }
     }
 
+  /**
+   * Wrap an http4s route IO in a request-scoped DB transaction.
+   *
+   * Installs a once-only lazy connection-acquisition scope (requestLazyAcquire): no real
+   * connection is borrowed until the FIRST fromFuture call that touches the DB.  The first
+   * fiber to complete the inner Deferred wins; concurrent losers discard their connection
+   * and share the winner's proxy, so all fibers use one underlying Connection / one
+   * transaction.  On success: commit then close.  On error/cancel: rollback then close.
+   * If no DB call was made: nothing to commit or close (pool unaffected).
+   *
+   * GET/HEAD must NOT be wrapped (they run on auto-commit vendor connections).  Used by
+   * ResourceDocMiddleware (v6/v7) and by services that build their own request scope
+   * without the middleware (e.g. Http4sDynamicEntity).
+   */
+  def withBusinessDBTransaction(io: IO[Response[IO]]): IO[Response[IO]] =
+    Deferred[IO, (Connection, Connection)].flatMap { deferred =>
+      // acquireOnce: idempotent across concurrent callers via the Deferred.
+      val acquireOnce: IO[Connection] = for {
+        realConn <- IO.blocking(APIUtil.vendor.HikariDatasource.ds.getConnection())
+        _        <- IO.blocking { realConn.setAutoCommit(false) }
+        proxy    =  makeProxy(realConn)
+        ok       <- deferred.complete((realConn, proxy))
+        _        <- if (!ok) IO.blocking { try { realConn.close() } catch { case _: Exception => () } }
+                    else IO.unit
+        p        <- deferred.get.map(_._2)
+      } yield p
+
+      requestLazyAcquire.set(Some(acquireOnce)).bracket(_ =>
+        io.guaranteeCase { outcome =>
+          deferred.tryGet.flatMap {
+            case None => IO.unit   // no DB calls — pool unaffected
+            case Some((realConn, _)) =>
+              requestProxyLocal.set(None) *>
+                (outcome match {
+                  case Outcome.Succeeded(_) =>
+                    IO.blocking { realConn.commit() }
+                  case _ =>
+                    IO.blocking { try { realConn.rollback() } catch { case _: Exception => () } }
+                }) *>
+                IO.blocking { try { realConn.close() } catch { case _: Exception => () } }
+          }
+        }
+      )(_ => requestLazyAcquire.set(None))
+    }
+
   /**
    * Returns the proxy for the current fiber, acquiring it lazily on first call.
    *