Migrate document intelligence agent to Agent Bricks

[SatyKrish]

Summary

• replace the custom MLflow pyfunc agent runtime with Agent Bricks Knowledge Assistant and Supervisor Agent bootstrap
• enforce Databricks Apps OBO at the app boundary and remove legacy service-principal fallback/runtime files
• harden bootstrap ordering, SDK version floor, endpoint readiness waits, and CI steady-state preflight
• share Agent Bricks response normalization between app and CLEARS evals

Validation

• bash -n scripts/bootstrap-demo.sh
• .venv/bin/python -m py_compile agent/tools.py app/agent_bricks_response.py app/app.py app/lakebase_client.py evals/clears_eval.py scripts/bootstrap_agent_bricks.py scripts/wait_for_kpis.py samples/synthesize.py
• .venv/bin/python -m pytest agent/tests -q
• databricks bundle validate --strict -t demo
• git diff --check

[SatyKrish]

Deployment validation update from feat/agent-bricks-migration:

• Real demo bootstrap reached Agent Bricks successfully. Knowledge Assistant: doc-intel-knowledge-demo / ka-d1554ef3-endpoint; Supervisor: doc-intel-supervisor-demo / mas-dc6aba10-endpoint.
• Fixed live issues discovered during bring-up: generated endpoint name resolution, permissions PATCH by endpoint id, direct /serving-endpoints/{endpoint}/invocations call path, KA footnote citation extraction, SDK-native Vector Search index sync, and eval metric-name mapping for MLflow/databricks-agents 1.x.
• Live smoke passed against mas-dc6aba10-endpoint: ACME FY2024 revenue returned $94.2 billion with parsed ACME_10K_2024.pdf citation.
• Databricks App deploy is blocked by workspace/org config: 403 PERMISSION_DENIED because “Databricks Apps - user token passthrough feature is not enabled for organization 7474654018084939”. This is required by the mandatory OBO design.
• CLEARS eval completed but failed quality/latency thresholds. Latest run 772e902cab92459f9bf569296fc5f801: correctness 0.323, adherence 0.000, relevance/groundedness 0.516, safety 1.000, execution 1.000, latency p95 31711ms.

Local/static checks after fixes:

• bash -n scripts/bootstrap-demo.sh scripts/resolve-agent-endpoint.sh
• Python compile checks
• pytest agent/tests -q -> 8 passed
• databricks bundle validate --strict -t demo --var warehouse_id=... --var agent_endpoint_name=... -> OK
• git diff --check