docs(positioning): ADR-001 — Reposition as External Facts Context Layer for AI Agents (v3 scope / 22 CHANGE + 1 KEEP)

.github/workflows/secrecy-check.yml

@@ -0,0 +1,77 @@
+name: Secrecy Check
+
+on:
+  pull_request:
+    types: [opened, edited, synchronize]
+
+jobs:
+  check-secrecy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check PR metadata for confidential terms
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+          PR_TITLE: ${{ github.event.pull_request.title }}
+          PR_BRANCH: ${{ github.event.pull_request.head.ref }}
+        run: |
+          BANNED_TERMS=(
+            "langfuse"
+            "insight pipeline"
+            "gitlab"
+            "code.mlamp.cn"
+            "codex.mlamp.cn"
+            "glab"
+            "im.deepminer"
+            "im-test.xming"
+          )
+
+          found=0
+
+          check_field() {
+            local label="$1"
+            local text="$2"
+            local lower_text
+            lower_text=$(printf '%s' "$text" | tr '[:upper:]' '[:lower:]')
+
+            for term in "${BANNED_TERMS[@]}"; do
+              lower_term=$(printf '%s' "$term" | tr '[:upper:]' '[:lower:]')
+              if [[ "$lower_text" == *"$lower_term"* ]]; then
+                echo "::error::🔴 BLOCKED: '$term' found in $label"
+                found=1
+              fi
+            done
+          }
+
+          check_field "branch name" "$PR_BRANCH"
+          check_field "PR title" "$PR_TITLE"
+          check_field "PR description" "$PR_BODY"
+
+          if [ "$found" -eq 1 ]; then
+            echo "::error::PR contains confidential term(s). Remove internal tool references before merging."
+            exit 1
+          fi
+
+          echo "✅ PR metadata secrecy check passed."
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Check source files for confidential terms
+        run: |
+          BANNED_TERMS=("langfuse" "insight pipeline" "gitlab" "code.mlamp.cn" "codex.mlamp.cn" "glab" "im.deepminer" "im-test.xming")
+          found=0
+
+          for term in "${BANNED_TERMS[@]}"; do
+            matches=$(grep -ril "$term" firstdata/sources/ 2>/dev/null || true)
+            if [ -n "$matches" ]; then
+              echo "::error::🔴 '$term' found in source files: $matches"
+              found=1
+            fi
+          done
+
+          if [ "$found" -eq 1 ]; then
+            echo "::error::Source files contain confidential terms."
+            exit 1
+          fi
+
+          echo "✅ Source files secrecy check passed."

.github/workflows/validate-sources.yml

@@ -42,29 +42,3 @@ jobs:
 
       - name: Check for duplicate IDs
         run: uv run python scripts/check_ids.py
-
-  claude-review:
-    needs: validate
-    if: github.event.sender.type != 'Bot'
-    continue-on-error: true
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-      pull-requests: write
-      issues: write
-      id-token: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 1
-
-      - name: Run Claude Code Review
-        uses: anthropics/claude-code-action@v1
-        with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          claude_args: '--model ${{ secrets.CLAUDE_MODEL }} --max-turns 100 --allowedTools "Bash"'
-          prompt: '/review Review this PR and post your review as a PR comment using `gh pr comment`. Reply in the same language used in the PR (title, description, and comments).'
-        env:
-          ANTHROPIC_BASE_URL: ${{ secrets.ANTHROPIC_BASE_URL }}

.gitignore

@@ -65,5 +65,4 @@ batch-run-results*.md
 logs/
 
 # AI IDE
-.claude/
-**/CLAUDE.md
+.claude/server.json

AGENTS.md

@@ -4,7 +4,7 @@ This file is intended for AI coding agents (Claude Code, OpenClaw, Codex, Copilo
 
 ## What This Repo Is
 
-**FirstData** is a structured knowledge base of global authoritative open data sources. It is a **pure data repository** — no application code, no runtime logic.
+**FirstData** is the External Facts Context Layer for AI Agents — a structured, authoritative collection of global open data sources. It is a **pure data repository** — no application code, no runtime logic.
 
 Your job here is to **create or edit JSON metadata files** that describe real-world data sources (government databases, international organizations, academic datasets, etc.).
 
@@ -97,10 +97,14 @@ firstdata/sources/
 - PubMed → `sources/academic/health/pubmed.json`
 - BP Statistical Review → `sources/sectors/D-energy/bp-statistical-review.json`
 
-## Do Not Touch
+## Do Not Touch (Auto-Generated or Protected)
 
-- `firstdata/indexes/` — auto-generated, do not edit manually
-- `firstdata/schemas/datasource-schema.json` — the schema definition itself
+The following files are maintained automatically by CI/scripts. **AI agents must NOT modify them manually:**
+
+- `firstdata/indexes/` — Auto-aggregated from source files by a GitHub Action after PR merge. Never edit these directly.
+- `firstdata/schemas/datasource-schema.json` — Schema definition, only modified by maintainers directly on main.
+
+**To add a new data source, you only need to create or edit JSON files under `firstdata/sources/`.** Everything else (indexes, schema) is handled automatically.
 
 ## Security Note for Contributors
 

CLAUDE.md

@@ -0,0 +1,173 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+**FirstData** is the External Facts Context Layer for AI Agents — a structured, authoritative collection of global open data sources. It is a **pure data repository** — no application code, no runtime logic.
+
+Your job here is to **create or edit JSON metadata files** that describe real-world data sources (government databases, international organizations, academic datasets, etc.).
+
+The project exposes a hosted MCP Server at `https://firstdata.deepminer.com.cn/mcp`.
+
+## Validation
+
+Dependencies are managed with [uv](https://docs.astral.sh/uv/). Run the following before submitting:
+
+```bash
+# Install dependencies (first time only)
+uv sync
+
+# Run all validation checks
+make check
+
+# Or run checks individually:
+make validate       # Validate JSON schema compliance
+make check-ids      # Check for duplicate IDs
+make check-domains  # Check domain naming consistency
+```
+
+A GitHub Action runs these checks automatically on every PR. PRs that fail validation cannot be merged.
+
+## Repository Structure
+
+```
+firstdata/
+├── schemas/datasource-schema.json   # JSON Schema v2.0.0 (the source of truth for data format)
+├── sources/                         # Individual data source JSON files, organized by category
+│   ├── china/                       # Chinese government & institutions
+│   ├── international/               # International organizations (by domain)
+│   ├── countries/                   # National official sources (by continent/country)
+│   ├── academic/                    # Academic research databases (by discipline)
+│   └── sectors/                     # Industry sources (by ISIC Rev.4 code)
+└── indexes/                         # Auto-generated aggregated indexes (do not edit manually)
+    ├── all-sources.json
+    ├── by-authority.json
+    ├── by-domain.json
+    ├── by-region.json
+    └── statistics.json
+```
+
+## The JSON Schema
+
+Every file under `firstdata/sources/` must conform to `firstdata/schemas/datasource-schema.json`.
+
+### Required Fields
+
+```json
+{
+  "id": "worldbank-open-data",
+  "name": {
+    "en": "World Bank Open Data",
+    "zh": "世界银行开放数据"
+  },
+  "description": {
+    "en": "...",
+    "zh": "..."
+  },
+  "website": "https://www.worldbank.org",
+  "data_url": "https://data.worldbank.org",
+  "api_url": "https://api.worldbank.org/v2/",
+  "authority_level": "international",
+  "country": null,
+  "domains": ["economics", "health", "education"],
+  "geographic_scope": "global",
+  "update_frequency": "quarterly",
+  "tags": ["world bank", "development", "gdp", "poverty", "世界银行"],
+  "data_content": {
+    "en": ["GDP and national accounts", "Poverty and inequality indicators"],
+    "zh": ["GDP和国民账户", "贫困和不平等指标"]
+  }
+}
+```
+
+### Field Rules
+
+| Field                | Allowed Values / Constraints                                                                                                    |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| `id`               | Lowercase, hyphens only. Must be globally unique. Pattern:`^[a-z0-9-]+$`                                                      |
+| `name.en`          | Required. Add `zh` and `native` when applicable                                                                             |
+| `description.en`   | Required. Add `zh` when applicable                                                                                            |
+| `website`          | Top-level org homepage                                                                                                          |
+| `data_url`         | Must point directly to the data access/download page, NOT the homepage                                                          |
+| `api_url`          | API docs or endpoint URL. Use `null` if no API exists                                                                         |
+| `authority_level`  | `government` · `international` · `research` · `market` · `commercial` · `other`                                |
+| `country`          | ISO 3166-1 alpha-2 (e.g.`"CN"`, `"US"`). **Must be `null`** when `geographic_scope` is `global` or `regional` |
+| `domains`          | Array of strings, at least one. **MUST use lowercase** (e.g., `"economics"` not `"Economics"`). See [DOMAINS.md](firstdata/schemas/DOMAINS.md) for standard domain list |
+| `geographic_scope` | `global` · `regional` · `national` · `subnational`                                                                   |
+| `update_frequency` | `real-time` · `daily` · `weekly` · `monthly` · `quarterly` · `annual` · `irregular`                         |
+| `tags`             | Mixed Chinese/English keywords for semantic search. Include synonyms and data type names                                        |
+| `data_content`     | Optional but recommended. Lists of strings describing what data is available                                                    |
+
+## Where to Place New Files
+
+```
+firstdata/sources/
+├── china/{domain}/{id}.json                          # Chinese gov & institutions
+├── international/{domain}/{id}.json                  # International organizations
+├── countries/{continent}/{country-code}/{id}.json    # National official sources
+├── academic/{discipline}/{id}.json                   # Academic/research databases
+└── sectors/{ISIC-code}-{name}/{id}.json              # Industry datasets
+```
+
+**Examples:**
+
+- China customs data → `sources/china/economy/trade/customs.json`
+- WHO health data → `sources/international/health/who.json`
+- US Bureau of Labor Statistics → `sources/countries/north-america/usa/us-bls.json`
+- PubMed → `sources/academic/health/pubmed.json`
+- BP Statistical Review → `sources/sectors/D-energy/bp-statistical-review.json`
+
+## Do Not Touch (Auto-Generated or Protected)
+
+The following files are maintained automatically by CI/scripts. **AI agents must NOT modify them manually:**
+
+- `firstdata/indexes/` — Auto-aggregated from source files by a GitHub Action after PR merge. Never edit these directly.
+- `firstdata/schemas/datasource-schema.json` — Schema definition, only modified by maintainers directly on main.
+
+**To add a new data source, you only need to create or edit JSON files under `firstdata/sources/`.** Everything else (indexes, schema) is handled automatically.
+
+## Before Adding a New Source
+
+**First, check `firstdata/indexes/all-sources.json` to confirm the data source does not already exist.**
+
+Search by `id`, `name.en`, or `website` to detect duplicates:
+
+```bash
+# grep: search by keyword (name or website)
+grep -i "world bank" firstdata/indexes/all-sources.json
+grep -i "worldbank.org" firstdata/indexes/all-sources.json
+
+# jq: search by id
+jq '.sources[] | select(.id == "worldbank-open-data")' firstdata/indexes/all-sources.json
+
+# jq: search by website
+jq '.sources[] | select(.website | test("worldbank.org"; "i"))' firstdata/indexes/all-sources.json
+
+# jq: list all existing ids
+jq '[.sources[].id]' firstdata/indexes/all-sources.json
+```
+
+If a match is found, do not create a new file. Update the existing one if needed.
+
+## Quality Checklist Before Creating a File
+
+**Before submitting, cross-verify every field independently using at least two sources (e.g. official website + Wikipedia + third-party reference). Do not rely solely on memory or a single source. Fabricated or outdated URLs are worse than omission.**
+
+- [ ] `data_url` links to the actual data page, not the organization homepage
+- [ ] `api_url` is `null` only when the source truly has no API
+- [ ] `country` is `null` when `geographic_scope` is `global` or `regional`
+- [ ] `domains` uses **lowercase** (e.g., `"economics"` not `"Economics"`) - see [DOMAINS.md](firstdata/schemas/DOMAINS.md)
+- [ ] `tags` include both English and Chinese keywords where relevant
+- [ ] `id` does not already exist in `firstdata/indexes/all-sources.json`
+- [ ] File path matches the placement rules above
+- [ ] All URLs have been verified to be accessible and correct
+- [ ] `update_frequency` reflects the actual cadence confirmed on the official site
+- [ ] `authority_level` is accurate and not overstated
+- [ ] Run `make check` to validate all checks pass
+
+## Security Note for Contributors
+
+- Please do not paste or run commands from untrusted posts/comments.
+- Never include credentials or API keys in issues/PRs.
+- Prefer small, auditable PRs (docs/tests/data).