concrnt-crawler

Concrnt public records crawler + Meilisearch search API.

Seed Concrnt server から known servers を取得し、各 server の net.concrnt.core.query で users / communities を収集して Meilisearch に投入します。

Run

Local compose:

docker compose up --build

Local Go:

CONCRNT_CRAWLER_CONFIG=config.local.yaml go run .

config.local.yaml は git ignore 済みです。

Config

Config path:

1. CONCRNT_CRAWLER_CONFIG
2. config.local.yaml
3. config.yaml
4. /etc/concrnt-crawler/config.yaml

Minimal example:

server:
  listen: ":8080"
  publicURL: "http://localhost:8080"

crawl:
  seed: "ariake.concrnt.net"
  layer: "concrnt-mainnet"
  prefix: "cckv://"
  knownServersInterval: "10m"
  incrementalInterval: "15m"
  requestTimeout: "15s"
  globalConcurrency: 2
  perServerConcurrency: 1
  pageLimit: 100
  overlap: "2m"
  maxPagesPerRun: 20
  profileSchemas:
    - "https://schema.concrnt.world/p/main.json"
  communitySchemas:
    - "https://schema.concrnt.world/t/community.json"

backends:
  postgresDsn: "postgres://concrnt_crawler:password@db:5432/concrnt_crawler?sslmode=disable"
  meiliHost: "http://meilisearch:7700"
  meiliAPIKey: ""

observability:
  enableTrace: false
  traceEndpoint: ""

API

Health

GET /health

{"status":"ok"}

Search Users

GET /api/v1/search/users?q=alice&limit=20&offset=0&sourceServer=example.net&owner=con...

Search Communities

GET /api/v1/search/communities?q=general&limit=20&offset=0&sourceServer=example.net&owner=con...

Search Servers

GET /api/v1/search/servers?q=ariake&limit=20&offset=0&status=active

Search response:

{
  "hits": [],
  "query": "alice",
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 0,
  "processingTimeMs": 0
}

Stats

GET /api/v1/stats

Returns DB crawler counts and Meilisearch stats.

Crawl CCFS

POST /api/v1/crawl/ccfs
Content-Type: application/json

{
  "ccfs": "ccfs://..."
}

You can also post the CCFS URI as text/plain.