feat(schema): SQLite Schema Storage + Migration Ledger

[hemanta212]

SQLite Schema Storage + Migration Ledger

TL;DR

Replace .scaf-schema.yaml with .scaf.db (SQLite). Store schema + nullability config + migration ledger + snapshots. Migrations stay as files; SQLite is the ledger.

Design Decisions

Nullability Semantics

SQLite stores nullable BOOLEAN DEFAULT FALSE. The internal representation is unambiguous. nullability_mode config only affects adapter extraction and YAML migration.

Migration Storage

Options considered:

Option	Approach	Trade-off
A	Scripts in SQLite only	Can't code review, git conflict hell
B	Scripts as files, SQLite ledger	Industry standard (Flyway/Liquibase/Prisma)

Chosen: B - migrations/ directory with text files. SQLite stores: version, name, checksum, applied_at, success, execution_ms.

Rename Detection

• Require explicit hint as Autodetection is ambiguous, else treat as drop+add
• CLI prompts: "Is this a rename or drop+add?" when unclear.

Graph DB Migration Scope (Neo4j/Cypher)

Neo4j doesn't enforce schema - nodes can have arbitrary properties. The question is what migrations should generate.

Decision: Migration generation is driven by the schema's nullable field:

Schema Change	Migration Generated
New field, nullable=true	None. Schema DB updated for LSP/codegen only.
New field, nullable=false	Backfill script: MATCH (n:Model) WHERE n.field IS NULL SET n.field = <default>
Field gains unique=true	CREATE CONSTRAINT
Field gains index	CREATE INDEX
Field removed	REMOVE n.field (with warning)
nullable: true -> false	Breaking change. Backfill required, CLI prompts for default value.

The adapter extracts nullable from ORM metadata (pointer types, struct tags, etc.) and populates the schema. Scaf's migration generator only sees the schema - it doesn't know ORM internals.

Schema Design

-- Config (nullability_mode, dialect, adapter, version)
CREATE TABLE schema_config (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL,
    updated_at INTEGER DEFAULT (strftime('%s', 'now'))
);

-- Models
CREATE TABLE models (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT UNIQUE NOT NULL,
    kind TEXT NOT NULL CHECK (kind IN ('node', 'relationship', 'table')),
    created_at INTEGER DEFAULT (strftime('%s', 'now')),
    updated_at INTEGER DEFAULT (strftime('%s', 'now'))
);

-- Fields (nullable is the primitive, not required)
CREATE TABLE fields (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_id INTEGER NOT NULL REFERENCES models(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    nullable BOOLEAN DEFAULT FALSE,  -- NOT NULL by default
    unique_field BOOLEAN DEFAULT FALSE,
    created_at INTEGER DEFAULT (strftime('%s', 'now')),
    updated_at INTEGER DEFAULT (strftime('%s', 'now')),
    UNIQUE(model_id, name)
);

-- Relationships
CREATE TABLE relationships (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    source_model_id INTEGER NOT NULL REFERENCES models(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    rel_type TEXT NOT NULL,
    target_model_id INTEGER NOT NULL REFERENCES models(id),
    many BOOLEAN DEFAULT FALSE,
    direction TEXT NOT NULL CHECK (direction IN ('outgoing', 'incoming')),
    created_at INTEGER DEFAULT (strftime('%s', 'now')),
    UNIQUE(source_model_id, name)
);

-- Schema snapshots (JSON, for diffing)
CREATE TABLE schema_snapshots (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    version INTEGER NOT NULL,
    snapshot_json TEXT NOT NULL,
    git_commit TEXT,
    created_at INTEGER DEFAULT (strftime('%s', 'now'))
);

-- Migration ledger (Flyway-style)
CREATE TABLE migrations (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    version INTEGER NOT NULL,
    name TEXT NOT NULL,           -- e.g., "add_user_bio"
    filename TEXT NOT NULL,       -- e.g., "001_add_user_bio.cypher"
    checksum TEXT NOT NULL,       -- SHA256 of file contents
    applied_at INTEGER,
    success BOOLEAN,
    execution_ms INTEGER,
    UNIQUE(version)
);

SQLite Config

db.Exec("PRAGMA journal_mode=WAL")      // LSP reads while CLI writes
db.Exec("PRAGMA foreign_keys=ON")
db.Exec("PRAGMA busy_timeout=5000")

Interface Changes

SchemaLoader

type SchemaLoader interface {
    Load(path string) (*TypeSchema, error)
    Save(path string, schema *TypeSchema) error
}

// Auto-detect by extension
func NewSchemaLoader(path string) SchemaLoader {
    if strings.HasSuffix(path, ".db") {
        return &SQLiteSchemaLoader{}
    }
    return &YAMLSchemaLoader{}
}

SchemaRepository (new)

type SchemaRepository struct {
    db *sql.DB
}

func (r *SchemaRepository) GetConfig(key string) (string, error)
func (r *SchemaRepository) SetConfig(key, value string) error
func (r *SchemaRepository) GetModels() ([]*Model, error)
func (r *SchemaRepository) UpsertModel(m *Model) error
func (r *SchemaRepository) SaveSnapshot(version int, schema *TypeSchema) error
func (r *SchemaRepository) GetLatestSnapshot() (*TypeSchema, int, error)
func (r *SchemaRepository) RecordMigration(m *MigrationRecord) error
func (r *SchemaRepository) GetPendingMigrations() ([]*MigrationRecord, error)

CLI Commands

go run ./cmd/schema      # user control schema extraction by populating to .scaf.db

scaf schema diff         # Compare extracted vs latest snapshot
scaf schema migrate generate [name]  # Diff -> migration file
scaf schema migrate apply            # Apply pending, update ledger
scaf schema migrate status           # Show applied/pending
scaf schema snapshot     # Store current as new snapshot

Migration File Format

migrations/
  001_initial.cypher
  002_add_user_bio.cypher
  003_rename_name_to_fullname.cypher

Each file:

-- Migration: add_user_bio
-- Version: 2
-- Generated: 2024-01-15T10:30:00Z

-- @up
MATCH (u:User)
WHERE u.bio IS NULL
SET u.bio = ""

-- @down
MATCH (u:User)
REMOVE u.bio

Diff Algorithm

type SchemaDiff struct {
    AddedModels    []*Model
    RemovedModels  []string
    ChangedModels  []ModelDiff
}

type ModelDiff struct {
    Name          string
    AddedFields   []*Field
    RemovedFields []string
    ChangedFields []FieldDiff  // type change, nullable change
    AmbiguousRenames []RenamePair  // needs user confirmation
}

Breaking changes flagged:

• nullable: true -> false (may have NULL data)
• Type change (needs conversion)
• Field/model removal (data loss)

Dialect Migration Interface

type MigrationGenerator interface {
    GenerateAddField(model, field string, typ *Type, nullable bool) string
    GenerateRemoveField(model, field string) string
    GenerateAddModel(model *Model) string
    GenerateRemoveModel(name string) string
    GenerateRenameField(model, oldField, newField string) string
}

Cypher impl: SET n.field = ..., REMOVE n.field
SQL impl: ALTER TABLE ... ADD COLUMN, DROP COLUMN