Skip to content

[WIP] Add real CSV/OFX/QFX import adapters with provenance#54

Closed
Copilot wants to merge 2 commits intomainfrom
copilot/feat-ingestion-real-adapters
Closed

[WIP] Add real CSV/OFX/QFX import adapters with provenance#54
Copilot wants to merge 2 commits intomainfrom
copilot/feat-ingestion-real-adapters

Conversation

Copy link
Contributor

Copilot AI commented Mar 22, 2026

  • Create packages/ingestion package with:
    • types.ts — ImportSession, RawTransaction, IngestionResult, RowError types
    • date-parser.ts — Multi-format date detection (MM/DD/YYYY, ISO, OFX YYYYMMDD, etc.)
    • csv-adapter.ts — Full CSV adapter with column mapping, date detection, per-row error tracking
    • ofx-adapter.ts — Full OFX/QFX adapter with SGML+XML support, per-row error tracking
    • storage.ts — IIngestionStore interface + InMemoryIngestionStore implementation
    • ingestion-service.ts — Main service: hash dedup → session → parse → store raw → report
    • index.ts — Package exports
    • package.json + tsconfig.json
  • Update root package.json build:packages and test:unit scripts to include ingestion
  • Update tsconfig.test.json with ingestion path alias
  • Add unit tests in test/unit/ingestion.test.ts
  • Verify build passes and tests pass
Original prompt

This section details on the original issue you should resolve

<issue_title>feat(ingestion): real CSV/OFX/QFX import adapters with provenance</issue_title>
<issue_description>## Summary
Build packages/ingestion with real, end-to-end file import that persists everything. The current importers are parse-only shells — make them real adapters.

Owns

  • CSV adapter: Configurable column mapping, date format detection, amount parsing
  • OFX adapter: Parse OFX/QFX XML, extract transactions, account info
  • File templates: Saved column mappings per institution/file type
  • Row parsing: Error-tolerant parsing with per-row error tracking
  • Raw transaction persistence: Store exactly what was imported, before any normalization
  • Provenance: SHA-256 file hashes, import timestamps, source tracking
  • Idempotent import: Re-importing same file is a no-op (hash dedup)

Import Flow

File → Hash check (dedup) → Create ImportSession → Parse rows →
Store raw transactions → Map to account → Generate postings → Report

Error Handling

  • Parse errors don't abort the import — skip bad rows, report them
  • Each ImportSession tracks: total rows, parsed rows, error rows, duplicate rows
  • Errors are reviewable in the UI

Acceptance

  • CSV import with configurable column mapping works end-to-end
  • OFX/QFX parsing extracts transactions correctly
  • File hash dedup prevents duplicate imports
  • Import sessions are persisted with full provenance
  • Parse errors are tracked per-row, not fatal</issue_description>

Comments on the Issue (you are @copilot in this section)


⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.

Co-authored-by: kayodebristol <3579196+kayodebristol@users.noreply.github.com>
Agent-Logs-Url: https://github.com/plures/FinancialAdvisor/sessions/6c92c668-9bf9-4672-97fb-7beacea85c3e
@kayodebristol
Copy link
Contributor

Closing this PR to enforce the 1-PR-per-repo rule. Only one Copilot PR is allowed per repository at a time. The oldest PR (#48) will be kept active.

Copilot AI requested a review from kayodebristol March 22, 2026 23:31
Copilot stopped work on behalf of kayodebristol due to an error March 22, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ingestion): real CSV/OFX/QFX import adapters with provenance

2 participants