Skip to content

feat(ovpack): add v2 manifest and backup restore#1927

Merged
MaojiaSheng merged 13 commits into
mainfrom
feat/ovpack-v2-manifest
May 11, 2026
Merged

feat(ovpack): add v2 manifest and backup restore#1927
MaojiaSheng merged 13 commits into
mainfrom
feat/ovpack-v2-manifest

Conversation

@qin-ctx
Copy link
Copy Markdown
Collaborator

@qin-ctx qin-ctx commented May 9, 2026

Description

这个 PR 将 OVPack 从“裸 ZIP 导入导出”重构为可校验、可恢复的 OpenViking 内容包格式,并把普通 export/import 与全量 backup/restore 的语义拆开。

核心目标:

  • 普通 OVPack 用于迁移一个明确的 viking:// 子树,导入时保留包根。
  • backup OVPack 用于全量备份公开 scope,只能通过 restore 恢复。
  • 导入前先完成 manifest、checksum、路径和冲突校验,避免边写边失败或把包导入到错误 scope。
  • 导入后重新生成目标环境的向量和运行态字段,不迁移 dense/sparse vector。

Changes Made

  • 新增 OVPack v2 manifest 约束:
    • kind=openviking.ovpack
    • format_version=2
    • root
    • entries
    • 每个文件的 size / sha256
    • 整体 content_sha256
  • 导入前执行完整校验:
    • ZIP 成员路径安全性
    • manifest 存在性和版本
    • manifest 文件/目录集合与 ZIP 内容一致性
    • 文件 size / sha256
    • 整体 content_sha256
    • source scope 与 target scope 兼容
    • 结构化 scope 的 root 层级不变
    • 目标 root 冲突策略
  • 移除导入侧旧控制参数:
    • 移除 vectorize / --no-vectorize
    • 移除 force / --force
    • 统一使用 on_conflict=fail|overwrite|skip
  • 支持 session 普通 OVPack:
    • viking://session/<session_id> 可以导出
    • 只能导回 viking://session/
    • 恢复 session 文件状态
    • 不触发向量化
  • 支持顶级公开 scope 包:
    • viking://resources
    • viking://user
    • viking://agent
    • viking://session
    • 这类包只能导入到 viking://
  • 新增全量备份恢复能力:
    • Python/local API: backup_ovpack / restore_ovpack
    • HTTP API: /api/v1/pack/backup / /api/v1/pack/restore
    • CLI: ov backup / ov restore
    • backup 包不能通过普通 import 导入
  • 重构导出写包逻辑:
    • 普通 export 和 backup 复用 manifest/checksum/archive 写入路径
    • 减少重复打包逻辑
  • 调整向量元数据迁移策略:
    • 导出可迁移标量:typecontext_typelevelnamedescriptiontagsabstract
    • 不导出 dense vector、sparse vector、id、uri、account_id、owner_user_id、owner_agent_id、owner_space 等运行态字段
    • created_atupdated_atactive_count 在目标环境重新生成
    • 导入后按目标 URI 重新计算 context_type
  • 排除内部文件:
    • .relations.json
    • OVPack manifest
    • 锁文件
    • temp/queue/upload 等内部运行态内容
  • 改进错误返回:
    • 导入冲突返回结构化 conflict,而不是内部 500
    • 缺失/错误 manifest、checksum 不匹配、scope 不兼容等返回明确错误
  • 更新文档:
    • 中英文 OVPack 指南
    • API 文档
    • 架构/概念文档
    • CLI 使用说明
  • 精简测试覆盖:
    • 保留 manifest/checksum、unsafe zip path、目录集合、旧包拒绝、session 不向量化、scope 导入兼容、backup/restore 合约等核心回归

Expected Behavior Examples

1. 普通资源子树导入保留包根

ov export viking://resources/projectA/ ./exports/projectA.ovpack
ov import ./exports/projectA.ovpack viking://resources/imported/

结果:

viking://resources/imported/projectA/...

viking://resources/imported/ 是父目录,不是最终 root。

2. 同 scope 才允许普通 import

允许:

ov export viking://resources/docs/ ./exports/docs.ovpack
ov import ./exports/docs.ovpack viking://resources/imported/

拒绝:

ov import ./exports/docs.ovpack viking://session/

原因:

source scope resources != target scope session

3. session 包只能导回 session scope

ov export viking://session/sess_123/ ./exports/sess_123.ovpack
ov import ./exports/sess_123.ovpack viking://session/ --on-conflict overwrite

结果:

viking://session/sess_123/...

session 包恢复文件状态,不写向量库。

下面两种会被拒绝:

ov import ./exports/sess_123.ovpack viking://resources/

原因:session 包不能变成 resources。

ov import ./exports/sess_123.ovpack viking://session/sess_123/

原因:导入会生成 viking://session/sess_123/sess_123 这种嵌套 root。

4. 顶级 scope 包只能导入 viking://

ov export viking://resources/ ./exports/resources.ovpack
ov import ./exports/resources.ovpack viking:// --on-conflict overwrite

结果:

viking://resources/...

下面这种会被拒绝:

ov import ./exports/resources.ovpack viking://resources/imported/

原因:顶级 scope 包不能作为普通子树嵌套导入。

5. 全量迁移使用 backup/restore

ov backup ./backups/openviking.ovpack
ov restore ./backups/openviking.ovpack --on-conflict overwrite

restore 会恢复公开 scope root:

viking://resources/...
viking://user/...
viking://agent/...
viking://session/...

backup 包不能被普通 import 使用:

ov import ./backups/openviking.ovpack viking://

会返回错误,提示使用 ov restore 或 restore API。

6. manifest / checksum 校验

下面情况会在写入前被拒绝:

  • 缺少 <root>/_._ovpack_manifest.json
  • format_version 不是当前支持版本
  • ZIP 里多出 manifest 未声明的文件或目录
  • manifest 声明的文件或目录缺失
  • 文件 size 不匹配
  • 文件 sha256 不匹配
  • 整体 content_sha256 不匹配
  • source scope 与 target scope 不一致

7. 冲突策略

on_conflict=fail

目标 root 已存在时返回 CONFLICT,不写入。

on_conflict=overwrite

先完成包校验;校验通过后删除目标 root,再写入新内容。

on_conflict=skip

目标 root 已存在时直接返回该 URI,不覆盖、不补齐文件。

Related Issue

N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

验证命令:

.venv/bin/ruff check openviking/storage/local_fs.py openviking/service/pack_service.py openviking/server/routers/pack.py openviking_cli/client/base.py openviking_cli/client/http.py openviking_cli/client/sync_http.py openviking/client/local.py openviking/async_client.py openviking/sync_client.py tests/misc/test_ovpack_import_policy.py
.venv/bin/python -m py_compile openviking/storage/local_fs.py openviking/service/pack_service.py openviking/server/routers/pack.py openviking_cli/client/base.py openviking_cli/client/http.py openviking_cli/client/sync_http.py openviking/client/local.py openviking/async_client.py openviking/sync_client.py tests/misc/test_ovpack_import_policy.py
.venv/bin/python -m pytest tests/misc/test_ovpack_import_policy.py -q
.venv/bin/python -m pytest tests/server/test_api_local_input_security.py::test_import_ovpack_conflict_returns_structured_conflict -q
.venv/bin/python -m pytest tests/client/test_import_export.py::TestImportOvpack::test_import_rejects_unsafe_entries -q
cargo check --manifest-path crates/ov_cli/Cargo.toml -q
git diff --check

最近一次针对 scope compatibility 调整额外验证:

.venv/bin/ruff check openviking/storage/local_fs.py tests/misc/test_ovpack_import_policy.py
.venv/bin/python -m py_compile openviking/storage/local_fs.py tests/misc/test_ovpack_import_policy.py
.venv/bin/python -m pytest tests/misc/test_ovpack_import_policy.py -q

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

N/A

Additional Notes

Breaking change 说明:

  • 旧版无 manifest 的 .ovpack 默认拒绝导入,需要在可信环境重新导出为 v2 包。
  • 导入侧移除了 vectorize / force 兼容参数,避免多个控制面表达同一语义。
  • 普通 import 不再允许跨 scope 导入。
  • 全量迁移改为专门的 backup/restore 接口,普通 export/import 不承担全量备份恢复语义。

Add a portable OVPack manifest for scalar metadata and make imports validate scope, derived files, and conflicts before writing.
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 9, 2026

PR Code Suggestions ✨

No code suggestions found for the PR.

qin-ctx added 11 commits May 9, 2026 11:37
Make OVPack imports always rebuild vectors in the target environment, keep legacy packages compatible, and reject unsupported manifest versions before writing.
Use on_conflict as the single OVPack import conflict policy and reject removed force inputs.
Keep type portable but stop exporting or applying created_at, updated_at, and active_count from OVPack manifests.
@qin-ctx qin-ctx changed the title feat(ovpack): add v2 manifest and conflict policy feat(ovpack): add v2 manifest and backup restore May 9, 2026
@qin-ctx qin-ctx force-pushed the feat/ovpack-v2-manifest branch from d3e8afe to 4f40557 Compare May 9, 2026 12:17
@MaojiaSheng MaojiaSheng merged commit e648b26 into main May 11, 2026
13 checks passed
@MaojiaSheng MaojiaSheng deleted the feat/ovpack-v2-manifest branch May 11, 2026 03:09
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project May 11, 2026
ZaynJarvis pushed a commit that referenced this pull request May 13, 2026
* feat(ovpack): add v2 manifest and conflict policy

Add a portable OVPack manifest for scalar metadata and make imports validate scope, derived files, and conflicts before writing.

* fix(ovpack): remove import vectorize option

Make OVPack imports always rebuild vectors in the target environment, keep legacy packages compatible, and reject unsupported manifest versions before writing.

* fix(ovpack): remove force import alias

Use on_conflict as the single OVPack import conflict policy and reject removed force inputs.

* fix(ovpack): regenerate runtime vector metadata

Keep type portable but stop exporting or applying created_at, updated_at, and active_count from OVPack manifests.

* fix(ovpack): validate manifest contents

* fix(ovpack): require manifests for imports

* fix(ovpack): close manifest validation gaps

* fix(ovpack): defer parent creation until validation passes

* fix(ovpack): remove export size guard

* fix(ovpack): support session and scope-root restores

* docs(ovpack): document full backup migration

* feat(ovpack): add backup restore workflow

* fix(ovpack): validate import scope compatibility
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants