Add encrypted backup generation and restore with progress tracking

[Copilot]

Implements complete backup and restore system that exports and imports database entries (documents, folders, revisions, access rules) and associated files using AES-256-CBC encrypted archives with separate key storage.

Changes

Backup Export

• include/util/backup.py: Backup generation utility

  • Serializes all documents, folders, revisions, and access rules to JSON
  • Collects referenced files into TAR archive
  • Encrypts archive with AES-256-CBC using 256-bit random key
  • Stores encryption key separately in JSON format

• include/handlers/management/system.py: RequestGenerateBackupHandler

  • Requires export_backup permission
  • Accepts optional backup_name parameter
  • Returns archive path, key path, and metadata (counts, version, timestamp)

Backup Import/Restore

• include/util/backup.py: Backup restore utility

  • Decrypts and extracts encrypted archives
  • Imports documents, folders, files, and access rules
  • Preserves relationships and hierarchy
  • Progress callbacks for real-time updates

• include/database/models/file.py: BackupTask model

  • Tracks import operations with status and progress
  • Stores imported counts and error messages
  • Persists progress across connections

• include/handlers/management/system.py: Import request handlers

  • RequestInitiateBackupImportHandler: Creates file upload tasks
  • RequestStartBackupImportHandler: Begins async import processing
  • RequestGetBackupImportStatusHandler: Queries import progress

API Integration

• include/connection_handler.py: Registered actions
  • generate_backup: Export backup
  • initiate_backup_import: Setup file uploads
  • start_backup_import: Begin import
  • get_backup_import_status: Query progress

Archive Structure

backup_name.cfms.enc (encrypted)
├── metadata.json          # Version, counts, creation time
├── documents.json         # Documents with revisions and access rules
├── folders.json           # Folders with access rules
├── files.json             # File metadata (id, path, sha256)
└── files/<file_id>        # Actual file contents

Import Process Flow

Step 1: Initiate - Client requests import, server creates upload tasks

{
  "action": "initiate_backup_import",
  "data": {"timeout_seconds": 1800},
  "username": "admin",
  "token": "<token>"
}
// Returns: backup_task_id, archive_task_id, key_task_id

Step 2: Upload - Client uploads files using existing transfer mechanism

{
  "action": "upload_file",
  "data": {"task_id": "<archive_task_id>"},
  "username": "admin",
  "token": "<token>"
}
// Upload encrypted archive chunks, then key file

Step 3: Start - Client starts import, server processes in background

{
  "action": "start_backup_import",
  "data": {"backup_task_id": "<backup_task_id>"},
  "username": "admin",
  "token": "<token>"
}
// Returns immediately, import runs asynchronously

Progress Monitoring - Poll status or wait for broadcast notification

{
  "action": "get_backup_import_status",
  "data": {"backup_task_id": "<backup_task_id>"},
  "username": "admin",
  "token": "<token>"
}
// Returns: status, current_step, progress_percent, counts

Features

• Async Processing: Import runs in background thread without blocking
• Progress Tracking: Database-backed with real-time updates (0-100%)
• Timeout Protection: Configurable timeout for file uploads
• Dual Notification: Active polling or passive broadcast listening
• Transaction Safety: Automatic rollback on any error
• File Transfer: Reuses existing AES-encrypted chunked transfer

Permissions

• export_backup: Generate backup archives
• import_backup: Import and restore backups

Testing

• test_backup.py: Validates backup generation and encryption
• test_restore.py: Complete backup→clear→restore cycle with data integrity verification

See BACKUP_IMPORT.md for complete API reference and usage examples.

Original prompt

First, analyze the structure of the entire code repository and, based on your understanding of the project, implement a backup generation function. This function will export the document and folder entry records, as well as the document-file relationships, from the database. This structure data and the corresponding files will be saved together in an encrypted archive file. The archive key will be stored in a separate file in a suitable format. The archive file generated by this function will allow other servers to automatically restore the document and folder entries and their dependencies and re-import the required files. This function should be a request and can be requested remotely by the client.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Creeper19472]

@copilot Please continue to implement the logic for re-importing the backup. Please note that this operation should be divided into the following steps:

1. If the request for this operation is from a client, the backup file and key file sent by the client should be accepted after authentication. However, if there is a better way to transfer the key, other means can also be used. The process of transferring files should call the method of creating a transfer task that has been implemented. Since the file transfer process may not be completed quickly, this request should probably return the result as soon as possible. However, in order to track the progress of the process, if necessary, the logic for implementing progress tracking should also be implemented. For this purpose, the overall structure of the project can be adjusted.
2. After confirming that all necessary files have been transferred, the backup import process should begin. If the file transfer is not completed within a certain period of time, the request should be considered timed out.
3. After the import operation is completed, the server should use some method to notify the client of the result of the request. This action can be active or passive.

Note that the permissions controlling export and import are named "export_backup" and "import_backup" respectively.

[Copilot]

@copilot Please continue to implement the logic for re-importing the backup. Please note that this operation should be divided into the following steps:

1. If the request for this operation is from...

Implemented the complete backup import/restore functionality as requested:

Commit: 234228f

Implementation:

1. File Transfer - Uses existing file transfer tasks with timeout support

   • initiate_backup_import: Creates upload tasks for archive and key files
   • Client uploads files via standard upload_file action (AES-encrypted chunks)
   • Configurable timeout (default 3600s), task marked as "timeout" if exceeded

2. Progress Tracking - Database-backed with BackupTask model

   • Tracks status (pending → processing → completed/failed/timeout)
   • Real-time step updates ("Decrypting", "Importing documents", etc.)
   • Progress percentage (0-100%)
   • Imported counts (documents, folders, files)
   • Persists across connections

3. Async Processing - Background thread with immediate response

   • start_backup_import: Validates files, starts background import, returns immediately
   • Progress callbacks update database during import
   • Transaction rollback on any error

4. Client Notification - Both active and passive

   • Active: Poll get_backup_import_status for current progress
   • Passive: Broadcast backup_import_completed event to all clients on completion

Permissions:

• export_backup: Generate backups
• import_backup: Import/restore backups

Testing: Complete backup→clear→restore cycle verified with data integrity checks.

Documentation: See BACKUP_IMPORT.md for complete API reference and client flow examples.