feat: Add WebDAV support for project synchronization and backup

[AinzRimuru]

Description

Pull Request: Add WebDAV Support for Project Synchronization and Backup

Summary

This PR introduces comprehensive WebDAV support for Overleaf, enabling users to configure WebDAV cloud storage for project synchronization and backup. The implementation spans both frontend and backend components, providing a complete solution for automatic and manual project file synchronization to WebDAV servers.

Motivation

Many users require the ability to backup their Overleaf projects to external storage services or keep local copies synchronized with cloud storage. WebDAV is a widely-supported protocol that enables file sharing and collaborative editing across different platforms. By integrating WebDAV support, users can:

• Automatically sync project changes to personal cloud storage (e.g., Nextcloud, ownCloud, Box, etc.)
• Maintain real-time backups of their LaTeX projects
• Access project files through standard file system interfaces
• Integrate with existing backup workflows and infrastructure

Key Features

1. WebDAV Persistor (WebDAVPersistor.js)

A new storage backend that implements the AbstractPersistor interface for WebDAV operations:

• File Operations: sendFile, sendStream, getObjectStream, copyObject, deleteObject
• Directory Operations: deleteDirectory, directorySize, listDirectoryKeys, listDirectoryStats
• Metadata Operations: getObjectSize, getObjectMd5Hash, getObjectMetadata, checkIfObjectExists
• Partial Content Support: Range header support for efficient partial file downloads
• Automatic Directory Creation: Ensures parent directories exist before file uploads

2. Sync Persistor (SyncPersistor.js)

A wrapper persistor that enables synchronization between the primary storage (FS/S3) and WebDAV:

• Dual-write Support: Files are written to both primary storage and WebDAV
• Project-aware Configuration: Dynamically retrieves WebDAV config per project
• Modification-based Sync: Only syncs files when changes are detected
• Bidirectional Sync Support: Framework for both push and pull synchronization

3. Project WebDAV Sync Service (ProjectWebDAVSync.mjs)

Handles synchronization of actual project content (documents and files):

• User-friendly Paths: Syncs files with their actual names as seen in the editor (not internal IDs)
• Document Sync: Fetches document content from Docstore and uploads to WebDAV
• File Sync: Streams binary files from Filestore to WebDAV
• Hash-based Incremental Sync: Uses content hashes (MD5 for documents, file.hash for binaries) to skip unchanged files
• Path Encoding: Encodes file paths for MongoDB storage (dots/dollars → percent-encoding)
• Full Project Sync: Supports complete project synchronization on demand
• Delete/Move Tracking: Maintains hash tracking during file deletion and rename operations

4. Auto-sync Service (ProjectWebDAVAutoSync.mjs)

Automatic synchronization triggered by project modifications:

• Debounced Sync: Batches rapid changes to reduce sync frequency
• Configurable Interval: Adjustable sync delay (default: 30 seconds)
• Binary File Priority: Immediate sync for binary file changes
• Project Close Sync: Ensures pending changes are synced when users leave
• Concurrent Sync Prevention: Uses locking to avoid duplicate sync operations

5. Frontend Components

Editor Integration

• SettingsWebDAV.tsx: WebDAV settings panel in the editor left menu
• WebDAVSettingsModal.tsx: Modal for configuring/editing WebDAV connection

Project List Integration

• LinkWebDAVModal.tsx: Modal for linking existing projects to WebDAV
• Enhanced DeleteProjectModal.tsx: Option to preserve cloud content when deleting projects
• Enhanced NewProjectForm.tsx: Optional WebDAV configuration during project creation

6. API Endpoints

Endpoint	Method	Description
/project/new	POST	Create project with optional WebDAV config
/project/:id/webdav/link	POST	Link existing project to WebDAV
/project/:id/webdav/unlink	POST	Disconnect WebDAV (optionally deletes cloud content via deleteRemoteContent body param)
/project/:id/webdav/sync	POST	Trigger manual full sync
/project/:id/webdav/config	GET	Get WebDAV configuration for a project

7. Data Model Changes

Added webdavConfig field to the Project schema:

webdavConfig: {
  url: String,              // WebDAV server URL
  username: String,         // Authentication username
  password: String,         // Password (should be encrypted)
  basePath: String,         // Base path (default: '/overleaf')
  enabled: Boolean,         // Enable/disable flag
  lastSyncDate: Date,       // Last sync timestamp
  syncedFileHashes: Mixed,  // Object mapping encoded file paths to content hashes
}

syncedFileHashes Field Details

This field stores a mapping of file paths to their content hashes, used to detect which files have actually changed and need to be synced. Due to MongoDB's restriction on field names containing dots (.) or dollar signs ($), file paths are encoded using URL-style percent encoding:

Character	Encoded
.	%2E
$	%24
%	%25

Example:

// Original path: /chapters/intro.tex
// Encoded key: /chapters/intro%2Etex
syncedFileHashes: {
  "/chapters/intro%2Etex": "d41d8cd98f00b204e9800998ecf8427e",
  "/images/logo%2Epng": "a1b2c3d4e5f6..."
}

Files Changed

New Files (9)

File	Description
libraries/object-persistor/src/WebDAVPersistor.js	WebDAV storage backend implementation
libraries/object-persistor/src/SyncPersistor.js	Dual-write sync wrapper
services/web/app/src/Features/Project/ProjectWebDAVSync.mjs	Project file sync service
services/web/app/src/Features/Project/ProjectWebDAVAutoSync.mjs	Automatic sync handling
services/filestore/app/js/ProjectConfigProvider.js	Project config provider for filestore
services/history-v1/storage/lib/ProjectConfigProvider.js	Project config provider for history
services/web/frontend/js/features/editor-left-menu/components/settings/settings-webdav.tsx	WebDAV settings component
services/web/frontend/js/features/editor-left-menu/components/settings/settings-webdav-modal.tsx	WebDAV settings modal
services/web/frontend/js/features/project-list/components/modals/link-webdav-modal.tsx	Link WebDAV modal

Modified Files (30)

Click to expand modified files list

Libraries

• libraries/object-persistor/README.md - Updated documentation
• libraries/object-persistor/index.js - Export new modules
• libraries/object-persistor/package.json - Added webdav dependency
• libraries/object-persistor/src/AbstractPersistor.js - Added getObjectMetadata interface
• libraries/object-persistor/src/FSPersistor.js - Implemented getObjectMetadata
• libraries/object-persistor/src/S3Persistor.js - Implemented getObjectMetadata
• libraries/object-persistor/src/PersistorFactory.js - Added WebDAV persistor creation

Services

• services/filestore/app.js - Added WebDAV config endpoint
• services/filestore/app/js/PersistorManager.js - SyncPersistor integration
• services/history-v1/storage/lib/persistor.js - SyncPersistor integration
• services/real-time/app/js/WebApiManager.js - WebDAV config fetching
• services/real-time/app/js/WebsocketController.js - Auto-sync integration

Web Service

• services/web/app/src/Features/Editor/EditorHttpController.mjs - WebDAV config endpoint
• services/web/app/src/Features/Editor/EditorRouter.mjs - New routes
• services/web/app/src/Features/Project/ProjectController.mjs - WebDAV API handlers
• services/web/app/src/Features/Project/ProjectEntityUpdateHandler.mjs - Sync triggers
• services/web/app/src/Features/Project/ProjectListController.mjs - WebDAV config in project list API
• services/web/app/src/Features/Project/ProjectUpdateHandler.mjs - WebDAV config handlers
• services/web/app/src/models/Project.mjs - Schema update
• services/web/app/src/router.mjs - Route registration

Frontend

• services/web/app/views/project/editor/_meta.pug - Meta data for WebDAV config
• services/web/frontend/extracted-translations.json - Translation keys for Webpack bundling
• services/web/frontend/js/features/editor-left-menu/components/settings-menu.tsx - Menu integration
• services/web/frontend/js/features/project-list/components/modals/delete-project-modal.tsx - Cloud preserve option
• services/web/frontend/js/features/project-list/components/new-project-button/modal-content-new-project-form.tsx - WebDAV during creation
• services/web/frontend/js/features/project-list/util/api.ts - API functions
• services/web/types/project/dashboard/api.d.ts - Type definitions

Localization

• services/web/locales/en.json - English translations
• services/web/locales/zh-CN.json - Chinese translations

Configuration

• .gitignore - Ignore patterns

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                          Frontend                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐   │
│  │ Settings     │  │ Link WebDAV  │  │ New Project Form     │   │
│  │ WebDAV       │  │ Modal        │  │ (with WebDAV option) │   │
│  └──────┬───────┘  └──────┬───────┘  └──────────┬───────────┘   │
└─────────┼─────────────────┼─────────────────────┼───────────────┘
          │                 │                     │
          ▼                 ▼                     ▼
┌──────────────────────────────────────────────────────────────────┐
│                         API Layer                                │
│  POST /project/:id/webdav/link    POST /project/:id/webdav/sync  │
│  POST /project/:id/webdav/unlink  GET  /project/:id/webdav/config│
└──────────────────────────────────────────────────────────────────┘
          │
          ▼
┌──────────────────────────────────────────────────────────────────┐
│                      Backend Services                            │
│  ┌──────────────────┐    ┌────────────────────────────────────┐  │
│  │ ProjectController│    │ ProjectWebDAVSync                  │  │
│  │                  │──▶│ - syncAllProjectFiles()            │  │
│  │ - linkWebDAV()   │    │ - syncDocument()                   │  │
│  │ - unlinkWebDAV() │    │ - syncFile()                       │  │
│  │ - syncWebDAV()   │    │ - deleteFromWebDAV()               │  │
│  └──────────────────┘    └────────────────────────────────────┘  │
│                              │                                   │
│  ┌──────────────────┐        │                                   │
│  │ ProjectWebDAV    │        │                                   │
│  │ AutoSync         │◄───────┘                                   │
│  │                  │                                            │
│  │ - markPending()  │                                            │
│  │ - triggerSync()  │                                            │
│  │ - forceSync()    │                                            │
│  └──────────────────┘                                            │
└──────────────────────────────────────────────────────────────────┘
          │
          ▼
┌──────────────────────────────────────────────────────────────────┐
│                      Storage Layer                               │
│  ┌──────────────────┐      ┌──────────────────┐                  │
│  │ SyncPersistor    │─────▶│ WebDAVPersistor  │────▶ WebDAV     │
│  │ (Wrapper)        │      │                  │       Server     │
│  └────────┬─────────┘      └──────────────────┘                  │
│           │                                                      │
│           ▼                                                      │
│  ┌──────────────────┐                                            │
│  │ Primary Persistor│────▶ Local/S3 Storage                     │
│  │ (FS/S3)          │                                            │
│  └──────────────────┘                                            │
└──────────────────────────────────────────────────────────────────┘

Configuration

Environment Variables

This commit introduces the following environment variables:

Environment Variable	Service	Description	Default
WEBDAV_SYNC_INTERVAL_MS	web	Auto-sync interval in milliseconds	30000 (30 seconds)

Usage

WEBDAV_SYNC_INTERVAL_MS

Controls the minimum interval between automatic syncs. When a user edits a document, the system will wait for this interval after the last sync before triggering the next sync.

# Example: Set to 60 seconds
WEBDAV_SYNC_INTERVAL_MS=60000

Auto-Sync Timing Constants

The following internal constants are defined in ProjectWebDAVAutoSync.mjs:

Constant	Value	Description
DEFAULT_SYNC_INTERVAL_MS	30,000ms (30 seconds)	Default sync interval
SYNC_DEBOUNCE_MS	5,000ms (5 seconds)	Debounce delay to prevent excessive syncing

Auto-Sync Behavior

The auto-sync triggering logic works as follows:

1. Document/File Modification → Calls markPendingSync(projectId, isBinaryChange)
2. Binary File Change (isBinaryChange=true) → Triggers immediate sync
3. Text Document Change → Checks time since last sync:
   • If exceeded WEBDAV_SYNC_INTERVAL_MS → Sync after SYNC_DEBOUNCE_MS
   • If not exceeded → Sync after remaining interval time
4. Project Close → Triggers background sync to ensure all changes are saved

Hash-based Sync Decision

During syncAllProjectFiles(), each file is compared by its content hash:

• Documents: MD5 hash is computed from the document's text content
• Binary Files: Uses the existing file.hash property from HistoryManager

If the hash matches the stored hash in syncedFileHashes, the file is skipped. This prevents redundant uploads when reopening a project that hasn't actually changed.

┌──────────────────┐     ┌─────────────────┐
│ Document Edit    │───▶│ markPendingSync │
└──────────────────┘     └────────┬────────┘
                                  │
                    ┌─────────────┴─────────────┐
                    │                           │
               Binary?                     Text Doc
                    │                           │
                    ▼                           ▼
          ┌─────────────────┐      ┌────────────────────────┐
          │ Immediate Sync  │      │ Check lastSyncDate     │
          └─────────────────┘      │ vs SYNC_INTERVAL_MS    │
                                   └───────────┬────────────┘
                                               │
                              ┌────────────────┴────────────────┐
                              │                                 │
                       Interval Exceeded               Interval Not Exceeded
                              │                                 │
                              ▼                                 ▼
                   ┌──────────────────┐            ┌──────────────────────┐
                   │ Debounce (5s)    │            │ Wait remaining time  │
                   │ then Sync        │            │ then Sync            │
                   └──────────────────┘            └──────────────────────┘

Dependencies

Added the following npm dependency:

{
  "webdav": "^5.8.0"
}

Note: The package-lock.json file is intentionally not included in this commit. Running npm install locally caused extensive modifications to the lock file, which would obscure the actual changes in this PR. Please run npm install after merging to regenerate the lock file.

Security Considerations

Current Implementation

1. Transport Security: All credentials are transmitted over HTTPS
2. Input Validation: Zod schema validation for all WebDAV configuration fields
3. Length Limits: Maximum lengths enforced on URL, username, password, and basePath
4. Data Cleanup: $unset operator removes entire webdavConfig on unlink
5. Permission Checks: Only project admins can configure WebDAV

Known Limitations & Future Improvements

• Password Storage: Currently stored in plain text (should be encrypted)
• Connection Testing: No real-time connection validation in UI
• Sync Status UI: No visual indicator for sync progress/status
• Conflict Resolution: No automated conflict resolution for concurrent edits

Testing

Manual Testing Checklist

• Create new project with WebDAV configuration
• Link existing project to WebDAV
• Verify automatic sync after document changes
• Test manual sync via editor menu
• Delete project with "keep cloud storage" option
• Delete project without preserve option
• Verify file renames/moves are synced
• Verify file deletions are synced

Recommended Test Environments

• WsgiDAV (quick local debugging: pip install wsgidav cheroot && wsgidav --host=0.0.0.0 --port=8080 --root=./webdav_test --auth=anonymous)
• Nextcloud server
• ownCloud server
• Any WebDAV-compatible storage service

Breaking Changes

None. This is an additive feature that does not affect existing functionality.

Migration Notes

No database migrations required. The webdavConfig field is optional and will be undefined for existing projects.

Screenshots

Configure WebDAV When Creating a New Project

Configuration Entry for WebDAV in the Project

WebDAV Configuration Interface

Choose Whether to Delete WebDAV Data When Deleting a Project

Choose Whether to Delete WebDAV Data When Unlinking

Choose Whether to Delete WebDAV Data When Unlinking

---

Commit: a9fab0d300
Branch: addWebDAVSupport
Author: AinzRimuru
Date: 2025-12-21

Related issues / Pull Requests

Contributes to #147 #161 #501 #526 #1040 #1300

Contributor Agreement

• I confirm I have signed the Contributor License Agreement

[AinzRimuru]

I've noticed that the JSON files related to localization translation are not editable. However, since the changes involve the interface, some new fields have been added. Please advise on how to properly resolve this issue.