Add comment system, highlights view, and markdown support

.env.example

@@ -0,0 +1,6 @@
+# DeepSeek API Configuration
+# Get your API key from: https://platform.deepseek.com/api_keys
+
+OPENAI_API_KEY=your_api_key_here
+OPENAI_BASE_URL=https://api.deepseek.com
+OPENAI_MODEL=deepseek-chat

.gitignore

@@ -9,6 +9,36 @@ wheels/
 # Virtual environments
 .venv
 
+# IDE settings
+.vscode/
+.idea/
+
 # Custom
-*_data/
 *.epub
+
+# Books directory (but keep the folder structure)
+books/*
+!books/.gitkeep
+
+# Temp directory for uploads
+temp/
+
+# AI Features & Data
+.env
+reader_data.db
+test.db
+
+# Backup files
+backups/
+*.db.backup
+
+# Export files
+reader_data_*.json
+highlights_*.csv
+ai_analyses_*.csv
+report_*.txt
+
+# OS files
+.DS_Store
+Thumbs.db
+desktop.ini

README.md

@@ -1,27 +1,171 @@
-# reader 3
+# Reader3 - EPUB Reader with AI Analysis
 
-![reader3](reader3.png)
+A lightweight, self-hosted EPUB reader with integrated AI analysis capabilities.
 
-A lightweight, self-hosted EPUB reader that lets you read through EPUB books one chapter at a time. This makes it very easy to copy paste the contents of a chapter to an LLM, to read along. Basically - get epub books (e.g. [Project Gutenberg](https://www.gutenberg.org/) has many), open them up in this reader, copy paste text around to your favorite LLM, and read together and along.
+## Features
 
-This project was 90% vibe coded just to illustrate how one can very easily [read books together with LLMs](https://x.com/karpathy/status/1990577951671509438). I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.
+### Reading Experience
+- 📚 **Clean Layout** - Three-column design (TOC, Content, AI Panel)
+- 📖 **Sticky Navigation** - Top navigation bar stays visible while scrolling
+- ⌨️ **Keyboard Shortcuts** - Arrow keys for prev/next chapter, ESC to close panels
+- 🔗 **Internal Links** - Footnotes and author comments open in modal popups
+- 🎯 **Clickable Covers** - Click book covers to start reading instantly
 
-## Usage
+### AI & Annotations
+- 🤖 **AI Analysis** - Right-click on text for fact-checking or discussion (DeepSeek)
+- � ***Personal Comments** - Add your own notes without AI (no API cost)
+- 💾 **Manual Save** - Choose what to save to avoid clutter
+- ✨ **Color-Coded Highlights** - Yellow (fact check), Blue (discussion), Green (comments)
+- 🏷️ **Smart Tooltips** - Hover over highlights to see type
+- 🗑️ **Edit & Delete** - Manage all your highlights and comments
+- 🎨 **Markdown Support** - AI responses render with proper formatting
+
+### Library & Organization
+- 📝 **Highlights View** - See all your notes and analyses for each book
+- 📤 **Export to Markdown** - Export highlights with AI context warnings
+- 🌐 **Web Upload** - Upload EPUB files via click or drag & drop
+- 🖼️ **Cover Images** - Automatic cover extraction and display
+- 🔍 **Search** - Find books by title or author
+- 🗂️ **Organized Storage** - All books in `books/` directory, data in SQLite
+
+## Quick Start
+
+### 1. Configure API Key
+
+Edit `.env` file:
+```bash
+OPENAI_API_KEY=your_deepseek_key
+OPENAI_BASE_URL=https://api.deepseek.com
+OPENAI_MODEL=deepseek-chat
+```
+
+Get your key from: https://platform.deepseek.com/api_keys
 
-The project uses [uv](https://docs.astral.sh/uv/). So for example, download [Dracula EPUB3](https://www.gutenberg.org/ebooks/345) to this directory as `dracula.epub`, then:
+### 2. Add Books
 
+**Option A: Upload via Web Interface (Recommended)**
+1. Start server: `uv run server.py`
+2. Open http://127.0.0.1:8123
+3. Click the "+" card OR drag & drop EPUB file
+4. Wait for automatic processing
+
+**Option B: Command Line**
 ```bash
-uv run reader3.py dracula.epub
+uv run reader3.py your_book.epub
 ```
 
-This creates the directory `dracula_data`, which registers the book to your local library. We can then run the server:
+### 3. Start Server
 
 ```bash
 uv run server.py
 ```
 
-And visit [localhost:8123](http://localhost:8123/) to see your current Library. You can easily add more books, or delete them from your library by deleting the folder. It's not supposed to be complicated or complex.
+### 4. Read and Analyze
+
+1. Open http://127.0.0.1:8123
+2. Select a book
+3. Right-click on text → Choose analysis type
+4. Review AI response in side panel
+5. Save if important
+6. Highlights appear on next visit!
+
+## Usage
+
+### AI Analysis
+- Select text → Right-click → Choose:
+  - **📋 Fact Check** - Verify facts and get context
+  - **💡 Discussion** - Deep analysis and insights
+  - **💬 Add Comment** - Your personal notes (no AI)
+- View response in right panel
+- Click "Save" for important insights
+
+### Highlights
+- **Yellow** - Fact checks
+- **Blue** - Discussions
+- **Green** - Your comments
+- Hover to see type, click to view/edit
+- All highlights are editable and deletable
+
+### View & Export Highlights
+- Click ⋮ menu on any book → "View Highlights"
+- See all your notes and analyses in one page
+- Filter by type (Fact Check, Discussion, Comment)
+- Export to markdown for AI processing
+- Context length warnings for large exports
+- Jump directly to any chapter
+
+### Keyboard Shortcuts
+- **← →** - Navigate between chapters
+- **ESC** - Close panels and modals
+- Works anywhere except when typing in text fields
+
+## Project Structure
+
+```
+reader3/
+├── reader3.py          # EPUB processor
+├── server.py           # Web server
+├── database.py         # SQLite operations
+├── ai_service.py       # AI integration
+├── books/              # All book data here
+│   └── book_name_data/
+│       ├── book.pkl
+│       └── images/
+├── templates/          # HTML templates
+├── reader_data.db      # SQLite database
+└── .env                # API configuration
+```
+
+## Data Management
+
+### View Your Highlights
+- Click ⋮ menu on any book → "View Highlights"
+- See all notes, comments, and analyses in one page
+- Filter by type and jump to chapters
+
+### View Database (Advanced)
+```bash
+uv run check_database.py
+```
+
+### Backup
+```bash
+# Double-click: backup.bat
+# Or manually:
+copy reader_data.db backups\reader_data_backup.db
+```
+
+## Tools
+
+- `check_database.py` - View raw database contents (advanced)
+- `backup.bat` - Quick database backup
+
+## Why DeepSeek?
+
+- ✅ Cost-effective (¥1/M tokens input, ¥2/M output)
+- ✅ Excellent Chinese language support
+- ✅ Fast response in China
+- ✅ OpenAI-compatible API
+
+## Troubleshooting
+
+### API Key Error
+1. Check `.env` file exists and has correct key
+2. Restart server
+
+### No Highlights Showing
+1. Check browser console (F12) for errors
+2. Verify data exists: `uv run check_database.py`
+3. Hard refresh (Ctrl+Shift+R)
+
+### Server Won't Start
+1. Check if port 8123 is available
+2. Verify `.env` configuration
 
 ## License
 
-MIT
+MIT
+
+---
+
+**Note**: This project is designed to be simple and hackable. Ask your LLM to modify it however you like!

TECHNICAL_CHALLENGES.md

@@ -0,0 +1,158 @@
+# Technical Challenges Solved
+
+This document outlines the key technical challenges we encountered and solved while building this AI-powered EPUB reader.
+
+## 1. EPUB Cover Image Extraction
+
+**Challenge**: Cover images weren't being extracted from EPUB files. Some books had covers marked as `ITEM_COVER` type instead of `ITEM_IMAGE`, causing them to be skipped.
+
+**Solution**: 
+- Modified image extraction to handle both `ITEM_COVER` and `ITEM_IMAGE` types
+- Implemented multi-method cover detection: check ITEM_COVER type → search by filename pattern → use first large image as fallback
+- Added size filtering (>10KB) to avoid using small icons as covers
+
+**Code**: `reader3.py` lines 190-230
+
+## 2. Multi-Paragraph Text Highlighting
+
+**Challenge**: When users highlighted text spanning multiple paragraphs, the highlight wouldn't display because wrapping `<p>` tags in a `<span>` creates invalid HTML that browsers reject.
+
+**Solution**:
+- Detect when highlighted text spans block elements
+- Apply highlight class directly to the paragraph elements instead of wrapping
+- Use Range API with whitespace-tolerant regex matching to handle text across multiple elements
+- Normalize whitespace in search patterns to handle variations in HTML structure
+
+**Code**: `templates/reader.html` - `applyHighlights()` and `findTextRange()` functions
+
+## 3. FastAPI Route Ordering for Image Serving
+
+**Challenge**: Image URLs like `/read/{book_id}/images/{image_name}` were returning 404 because the catch-all route `/read/{book_id}/{chapter_ref:path}` was matching first.
+
+**Solution**:
+- Moved the specific image route definition before the generic chapter route
+- FastAPI matches routes in order, so more specific routes must come first
+- Also fixed path handling to preserve spaces in book folder names (removed incorrect `os.path.basename()` usage)
+
+**Code**: `server.py` - route ordering around line 125-175
+
+## 4. Reading Progress with Precise Scroll Position
+
+**Challenge**: 
+- `scrollTop` was always returning 0 when read directly
+- `beforeunload` event doesn't fire reliably
+- Need to track exact scroll position within chapters, not just chapter numbers
+
+**Solution**:
+- Use scroll event listener to continuously track `currentScrollPosition` variable
+- Intercept navigation clicks with `preventDefault()` to ensure save completes before navigation
+- Add `pagehide` event as backup for mobile browsers
+- Store both chapter index and scroll position in database
+- Implement retry mechanism for scroll restoration to handle content loading delays
+
+**Code**: `templates/reader.html` - scroll tracking and `saveProgress()` function
+
+## 5. Database Schema Migration
+
+**Challenge**: Adding `scroll_position` column to existing `reading_progress` table without breaking existing data.
+
+**Solution**:
+- Created migration script that checks if column exists before adding
+- Used `ALTER TABLE ADD COLUMN` with `DEFAULT 0` for backward compatibility
+- Gracefully handles both new installations and existing databases
+
+**Code**: `migrate_progress.py`
+
+## 6. AI Prompt Engineering for Reading Context
+
+**Challenge**: Generic AI prompts weren't providing useful reading assistance. Needed different types of help for different reading scenarios.
+
+**Solution**:
+- Split into two distinct functions:
+  - **解释说明 (Explanation)**: Quick lookups for terms, people, events, concepts
+  - **深入讨论 (Discussion)**: Academic analysis with theoretical frameworks and critical thinking
+- Structured prompts with clear dimensions (论点解析, 理论视角, 批判思考, 启发问题)
+- Removed context parameter from fact-check to keep it focused and fast
+
+**Code**: `ai_service.py` - `fact_check()` and `discuss()` methods
+
+## 7. Dark Mode Implementation
+
+**Challenge**: Implementing comprehensive dark mode across all pages with proper contrast and readability.
+
+**Solution**:
+- Used CSS class toggle (`body.dark-mode`) instead of media queries for user control
+- Defined dark mode colors for every UI element including highlights, progress bars, modals
+- Persisted theme preference in localStorage
+- Synchronized theme across all pages (library, reader, highlights)
+- Used `!important` for highlight colors to override inline styles
+
+**Code**: All template files - CSS dark mode sections
+
+## 8. TOC Auto-Scroll to Active Item
+
+**Challenge**: When opening a book mid-way through, the TOC sidebar didn't show the current chapter, requiring manual scrolling.
+
+**Solution**:
+- Calculate active TOC item position using `offsetTop`
+- Scroll sidebar to center the active item in viewport
+- Execute after DOM load to ensure elements are rendered
+
+**Code**: `templates/reader.html` - TOC auto-scroll in DOMContentLoaded
+
+## 9. Book Detection Without Naming Convention
+
+**Challenge**: Initially required `_data` suffix in folder names, limiting flexibility and creating ugly folder names.
+
+**Solution**:
+- Changed detection from filename pattern matching to presence of `book.pkl` file
+- Updated library scanning to check for file existence instead of name patterns
+- Maintained backward compatibility with old `_data` folders
+
+**Code**: `server.py` - `library_view()` function
+
+## 10. Whitespace-Tolerant Text Matching
+
+**Challenge**: Saved highlights couldn't be found when text spanned multiple paragraphs due to whitespace differences (newlines, multiple spaces).
+
+**Solution**:
+- Created regex pattern that replaces `\s+` in search text with `\s+` pattern
+- Allows flexible matching of any whitespace sequence
+- Escapes special regex characters in user text before pattern creation
+- Falls back to exact match first for performance
+
+**Code**: `templates/reader.html` - `findTextRange()` function
+
+---
+
+## Key Technologies Used
+
+- **FastAPI**: Async web framework with automatic API documentation
+- **SQLite**: Lightweight database for highlights and progress
+- **ebooklib**: EPUB parsing and extraction
+- **BeautifulSoup**: HTML processing and cleaning
+- **MathJax**: Mathematical equation rendering
+- **Marked.js**: Markdown rendering for AI responses
+- **Jinja2**: Server-side templating
+- **Vanilla JavaScript**: No framework dependencies for frontend
+
+## Architecture Decisions
+
+1. **Server-side rendering** for initial page load (SEO-friendly, fast first paint)
+2. **Client-side interactivity** for highlights and AI features (responsive UX)
+3. **SQLite for data** (simple, portable, no separate database server)
+4. **Pickle for book data** (fast serialization, preserves Python objects)
+5. **localStorage for preferences** (theme, font settings persist across sessions)
+6. **Event-driven progress saving** (reliable, doesn't interfere with reading)
+
+## Performance Optimizations
+
+- **LRU cache** for book loading (avoid repeated disk reads)
+- **Lazy AI service initialization** (only load when needed)
+- **Async/await** throughout (non-blocking I/O)
+- **keepalive flag** on fetch requests (ensures completion on page unload)
+- **Debounced scroll tracking** (via event listener, not polling)
+
+---
+
+*This document serves as a reference for understanding the technical depth and problem-solving approaches used in this project.*