Add per-project storage and PyCharm API integration

Co-authored-by: Mte90 <403283+Mte90@users.noreply.github.com>

Author: Copilot

PYCHARM_INTEGRATION.md

@@ -0,0 +1,333 @@
+# PyCharm Plugin Integration Guide
+
+This document describes the REST API for integrating PicoCode with PyCharm and other IDEs.
+
+## Overview
+
+PicoCode provides a production-ready local RAG backend with per-project persistent storage. Each project gets its own SQLite database for isolation, and the API is designed to be compatible with IDE plugins.
+
+## API Endpoints
+
+### Base URL
+```
+http://127.0.0.1:8000/api
+```
+
+### Health Check
+```http
+GET /api/health
+```
+
+Returns server status and available features.
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "version": "0.2.0",
+  "features": ["rag", "per-project-db", "pycharm-api"]
+}
+```
+
+### Project Management
+
+#### Create/Get Project
+```http
+POST /api/projects
+Content-Type: application/json
+
+{
+  "path": "/absolute/path/to/project",
+  "name": "Optional Project Name"
+}
+```
+
+Creates a new project or returns existing one. Each project gets its own database.
+
+**Response:**
+```json
+{
+  "id": "1234567890abcdef",
+  "name": "MyProject",
+  "path": "/absolute/path/to/project",
+  "database_path": "~/.picocode/projects/1234567890abcdef.db",
+  "created_at": "2025-11-06T14:30:00",
+  "last_indexed_at": null,
+  "status": "created",
+  "settings": null
+}
+```
+
+#### List All Projects
+```http
+GET /api/projects
+```
+
+Returns list of all registered projects.
+
+**Response:**
+```json
+[
+  {
+    "id": "1234567890abcdef",
+    "name": "MyProject",
+    "path": "/absolute/path/to/project",
+    "status": "ready",
+    ...
+  }
+]
+```
+
+#### Get Project Details
+```http
+GET /api/projects/{project_id}
+```
+
+Returns details for a specific project.
+
+#### Delete Project
+```http
+DELETE /api/projects/{project_id}
+```
+
+Deletes project and its database.
+
+**Response:**
+```json
+{
+  "success": true
+}
+```
+
+### Indexing
+
+#### Index Project
+```http
+POST /api/projects/index
+Content-Type: application/json
+
+{
+  "project_id": "1234567890abcdef"
+}
+```
+
+Starts background indexing of the project. This processes all files, generates embeddings, and stores them in the project's database.
+
+**Response:**
+```json
+{
+  "status": "indexing",
+  "project_id": "1234567890abcdef"
+}
+```
+
+### Code Intelligence
+
+#### Semantic Search
+```http
+POST /api/query
+Content-Type: application/json
+
+{
+  "project_id": "1234567890abcdef",
+  "query": "How does authentication work?",
+  "top_k": 5
+}
+```
+
+Performs semantic search across the indexed project.
+
+**Response:**
+```json
+{
+  "results": [
+    {
+      "file_id": 123,
+      "path": "src/auth.py",
+      "chunk_index": 0,
+      "score": 0.8542
+    }
+  ],
+  "project_id": "1234567890abcdef",
+  "query": "How does authentication work?"
+}
+```
+
+#### Code Completion / Question Answering
+```http
+POST /api/code
+Content-Type: application/json
+
+{
+  "project_id": "1234567890abcdef",
+  "prompt": "Explain the authentication flow",
+  "context": "",
+  "use_rag": true,
+  "top_k": 5
+}
+```
+
+Gets code suggestions or answers using RAG + LLM.
+
+**Response:**
+```json
+{
+  "response": "The authentication flow works as follows...",
+  "used_context": [
+    {
+      "path": "src/auth.py",
+      "score": 0.8542
+    }
+  ],
+  "project_id": "1234567890abcdef"
+}
+```
+
+## PyCharm Plugin Workflow
+
+### 1. On Project Open
+When a project is opened in PyCharm:
+```
+1. POST /api/projects with project path
+2. Store returned project_id
+3. Check if project needs indexing (status != "ready")
+4. If needed, POST /api/projects/index
+```
+
+### 2. Code Assistance
+When user requests code help:
+```
+1. POST /api/code with prompt and project_id
+2. Display response in IDE
+3. Show used_context sources if available
+```
+
+### 3. Semantic Search
+When user searches for code:
+```
+1. POST /api/query with search term and project_id
+2. Display matching files and scores
+3. Allow navigation to results
+```
+
+### 4. Background Monitoring
+Poll project status periodically:
+```
+1. GET /api/projects/{project_id}
+2. Check status field
+3. Update UI indicators
+```
+
+## Configuration
+
+Create a `.env` file with:
+
+```bash
+# API endpoint for embeddings and LLM
+API_URL=https://api.openai.com/v1/
+API_KEY=your-api-key-here
+
+# Model names
+EMBEDDING_MODEL=text-embedding-3-small
+CODING_MODEL=gpt-4o
+
+# Server configuration
+UVICORN_HOST=127.0.0.1
+UVICORN_PORT=8000
+
+# File processing
+MAX_FILE_SIZE=200000
+```
+
+## Error Handling
+
+All endpoints return standard HTTP status codes:
+- 200: Success
+- 400: Bad request (validation error)
+- 404: Resource not found
+- 500: Server error
+
+Error responses include a JSON object:
+```json
+{
+  "error": "Description of the error"
+}
+```
+
+## Security Considerations
+
+1. **Local-only**: Server binds to 127.0.0.1 by default (localhost only)
+2. **File access**: Only indexed files are accessible
+3. **Path validation**: Project paths are validated and normalized
+4. **Database isolation**: Each project has separate database
+
+## Performance
+
+- **Concurrent indexing**: Uses thread pool for parallel file processing
+- **Connection pooling**: Efficient SQLite connection management
+- **Caching**: Analysis results cached to reduce database queries
+- **Vector search**: Uses sqlite-vector for fast similarity search
+
+## Limitations
+
+1. Maximum file size: 200KB by default (configurable)
+2. Text files only (binary files skipped)
+3. Requires OpenAI-compatible embedding API
+4. SQLite database per project (not suitable for extremely large projects)
+
+## Example Integration (Python)
+
+```python
+import requests
+
+class PicoCodeClient:
+    def __init__(self, base_url="http://127.0.0.1:8000/api"):
+        self.base_url = base_url
+    
+    def create_project(self, path, name=None):
+        response = requests.post(
+            f"{self.base_url}/projects",
+            json={"path": path, "name": name}
+        )
+        return response.json()
+    
+    def index_project(self, project_id):
+        response = requests.post(
+            f"{self.base_url}/projects/index",
+            json={"project_id": project_id}
+        )
+        return response.json()
+    
+    def query(self, project_id, query, top_k=5):
+        response = requests.post(
+            f"{self.base_url}/query",
+            json={
+                "project_id": project_id,
+                "query": query,
+                "top_k": top_k
+            }
+        )
+        return response.json()
+    
+    def get_code_suggestion(self, project_id, prompt, use_rag=True):
+        response = requests.post(
+            f"{self.base_url}/code",
+            json={
+                "project_id": project_id,
+                "prompt": prompt,
+                "use_rag": use_rag
+            }
+        )
+        return response.json()
+
+# Usage
+client = PicoCodeClient()
+project = client.create_project("/path/to/my/project", "MyProject")
+client.index_project(project["id"])
+results = client.query(project["id"], "authentication flow")
+suggestion = client.get_code_suggestion(project["id"], "Explain auth")
+```
+
+## Support
+
+For issues or questions, please refer to the main PicoCode repository.

README.md

@@ -9,11 +9,29 @@ This tool is a way to achieve this!
 
 ## Overview
 
-- Indexes files, computes embeddings using an OpenAI-compatible embedding endpoint, stores data in SQLite (with SQlite-vector).
-- Reads dependencies by running `python -m pip list --format=json` inside a virtualenv when available.
-- Detects Astral "uv" usage (https://docs.astral.sh/uv/) by inspecting `pyproject.toml` and/or installed packages in a venv; if uv is detected it tries to locate a venv managed by uv and uses it for `pip list`.
-- Analysis runs asynchronously (FastAPI BackgroundTasks) so the UI remains responsive.
-- Minimal web UI for starting analysis and asking questions (semantic search + coding model).
+- **Production-ready RAG backend** with per-project persistent storage
+- **PyCharm/IDE integration** via REST API (see [PYCHARM_INTEGRATION.md](PYCHARM_INTEGRATION.md))
+- **Per-project databases**: Each project gets isolated SQLite database
+- Indexes files, computes embeddings using an OpenAI-compatible embedding endpoint
+- Stores vector embeddings in SQLite using sqlite-vector for fast semantic search
+- Analysis runs asynchronously (FastAPI BackgroundTasks) so the UI remains responsive
+- Minimal web UI for starting analysis and asking questions (semantic search + coding model)
+- Health check and monitoring endpoints for production deployment
+
+## New Features (v0.2.0)
+
+### Per-Project Persistent Storage
+- Each opened project gets its own SQLite database
+- Isolated storage prevents cross-project data leakage
+- Project registry tracks all indexed projects
+- Automatic project ID generation based on path
+
+### PyCharm Plugin API
+- RESTful API designed for IDE integration
+- Create/manage projects via API
+- Background indexing with status tracking
+- Semantic search and code completion endpoints
+- See [PYCHARM_INTEGRATION.md](PYCHARM_INTEGRATION.md) for full API documentation
 
 Prerequisites
 - Python 3.8+ (3.11+ recommended for builtin tomllib)