BE-20: Local Query Processing via LLaMA 3 Model

[tecnodeveloper]

Description:
Implement how user queries are processed locally using LLaMA 3. Input comes from chat → backend builds context → model runs locally → response is generated → cleaned → stored → sent back to frontend.

---

User Story

Given user sends a message
When backend receives it
Then query should be processed locally using LLaMA 3 and return a contextual response

---

Tasks

---

Local Model Execution Setup

1. Ensure Local Model is Running

   • Load LLaMA 3
   • Verify inference works locally
   • Confirm GPU/CPU support

2. Set Execution Environment

   • Configure RAM/GPU limits
   • Optimize runtime settings

---

Query Processing Pipeline

3. Define Input Flow

   • User message received
   • Session context loaded
   • Prompt constructed

4. Create Processing Layer

   • /app/services/query_processor.py
   • Handle full request flow

---

Prompt Construction

5. Build Structured Prompt

   • System instruction (chat behavior rules)
   • Session history
   • Latest user query

6. Context Filtering

   • Keep last N messages
   • Remove irrelevant history

---

Local Inference Execution

7. Run Model Inference

   • Pass prompt to LLaMA 3
   • Generate response
   • Handle token limits

8. Control Output Quality

   • Prevent hallucination (basic guardrails)
   • Ensure relevant responses

---

Response Processing

9. Clean Model Output

   • Remove unwanted tokens
   • Fix formatting issues
   • Normalize text

10. Post-Processing Rules

• Trim long responses
• Ensure readability

---

Session Integration

11. Attach Session Context

• Link response to session_id
• Maintain conversation continuity

12. Store Conversation

• Save user + assistant messages
• Update MongoDB session

---

Performance Optimization

13. Reduce Latency

• Limit prompt size
• Cache frequent responses (optional)
• Optimize inference calls

14. Efficient Memory Use

• Avoid redundant context loading
• Streamline token usage

---

Error Handling

15. Model Failures

• Handle crash or timeout
• Retry mechanism

16. Fallback Response

• Return safe message if model fails
• Log error for debugging

---

Logging & Monitoring

17. Track Queries

• Log user input
• Log model output
• Track response time

18. Debug Information

• Enable debug mode
• Store inference metadata

---

Postman Testing 🧪

19. Setup Postman

• Test /chat/message endpoint

20. Validate Processing

• Send query
• Check model response
• Verify context usage

---

Frontend Integration

21. Display Response

• Show processed answer
• Auto-scroll chat

22. Loading State

• Show "thinking..." indicator
• Replace with final response

---

Acceptance Criteria

• Local LLaMA 3 query processing works
• Context-aware responses generated
• MongoDB session updated
• Clean response returned to UI
• Postman testing completed
• Stable inference pipeline

---

Testing Steps

1. Start local model
2. Send API request
3. Verify response generation
4. Check session memory usage
5. Test multiple turns
6. Measure latency

---

Definition of Done

• Local query processing fully working
• LLaMA 3 integrated into pipeline
• Context-aware chat functioning
• Backend stable and optimized