BE-19: LLaMA 3 Integration as Core Chatbot Engine

[tecnodeveloper]

Description:
Integrate local LLaMA 3 model as the main chatbot brain. User message → backend → session context → LLaMA 3 → response → stored in MongoDB → returned to UI.

---

User Story

Given user sends a chat message
When backend processes request
Then LLaMA 3 should generate a contextual response

---

Tasks

---

Model Setup

1. Install Model Runtime

   • Install required dependencies (torch / transformers / llama.cpp)
   • Set up local environment

2. Load LLaMA 3

   • Initialize LLaMA 3
   • Load model weights
   • Verify model runs locally

---

Backend Integration

3. Create Model Service Layer

   • /app/services/model_service.py
   • Handle inference logic only

4. Define Inference Function

   • Input: prompt + context
   • Output: generated response

---

Prompt Engineering (Minimal)

5. Build Prompt Format

   • System instruction (optional)
   • Conversation history
   • Latest user message

6. Context Injection

   • Attach session messages
   • Maintain chat flow

---

Chat Pipeline Integration

7. Connect with Message Router

   • Receive message from /chat/message
   • Pass to session handler
   • Send formatted prompt to model

8. Return Model Response

   • Clean output
   • Remove noise/tokens
   • Return final answer

---

Session Awareness

9. Use Session Context

   • Load last N messages
   • Maintain conversation memory
   • Pass into model input

10. Update Session After Response

• Save assistant reply
• Update timestamp

---

Performance Optimization

11. Reduce Latency

• Limit context size
• Trim long history
• Optimize token usage

12. Model Efficiency

• Use quantized model (if possible)
• Reduce inference overhead

---

Streaming Response (Optional Upgrade)

13. Enable Streaming Output

• Token-by-token response
• Real-time UI updates

14. Frontend Sync

• Show typing effect
• Stream message live

---

Error Handling

15. Model Failures

• Model not loaded
• Timeout handling
• Memory errors

16. Fallback Response

• Return safe message
• Log error

---

Logging & Debugging

17. Track Inference Logs

• Input prompt
• Output response
• Response time

18. Debug Mode

• Enable detailed logs
• Track token usage

---

Postman Testing 🧪

19. Setup Postman

• Test /chat/message endpoint

20. Validate Model Response

• Send sample prompt
• Verify LLaMA output
• Check latency

---

Frontend Integration

21. Display Response

• Render bot message
• Auto-scroll chat

22. Typing Indicator

• Show "thinking..." state
• Replace with response

---

Acceptance Criteria

• LLaMA 3 successfully integrated
• Model generates contextual responses
• Session memory used correctly
• MongoDB stores responses
• Postman testing completed
• Frontend receives output

---

Testing Steps

1. Start model locally
2. Send API request
3. Verify response quality
4. Check session context usage
5. Test multiple turns
6. Measure response time

---

Definition of Done

• LLaMA 3 fully integrated
• Chatbot engine working
• Context-aware responses enabled
• Backend pipeline stable