[Feature]:zdd temporal litellm and more security

[abhinayychaudharyy]

🔖 Feature description

there is no security in fetching the api so adding the api security system

🎤 Why is this feature needed ?

No response

✌️ How do you aim to achieve this?

i am an open source contributing

👀 Have you searched issues and PRs to see if this feature request has been raised before?

• I checked and didn't find similar issue

[creatorrr]

Hey @abhinayychaudharyy , can you please add some more details on your proposal. What sort of security were you thinking of?

@claude - please look into this and investigate avenues to work on

[claude]

Claude finished @creatorrr's task —— View job

---

Security Analysis Complete

I've completed a comprehensive security analysis of the Julep platform. Here's my findings and recommendations for improving security across the API, Temporal, and LiteLLM components.

Current Security Implementation

✅ Positive Security Measures

1. Parameterized SQL Queries: All database queries use proper parameterization with placeholders ($1, $2) preventing SQL injection
2. Pydantic Input Validation: Request bodies are validated using Pydantic models with type checking
3. Secrets Encryption: Master key validation and encryption for stored secrets
4. Temporal TLS Support: Proper TLS and certificate-based authentication for Temporal workflows
5. Error Handling: Typed exceptions and proper error responses without exposing internal details

⚠️ Security Gaps Identified

1. API Authentication & Authorization

• Issue: Single shared API key for all users
• Risk: No user isolation, compromised key affects all users
• Recommendation:
  • Implement per-user API keys or JWT tokens
  • Add role-based access control (RBAC)
  • Implement API key rotation mechanism

2. Rate Limiting

• Issue: No rate limiting middleware
• Risk: Vulnerable to DoS attacks and resource exhaustion
• Recommendation:
  • Implement rate limiting middleware (e.g., slowapi for FastAPI)
  • Add per-endpoint and per-IP rate limits
  • Enforce the configured transition_requests_per_minute

3. CORS Configuration

• Issue: Overly permissive (allow_origins=["*"])
• Risk: Cross-origin attacks possible
• Recommendation:
  • Restrict to specific allowed origins
  • Configure CORS based on environment (dev/staging/prod)

4. API Key Management

• Issue: API keys stored as plain environment variables
• Risk: Keys exposed in logs, process listings
• Recommendation:
  • Use a secrets manager (HashiCorp Vault, AWS Secrets Manager)
  • Implement key encryption at rest
  • Add audit logging for key usage

5. LiteLLM Proxy Security

• Issue: No per-user quota management or audit logging
• Risk: Uncontrolled LLM usage, cost overruns
• Recommendation:
  • Implement per-user API key management
  • Add usage tracking and quotas
  • Enable audit logging for all LLM calls

Recommended Security Improvements

Priority 1 (Critical)

1. Implement JWT/OAuth2 Authentication

   # Example using FastAPI-Users or authlib
   from fastapi_users import FastAPIUsers
   from fastapi_users.authentication import JWTStrategy

2. Add Rate Limiting

   from slowapi import Limiter, _rate_limit_exceeded_handler
   from slowapi.util import get_remote_address
   
   limiter = Limiter(key_func=get_remote_address)
   app.state.limiter = limiter
   app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)

3. Restrict CORS

   app.add_middleware(
       CORSMiddleware,
       allow_origins=["https://app.julep.ai", "http://localhost:3000"],
       allow_credentials=True,
       allow_methods=["GET", "POST", "PUT", "DELETE"],
       allow_headers=["Authorization", "Content-Type"],
   )

Priority 2 (High)

1. Add Security Headers

   from secure import SecureHeaders
   secure_headers = SecureHeaders()
   
   @app.middleware("http")
   async def set_secure_headers(request, call_next):
       response = await call_next(request)
       secure_headers.framework.fastapi(response)
       return response

2. Implement Audit Logging

   • Log all API access with user, endpoint, timestamp
   • Track LLM usage per user
   • Monitor failed authentication attempts

3. Add Input Sanitization for XSS

   import bleach
   
   def sanitize_html(content: str) -> str:
       return bleach.clean(content, tags=[], strip=True)

Priority 3 (Medium)

1. Implement API Versioning
2. Add request signing for webhook endpoints
3. Implement field-level encryption for sensitive data
4. Add IP allowlisting for admin endpoints
5. Implement session management with timeout

Implementation Roadmap

1. Phase 1 (Week 1-2): Authentication & Rate Limiting

   • Replace single API key with JWT authentication
   • Implement basic rate limiting
   • Fix CORS configuration

2. Phase 2 (Week 3-4): Security Headers & Audit

   • Add security headers middleware
   • Implement audit logging
   • Add monitoring dashboards

3. Phase 3 (Week 5-6): Advanced Security

   • Secrets management integration
   • Per-user LLM quotas
   • Advanced rate limiting rules

Summary

While Julep has some good security foundations (parameterized queries, input validation, secrets encryption), there are critical gaps in authentication, authorization, and rate limiting that should be addressed. The single shared API key pattern is the most critical issue that needs immediate attention.

Would you like me to start implementing any of these security improvements? I recommend beginning with JWT authentication and rate limiting as they address the most critical vulnerabilities.