Skip to content

Add get_user tool to codebase #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
danenania wants to merge 1 commit into from
Closed

Add get_user tool to codebase #8

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 114 additions & 16 deletions src/routes/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,50 @@ import { getSystemPrompt } from '../domains';

const LITELLM_SERVER_URL = process.env.LITELLM_SERVER_URL || 'http://localhost:4000';

// Tool definitions for AI function calling
const TOOLS = [
{
type: 'function',
function: {
name: 'get_user',
description: 'Get information about the current authenticated user including their ID, role, and token details',
parameters: {
type: 'object',
properties: {},
required: []
}
}
}
];

// Execute tool calls and return results
function executeToolCall(toolName: string, args: any, req: Request): string {
switch (toolName) {
case 'get_user':
if (req.user) {
return JSON.stringify({
userId: req.user.sub,
role: req.user.role || 'unknown',
issuer: req.user.iss,
audience: req.user.aud,
issuedAt: req.user.iat ? new Date(req.user.iat * 1000).toISOString() : null,
expiresAt: req.user.exp ? new Date(req.user.exp * 1000).toISOString() : null,
scope: req.user.scope || null
});
Comment on lines +28 to +36
Copy link

@promptfoo-scanner promptfoo-scanner bot Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MediumThe get_user tool sends user PII (user ID, role, token issuer, audience, timestamps, and scope) to the external LiteLLM service. While this appears to be a single-user context where users access only their own data, the token metadata fields (issuer, audience, issuedAt, expiresAt, scope) may be unnecessary for typical chat personalization and increase the PII footprint sent to the LLM provider.

💡 Suggested Fix

Apply data minimization by limiting the tool to essential fields only:

case 'get_user':
  if (req.user) {
    return JSON.stringify({
      userId: req.user.sub,
      role: req.user.role || 'unknown',
      // Removed: issuer, audience, issuedAt, expiresAt, scope
      // Token metadata typically not needed for chat personalization
    });
  } else {
    return JSON.stringify({
      error: 'No authenticated user',
      message: 'This endpoint requires authentication to retrieve user information'
    });
  }

Additionally, ensure your LiteLLM deployment has appropriate data protection agreements in place, and consider updating your privacy policy to disclose that user data may be sent to the LLM provider during chat interactions.

🤖 AI Agent Prompt

The get_user tool at src/routes/chat.ts:28-36 exposes user PII including JWT token metadata (issuer, audience, timestamps, scope) to an external LiteLLM service. Investigate whether all these fields are necessary for the chat personalization use case. Check if there are any privacy policy or user consent mechanisms that cover this data sharing. Consider whether data minimization would be appropriate - typically only userId and role are needed for personalization. Also investigate the LiteLLM deployment model (self-hosted vs. cloud) and whether data processing agreements are in place. If this is a self-hosted deployment, the risk may be lower. Look for any application-wide patterns for handling PII in LLM contexts.

Comment on lines +28 to +36
Copy link

@promptfoo-scanner promptfoo-scanner bot Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MediumThe get_user tool sends user authentication data (user ID, role, JWT claims) to external LLM providers via LiteLLM. When the LLM calls this tool, the results are added to the message history and forwarded to third-party APIs (OpenAI, Anthropic, etc.). This exposes user identifiers and authentication metadata to external parties, which could raise compliance concerns under GDPR or CCPA if users haven't consented to this data sharing.

💡 Suggested Fix

Consider anonymizing user data before sending it to the LLM. Instead of real user IDs, use session-scoped hashes and generalized role categories:

case 'get_user':
  if (req.user) {
    return JSON.stringify({
      // Use session-scoped anonymous ID instead of actual user ID
      sessionId: generateSessionHash(req.user.sub, req.sessionID),
      // Generalize role to broader categories
      roleCategory: categorizeRole(req.user.role),
      authenticated: true
    });
  } else {
    return JSON.stringify({
      error: 'No authenticated user',
      message: 'This endpoint requires authentication to retrieve user information'
    });
  }

// Helper functions
function generateSessionHash(userId: string, sessionId: string): string {
  const crypto = require('crypto');
  return crypto.createHash('sha256')
    .update(`${userId}-${sessionId}`)
    .digest('hex')
    .substring(0, 16);
}

function categorizeRole(role: string | undefined): string {
  const adminRoles = ['admin', 'superadmin', 'owner'];
  const powerRoles = ['moderator', 'editor', 'contributor'];
  if (!role) return 'user';
  if (adminRoles.includes(role.toLowerCase())) return 'admin';
  if (powerRoles.includes(role.toLowerCase())) return 'power-user';
  return 'user';
}

Alternatively, if user context isn't essential for the chat functionality, consider removing the tool entirely to avoid any PII exposure.

🤖 AI Agent Prompt

The code at src/routes/chat.ts:28-36 sends user authentication data (user ID from req.user.sub, role, JWT issuer, audience, and timestamps) to external LLM providers. This happens when the LLM calls the get_user tool - the tool results are added to messages at lines 224-228 and then sent to LiteLLM at line 186, which forwards to external APIs.

Your task is to investigate and implement a privacy-preserving solution:

  1. First, determine if the user context provided by get_user is actually necessary for the application's chat functionality. Review how the LLM uses this information and whether it's essential for the user experience.

  2. If user context IS needed, implement anonymization:

    • Replace actual user IDs with session-scoped hashes that can't be used to identify users across sessions
    • Generalize sensitive fields like roles into broader categories
    • Remove JWT metadata that isn't needed by the LLM (issuer, audience, timestamps)
    • Consider what minimal context the LLM actually needs

  3. If user context is NOT needed, remove the tool entirely to eliminate PII exposure.

  4. Check if there are existing privacy utilities in the codebase that could be leveraged or extended for this purpose.

  5. Consider the application's architecture: Is LiteLLM configured to use self-hosted models or external cloud providers? If using cloud providers, check if data processing agreements are in place that cover this use case.

The goal is to minimize PII exposure to external parties while maintaining the functionality that users need.

Comment on lines +28 to +36
Copy link

@promptfoo-scanner promptfoo-scanner bot Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium

The get_user tool sends user identifiers (including the user ID from req.user.sub) to the external LLM provider. While this is in a same-user context (each user only sees their own data), sending PII to a third-party LLM service creates privacy exposure depending on the provider's data handling practices, logging, and whether they have data protection agreements in place.

💡 Suggested Fix

Consider minimizing the data sent to the LLM. If per-user personalization isn't required, send only the role instead of the user ID:

case 'get_user':
  if (req.user) {
    return JSON.stringify({
      role: req.user.role || 'unknown',
      authenticated: true
    });
  } else {
    return JSON.stringify({
      error: 'No authenticated user',
      message: 'This endpoint requires authentication to retrieve user information'
    });
  }

Alternatively, if you need per-user context, hash the user ID to reduce PII exposure:

import { createHash } from 'crypto';

function hashUserId(userId: string): string {
  return createHash('sha256').update(userId).digest('hex').substring(0, 16);
}

// In the get_user case:
return JSON.stringify({
  userId: hashUserId(req.user.sub),
  role: req.user.role || 'unknown',
  // ... other fields
});
🤖 AI Agent Prompt

The code at src/routes/chat.ts:28-36 sends user PII (specifically the user ID from JWT claims) to an external LLM provider via the get_user tool. This creates privacy exposure as the data is sent to a third party.

Investigate the following to determine the appropriate fix:

  1. Check if user ID is actually needed: Review how the LLM responses use the user context. Look at system prompts (referenced by getSystemPrompt imported at line 3) and any existing chat functionality to understand if the LLM needs the actual user ID for personalization, or if just knowing the user's role would be sufficient.

  2. Assess current usage patterns: Search for other places in the codebase where user context is handled. Look for patterns around privacy, consent mechanisms, or data minimization principles that might already be established.

  3. Evaluate LLM provider configuration: Check environment configuration and documentation to understand which LLM provider is being used (via LiteLLM), whether data protection agreements exist, and what the data retention/logging policies are.

  4. Determine fix approach:

    • If user ID isn't needed for functionality, remove it and send only role/authentication status
    • If per-user personalization is required, implement hashing/anonymization of the user ID
    • If raw user ID is truly necessary, add configuration controls to disable the feature in environments without appropriate privacy safeguards

Start by examining the system prompt and chat context to understand what information the LLM actually needs to provide useful responses.

} else {
return JSON.stringify({
error: 'No authenticated user',
message: 'This endpoint requires authentication to retrieve user information'
});
}
default:
return JSON.stringify({
error: 'Unknown tool',
message: `Tool '${toolName}' is not available`
});
}
}

// Map fish names to internal security levels
const FISH_TO_LEVEL: Record<string, 'insecure' | 'secure'> = {
minnow: 'insecure',
Expand Down Expand Up @@ -116,34 +160,88 @@ export async function chatHandler(req: Request, res: Response): Promise<void> {

// Prepare LiteLLM request
const litellmRequest: any = {
messages: messages
messages: messages,
tools: TOOLS,
tool_choice: 'auto'
};

// Add model if provided
if (model) {
litellmRequest.model = model;
}

// Forward request to LiteLLM server
const response = await fetch(`${LITELLM_SERVER_URL}/v1/chat/completions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(litellmRequest),
});
// Tool call loop - continue until we get a final response
const MAX_TOOL_ITERATIONS = 10;
let iteration = 0;

if (!response.ok) {
const errorText = await response.text();
res.status(response.status).json({
error: 'LiteLLM server error',
message: errorText
while (iteration < MAX_TOOL_ITERATIONS) {
iteration++;

// Forward request to LiteLLM server
const response = await fetch(`${LITELLM_SERVER_URL}/v1/chat/completions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(litellmRequest),
});

if (!response.ok) {
const errorText = await response.text();
res.status(response.status).json({
error: 'LiteLLM server error',
message: errorText
});
return;
}

const data = await response.json();
const assistantMessage = data.choices?.[0]?.message;

// Check if the model wants to call tools
if (assistantMessage?.tool_calls && assistantMessage.tool_calls.length > 0) {
// Add the assistant's message with tool calls to the conversation
litellmRequest.messages.push({
role: 'assistant',
content: assistantMessage.content || null,
tool_calls: assistantMessage.tool_calls
});

// Execute each tool call and add results
for (const toolCall of assistantMessage.tool_calls) {
const toolName = toolCall.function.name;
let toolArgs = {};

try {
toolArgs = JSON.parse(toolCall.function.arguments || '{}');
} catch {
// If parsing fails, use empty args
}

const toolResult = executeToolCall(toolName, toolArgs, req);

// Add tool result to messages
litellmRequest.messages.push({
role: 'tool',
tool_call_id: toolCall.id,
content: toolResult
});
}

// Continue the loop to get the model's response to tool results
continue;
}

// No tool calls - return the final response
res.json(data);
return;
}

const data = await response.json();
res.json(data);
// If we hit max iterations, return an error
res.status(500).json({
error: 'Tool call limit exceeded',
message: `Maximum tool call iterations (${MAX_TOOL_ITERATIONS}) reached`
});
} catch (error) {
console.error('Error in chat handler:', error);
res.status(500).json({
Expand Down