Run returning obfuscated error (400 error given a valid request) #94
Description
I wanted to run evaluations for this model https://replicate.com/cuuupid/gte-qwen2-7b-instruct.
But I kept getting this error when I triggered larger number of "run" requests:
Unknown Error: invalid character '<' looking for beginning of value
I couldn't get the info I needed from the default client error response, so I forked the repo to add more debug info: bartolomej#1
This is the full http body sent to the API:
{
"input": {
"text": [
"About 300 words of normal sanitized text (without any special characters/newlines/...."
]
},
"version": "67b1736bae9312a321217b2e10547882943b9e4a285eac4cba4043fab954b954"
}And this is the full response body:
<html>
<head>
<title>400 Bad Request</title>
</head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr>
<center>cloudflare</center>
</body>
</html>Example Go code I used:
func (r *EmbeddingModel) Create(ctx context.Context, reqs []types.CreateEmbeddingRequest) ([][]float32, error) {
text := make([]string, 0, len(reqs))
for _, r := range reqs {
// r.Instruct and r.Query were both sanitized, without any special characters or newlines
text = append(text, fmt.Sprintf("Instruct: %s Query: %s", r.Instruct, r.Query))
}
res, err := r.client.Run(ctx, r.modelID, map[string]any{
"text": text,
}, nil)
if err != nil {
return nil, errors.Wrap(err, "replicate run")
}
// ... irrelevant response parsing code ...
return out, nil
}Then I refactored my code to send the request via barebones http client and started receiving 429 (error limit) responses:
payload := map[string]interface{}{
"version": r.modelVersion,
"input": map[string]interface{}{
"text": text,
},
}
body, err := json.Marshal(payload)
if err != nil {
return nil, errors.Wrap(err, "marshal json")
}
req, err := http.NewRequestWithContext(ctx, "POST", "https://api.replicate.com/v1/predictions", bytes.NewReader(body))
if err != nil {
return nil, errors.Wrap(err, "build request")
}
req.Header.Set("Authorization", "Bearer "+r.apiToken)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Prefer", "wait")
client := &http.Client{Timeout: 60 * time.Second}
resp, err := client.Do(req)
if err != nil {
return nil, errors.Wrap(err, "http request")
}
defer resp.Body.Close()
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return nil, errors.Wrap(err, "read response body")
}In summary:
- I'm I just misusing the replicate client (doesn't seem like it)?
- ... or is the Replicate API / proxy returning some weird responses for certain requests (400 instead of 429)?
- I feel like these edge case responses could be better handled to include more debug info for cases like these