Refactor fetch_official_vectors.py for improvements#26
Refactor fetch_official_vectors.py for improvements#26krataratha wants to merge 1 commit intoantirez:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors tests/test-vectors/fetch_official_vectors.py, expanding it from a simple sequential fetcher into a more featureful tool that adds retry behavior, parallel fetching, additional metadata in emitted JSON, and gzip exports for the official vector artifacts.
Changes:
- Add retry logic to the API call path and factor request payload construction into a helper.
- Parallelize vector fetching via
ThreadPoolExecutor, and emit additional metadata (e.g., prompt hash, basic stats, generated timestamps). - Add gzip export support and adjust the compact
official.vecgenerator output.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| output.append( | ||
| f"case {prompt['id']} {CTX_BY_ID[prompt['id']]} {len(record['steps'])}" | ||
| ) |
There was a problem hiding this comment.
Good catch — this is a real compatibility regression. The current output format drops the prompt file path, which breaks tests/ds4_test.c:test_read_vector_case, since the parser explicitly expects 4 fields in the case line: case <prompt_path>. Without that field, the generated .vec file becomes non-compliant and will fail --logprob-vectors validation. I will fix this by restoring prompt['prompt_file'] in the case output and ensuring it is consistently included across all generated fixtures, so the format remains fully aligned with the parser contract and test specification.
| futures = [ | ||
| executor.submit(fetch_and_store, api_key, spec, root) | ||
| for spec in selected | ||
| ] | ||
|
|
||
| for future in concurrent.futures.as_completed(futures): | ||
| manifest["prompts"].append(future.result()) | ||
|
|
| The API exposes top-logprobs, not full logits. These vectors are therefore | ||
| golden continuation slices: useful for catching tokenizer/template/attention | ||
| regressions, but not a replacement for a full internal logit dump. | ||
| """Fetch DeepSeek V4 Flash logprob vectors with retries, caching, gzip export, |
No description provided.