LP: vllm benchmarking with quantised models#3207
Open
almayne wants to merge 5 commits intoArmDeveloperEcosystem:mainfrom
Open
LP: vllm benchmarking with quantised models#3207almayne wants to merge 5 commits intoArmDeveloperEcosystem:mainfrom
almayne wants to merge 5 commits intoArmDeveloperEcosystem:mainfrom
Conversation
Signed-off-by: Anna Mayne <anna.mayne@arm.com>
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
nSircombe
reviewed
Apr 24, 2026
fadara01
reviewed
Apr 24, 2026
fadara01
left a comment
fadara01
left a comment
There was a problem hiding this comment.
Thank you for your work!
I added some initial comments
| lm_eval --model vllm --model_args pretrained=RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8,dtype=bfloat16,max_model_len=4096 --tasks gsm8k --batch_size 4 --limit 10 | ||
| ``` | ||
|
|
||
| We would expect to see the precision is slightly lower with INT8. |
There was a problem hiding this comment.
we should expect to see numbers similar to the ones reported here for int8: https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8
|
|
||
| ## Set up access to LLama3.1-8B models | ||
|
|
||
| To access the Llama models hosted by Hugging Face, you will need to install the Hugging Face cli so that you can authenticate yourself and the harness can download what it needs. You should create an account on https://huggingface.co/ and follow the instructions [in the Hugging Face cli guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli) to set up your access token. You can then install the cli and login: |
There was a problem hiding this comment.
is it worth adding an instruction that you should also sign the licence agreement etc for the meta-llama model?
Author
There was a problem hiding this comment.
Requesting access to the model is covered in the paragraph below. Is there an additional step I've forgotten?
| * Accuracy: --limit mmlu=10,gsm8k=500 | ||
|
|
||
| ### Throughput ratios: INT8/BF16 | ||
| | Requests/s | Total Tokens/s | Output Tokens/s | |
There was a problem hiding this comment.
given that we ran a serving benchmark, I think we should report latency here too.
fadara01
reviewed
Apr 24, 2026
fadara01
reviewed
Apr 24, 2026
nikhil-arm
reviewed
Apr 25, 2026
Contributor
nikhil-arm
left a comment
nikhil-arm
left a comment
There was a problem hiding this comment.
I think we need to redo the inference and benchmarking pages from scratch.
Also I did not find any mention of whisper which was one of the requirement if I understand correctly
Llama and/or Whisper I think. |
Signed-off-by: Anna Mayne <anna.mayne@arm.com>
Signed-off-by: Anna Mayne <anna.mayne@arm.com>
Signed-off-by: Anna Mayne <anna.mayne@arm.com>
nikhil-arm
reviewed
Apr 28, 2026
nikhil-arm
reviewed
Apr 28, 2026
nikhil-arm
suggested changes
Apr 28, 2026
…page to use custom scripts and added whisper inference back in. Accuracy results in benchmarking page are now full runs. Signed-off-by: Anna Mayne <anna.mayne@arm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before submitting a pull request for a new Learning Path, please review Create a Learning Path
Please do not include any confidential information in your contribution. This includes confidential microarchitecture details and unannounced product information.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the Creative Commons Attribution 4.0 International License.