Replies: 27 comments 31 replies
-
|
Some more on Ryzen 7940HS (on Framwork 16) + 64Go of RAM / Linux (fc40)
|
Beta Was this translation helpful? Give feedback.
-
|
Wow thank you for posting these numbers. Your AMD CPUs are a lot cheaper than the Intel Core i9-14900K yet have such better Q6_K and F16 performance. On your znver4 CPU you may want to try using BF16 weights and see how those go, since it should have special opcodes that make those weights go fast. |
Beta Was this translation helpful? Give feedback.
-
|
Only wait for "GPU" llamafile-benchmark it have a x2 on this CPU for more benchmark (#439 (comment)) |
Beta Was this translation helpful? Give feedback.
-
|
Wait a minute. Hold on a second. Am I correct in understanding you ran Mixtral 8x22b on a $362 CPU? And it processed your prompt at 20 tokens per second?! That's nuts. Here's what I get with that model.
|
Beta Was this translation helpful? Give feedback.
-
|
Yes... sorry for not having enough memory on the 7940HS to test it I really have get 96Go of RAM 😎 |
Beta Was this translation helpful? Give feedback.
-
|
Does 't/s' refer to the total computation time for inference? |
Beta Was this translation helpful? Give feedback.
-
|
no token/second for pp: the number of token depend a the size of the prompt to process |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
I really like to know what we can get with this CPU.
expect more than 1/2 compare with 7995WX on pp and same on tg. |
Beta Was this translation helpful? Give feedback.
-
|
What memory speed were these benchmarks run at for mixtral 8x22b Q6_K? I have the same machine, Ryzen 9 5950x with 128GB 3600mhz CL16 memory And got the following results for the same tests: |
Beta Was this translation helpful? Give feedback.
-
|
I am running an optimised profile for my memory, so I was wondering what the main discrepancy was between my results and your results. For added context I have two of these kits of memory: Patriot Viper Steel DDR4 64GB (2 x 32GB) 3600MHz Kit - PVS464G360C8K |
Beta Was this translation helpful? Give feedback.
-
|
Crucial Ballistix DDR4 3600MHz DIMM (CL16) |
Beta Was this translation helpful? Give feedback.
-
|
This one https://www.amazon.com/Patriot-Viper-Steel-DDR4-3600MHz/dp/B08688GFPD?th=1 |
Beta Was this translation helpful? Give feedback.
-
|
Run this command on your computers to get more clarity about the memory capabilities and performance. https://www.intel.com/content/www/us/en/developer/articles/tool/intelr-memory-latency-checker.html |
Beta Was this translation helpful? Give feedback.
-
|
for me: |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Can you guys explain what does the test column mean? |
Beta Was this translation helpful? Give feedback.
-
|
It's "the test being performed" which can be "process a prompt with 256 tokens" (pp256) or "generate 16 tokens" (tg16). |
Beta Was this translation helpful? Give feedback.
-
|
Oh I see. Thank you! I am trying to see how well just using CPU inference will do for me. The performance using llamafile seems promising for prompt processing, but then the text generation is still far from GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0429 llamafile]$ ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile"
|
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0206 llamafile]$ ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile" -t 128
|
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0206 llamafile]$ ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile" -t 128 --numa distribute
|
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0209 llamafile]$ NM_BIND_THREADS_TO_CORES=1 numactl --i=all ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile" -t 128 --numa distribute
|
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0209 llamafile]$ NM_BIND_THREADS_TO_CORES=8 numactl --i=all ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile" -t 128 --numa distribute
|
Beta Was this translation helpful? Give feedback.
-
|
[fbongiovanni@mel0209 llamafile]$ NM_BIND_THREADS_TO_CORES=16 numactl --i=all ./0.8.6/llamafile-bench-0.8.6 -p "256,512,1024" -m "mistral-7b-instruct-v0.2.Q6_K.llamafile" -t 128 --numa distribute
|
Beta Was this translation helpful? Give feedback.
-
|
#> mem stock 32Go DDR4@2666
#> mem XMP 32Go DDR4@3200
#> mem XMP 32Go DDR4@3600
|
Beta Was this translation helpful? Give feedback.
-
|
#> mem stock 32Go DDR4@2400
|
Beta Was this translation helpful? Give feedback.
-
|
32GB DDR4@3000
|
Beta Was this translation helpful? Give feedback.
-
@Djip007 any substantial performance changes with current llamafile for this model? Edit: found #367 |
Beta Was this translation helpful? Give feedback.
-
|
last bench is with V0.8.11/12 + CPU ...
|
Beta Was this translation helpful? Give feedback.
-
llamafile 0.8.12+
|
Beta Was this translation helpful? Give feedback.
-
|
last on for now...
|
Beta Was this translation helpful? Give feedback.
-
This is quite impressive. |
Beta Was this translation helpful? Give feedback.
-
|
@Djip007 maybe adjust the title of this discussion to reflect the contents? |
Beta Was this translation helpful? Give feedback.
-
|
Updated. |
Beta Was this translation helpful? Give feedback.
-
|
@jart I find it bonkers that we now can run these big models which is kinda out of reach even with 48GB of VRAM, and have decent (or close to it :-) performance. Very grateful for the work you and the rest of the team do. Hope you have a deep stack of ideas to try out to squeeze even more performance out of consumer hardware. :-) Meta-Llama-3-70B-Instruct.Q6_K Latest crop of AMD offerings (zen5 mobile, Strix Point) appear to top out at 120GBps if matched with LPDDR5X-7500. But desktop Zen 5 (Granite Ridge) isn't out until mid august. (in two weeks). Interesting times ahead. Would be interesting to see to what extent LLM performance scales with memory bandwidth and number of threads. (a 3D plot) |
Beta Was this translation helpful? Give feedback.
-
|
For zen4 llamafile get ~80% of bf16 theorical max FLOPS, (Zen3 pretty as close with his fp32 FLOPS). so don't hop for a new x2 speed up on this CPU. (I can get some 5/10 % more perf with test on BF16/zen4 CPU, but the code may be too specifique for other CPU gain...) with so high FLOPS hyperthreding is useless => nb_thread == nb_physical_core is the best I think and made the default on llamafile For Strix Point ... we need to use GPU or TPU ... for 7940HS+GPU we can have a x2 perf over CPU, but it is far from GPU pick FLOPS... (25% of max BF16/FP16 FLOPS...) but cache size is small on GPU ... for higher speed we may have to change ggml tensor (weight) structure. For TPUs it's probably even worse... Note: can be use to make nb_thread bench, but the thread "colone" is not reported in output... we need to "correct" that with llama-bench (llama.cpp bench) on Ryzen 7940HS (8coeur/16 thread)I have:
(and yes llamafile is 2x faster for pp ;) in that case...) |
Beta Was this translation helpful? Give feedback.
-
|
Benchmarks of recent
|
Beta Was this translation helpful? Give feedback.
-
|
tg16 3.91 -> 5.29 with Ryzen 7940HS -> HX 370 ! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, exactly 4 banks of: |
Beta Was this translation helpful? Give feedback.
-
|
But the best surprise is Deepseek Coder V2 Lite which is usable on AMD Ryzen AI 9 HX 370 for tab autocompletion model even on 8-bit quantization! Now I understand why they choose MoE with this model.
|
Beta Was this translation helpful? Give feedback.
-
|
so imagine if you can use your GPU! (x2/x4 is expected on pp ;) ) |
Beta Was this translation helpful? Give feedback.
-
|
with v0.8.13 now it work!:
=> only wait for FP8 support for this last model! |
Beta Was this translation helpful? Give feedback.
-
|
Here's some TinyLLaMA benchmarks with 0.8.13 on the flagship Threadripper.
|
Beta Was this translation helpful? Give feedback.
-
|
Also I know it isn't Ryzen, but here's TinyLLaMA on the Apple M2 Ultra with llamafile 0.8.13.
|
Beta Was this translation helpful? Give feedback.
-
|
nice to have result from other type of CPU. |
Beta Was this translation helpful? Give feedback.
-
|
I remove "Ryzen" on title |
Beta Was this translation helpful? Give feedback.
-
|
As you wish. |
Beta Was this translation helpful? Give feedback.
-
|
More benchmarks for you all with llamafile 0.8.13
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
i wonder if the new 9950x amd cpu's will improve things with their improved avx512 support, they seem to have improved some of the other AI related benchmarks - https://www.phoronix.com/review/amd-ryzen-9950x-9900x/13 |
Beta Was this translation helpful? Give feedback.
-
|
we may ask him to add bench with llamafile-bench on CPU... |
Beta Was this translation helpful? Give feedback.
-
|
Already did, although I did not mention that there already is a benchmark suite/script. |
Beta Was this translation helpful? Give feedback.
-
|
Did anyone run these tests on any EPYC setup with 8-channel RAM? |
Beta Was this translation helpful? Give feedback.
-
|
There is some benchmark on this page with EPYC CPU (I think all of this avec 8-channel.) |
Beta Was this translation helpful? Give feedback.
-
(For "history": bench with llamafile V0.8.6)
Some result with my zen3+128Go of RAM / Linux (fc40)
RAM: DDR4@3600
As you see matmul is memory limited on this CPU (DDR-4 + zne3)
Beta Was this translation helpful? Give feedback.
All reactions