Rendered at 03:36:57 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
viraptor 19 hours ago [-]
That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
8 hours ago [-]
AmSach 8 hours ago [-]
[flagged]
8 hours ago [-]
aetherspawn 18 hours ago [-]
Does it improve tps for Mac?
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
hxii 19 hours ago [-]
Can the same process be theoretically repeated with LFM2.5-24B-A2B ?
8 hours ago [-]
tamnv 20 hours ago [-]
I really like topics about optimization like this one. Great job, OP!
peter_d_sherman 22 hours ago [-]
Looks absolutely stellar!
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!