My biggest learning after some experiments - a BF16 (unquantized) Qwen beats a Q8 of double its size for decisions. I guess that’s the reason Kev switched to 4B BF16, from the original 8B version. Isn’t it interesting that quantization seems to mess with decision accuracy?
That’s fascinating, but not that surprising to me. We act like quantisation is free “Q8 is basically lossless” is often said in the local LLM community, but it really isn’t. The trade offs are worth it, personally, and the damage to coding ability seems low: decision model approaches are stricter though
„Sweet solution“ - anyone having tasted pure vanillin in a nontrivial dose, will tell you that it tastes absolutely disgusting. It only works in traces.
This is true for many (most?) flavor compounds. A little salt is nice, a teaspoon isn't.
Some sort of clove extract is used in dentistry as an temporary fix if a full root canal isn't feasible at the time. Got the tiniest whiff of that on my tongue during rinsing and it tasted downright toxic.
While this may be true, their throughput is practically unbounded. this has been a primary challenge to using local models at work, where its normal for me to have 1-6 sessions churning away at once. So while 100t/s on a single session is very fast, theres no way you could get a cumulative ~500t/s from a consumer setup like this, do the bandwitdth bottleneck on a single card/cpu/ram/ssd (tends to be gpu ram bandwidth is the limiter in my setup, but they all have respective limiters to this kind of throughput)
I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot
I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.
> That speed is what we used to have in software engineering. So much careful craftsmanship, no line just yeeted out.
That reads like nostalgia. What you describe sounds like the code I brought forth on a sunny Sunday afternoon, but certainly not on a hectic work day under deadline stress. I had Stackoverflow open _all the time_, and started googling at the earliest inconvenience. There were the dangerous parts of the codebase, where I paid more attention, much like I do now. But the rest, well - not everything needed the polish, and corners were cut all the time.
Your tool calls (MCPs?) are very likely too wasteful. Apply some filtering logic on the offending tool’s output. Either a wrapper CLI, or just tell codex how to filter.
I have a Kemper. Yes, the base tone is what it is - you’ll usually download a profile like “Basket case” and it will sound roughly like the song if you have the right guitar. It’s heavily pre-styled and you can’t really make it sound that different - and why would you?
Nuno B said that he played EVH's guitar and rig, and he sounded like....nuno.
Thanks to Rick Beato for the interview, and making music interesting again.
reply