I use separate machines altogether. I only do work-related things on the machines that my employer provides, and only do personal things on my own machines.
I totally understand why they ended up not using the ‘CSS IS AWESOME’ style overflow, but I think it would have been hilarious and not everything needs to be so serious :)
I’m curious about how good the performance with local LLMs is on ‘outdated’ hardware like the author’s 2060. I have a desktop with a 2070 super that it could be fun to turn into an “AI server” if I had the time…
I've been playing with some LLMs like Llama 3 and Gemma on my 2080Ti. If it fits in GPU memory the inference speed is quite decent.
However I've found quality of smaller models to be quite lacking. The Llama 3.2 3B for example is much worse than Gemma2 9B, which is the one I found performs best while fitting comfortably.
Actual sentences are fine, but it doesn't follow prompts as well and it doesn't "understand" the context very well.
Quantization brings down memory cost, but there seems to be a sharp decline below 5 bits for those I tried. So a larger but heavily quantized model usually performs worse, at least with the models I've tried so far.
So with only 6GB of GPU memory I think you either have to accept the hit on inference speed by only partially offloading, or accept fairly low model quality.
Doesn't mean the smaller models can't be useful, but don't expect ChatGPT 4o at home.
That said if you got a beefy CPU then it can be reasonable to have it do a few of the layers.
Personally I found Gemma2 9B quantized to 6 bit IIRC to be quite useful. YMMV.
If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button.
Testing performance this way, I got about 0.5-1.5 tokens per second with an 8GB 4bit quantized model on an old DL360 rack-mount server with 192GB RAM and 2 E5-2670 CPUs. I got about 20-50 tokens per second on my laptop with a mobile RTX 4080.
I am using an old laptop with a GTX 1060 6 GB VRAM to run a home server with Ubuntu and Ollama. Because of quantization Ollama can run 7B/8B models on an 8 year old laptop GPU with 6 GB VRAM.
Last time I tried a local llm was about a year ago with a 2070S and 3950x and the performance was quite slow for anything beyond phi 3.5 and the small models quality feels worse than what some providers offer for cheap or free so it doesn't seem worth it with my current hardware.
Edit: I've loaded llama 3.1 8b instruct GGUF and I got 12.61 tok/sec and 80tok/sec for 3.2 3b.
I built my new site (https://varun.ch) using 11ty, because they seem to commit to never making breaking API changes. They have a YouTube video where they build a pre 1.0 project in 2.0. https://youtu.be/bPtQmsjXMuo
Definitely agree. But 11ty’s core dependencies are relatively small (compared to other NodeJS things), and I am personally contributing PRs for getting rid of some long ljharb-y dependency chains.
Chrome to Firefox is a relatively easy switch, especially for those that don’t depend on Google sync. The main sources of friction for me were the lack of a good profile switching UI (solved with a browser extension that mimics the Chrome menu), and weird security requirements for homemade extensions (IIRC if you want to have the extension persist after restarting Firefox, you need to sign the extension, which is a pain)
For users switching from Arc, there is no good alternative, but Firefox with Sidebery and custom CSS comes close.
I don't know if this is what you meant, but as an alternative to profile switching, there are Multi Account Containers [1].
It allows assigning a container to each tab, and the containers are isolated from each other. If you have an MS or Google account for both work and personal, you can open them at the same time in different tabs.
I'm using using multiple profiles when I want to have a different set of extensions, bookmarks and browsing history. Multi Account Containers help with none of that.
this is such a killer feature I don't understand why it even is an extension, every browser that isn't adversarial to the user should have that feature tbh
Note that Firefox profile management is getting an overhaul right now, including an easy profile switching UI. I'm not sure when it will be landing in release, but it is being actively built!
I've tried to switch from Vivaldi to Floorp and there is some things that Firefox does that drive me absolutely nuts.
The main one is the behaviour of pinned tabs. Pinning in Firefox turns it into an icon that is harder to hit and doesn't even protect it from closing. This makes them essentially useless, they should be moved to the front of the tab bar and be protected from closing.
The second is that when you use vertical tabs the tab bar acts like a title bar instead of a separate entity. This means you can't double click to create a new tab, and trying to drag a tab often results in the entire window moving. I have to use Tree style tabs and disable the normal tab bar completely to prevent this.
There are also things that I don't like such as how downloads are handled and I've has issues with my session tabs being saved properly.
> they should be moved to the front of the tab bar and be protected from closing.
Firefox pinned tabs are moved to the LHS of the tab bar, they have no close button and ctrl-W doesn't close them. How much more do you want them to be protected from closing?
This is actually one thing where Firefox is clearly better than Chrome ... Chrome pinned tabs close with ctrl-W which is really easy to do accidentally.
Middle click still closes tabs which is my most frequent way to close tabs.
As I mentioned Vivaldi is my Chromium based browser of choice, and it is far better in the handling of pinned tabs. They go up to a separate section at the start of the tab list without reducing in size, and then they absolutely cannot be closed without unpinning the tab first
> Also in chrome, multiple profiles need multiple google account(If I understand the UI correctly)connected, but in Firefox no account is needed.
You can use Chrome with multiple profiles by disabling the "Allow Chrome sign-in" option so that none of your browser profiles are tied to a Google account. I don't know if that option can be toggled on a per-profile basis, because I happen to prefer it off for all of my browser profiles.
I don't know much about Arc. But Arc users could give Firefox "Nightly" a try to preview new features coming up. It has vertical tabs and you can "pin" a few tabs at the top. Nightly also has containers already built-in, so you can have multiple accounts open for the same site in different container tabs.
In the past I've had a lot of issues with Google properties not working properly on Firefox -- either outright broken or using crazy amounts of CPU on Firefox but not Chromium-based browsers. Does anyone know if this is still an issue? I'd love to try again before I'm forced to by uBO breaking.
