How does this compare to Agent Zero (frdel/agent-zero on GitHub)? Seems that provides similar functionality and uses docker for running the scripts / code generated.
Ollama provides an API endpoint that now supports the ability for an LLM to use tools/functions. Ollama is not a framework itself.
Agent Zero already can use Ollama and alternatives to run the LLMs, and this new feature should enable it to more accurately call tools that is getting built into the models that support it.
I’d love to read a blog post from successful one person startups that are in the highly competitive spaces like DevOps tooling (uptime, analytics…) on how they go-to-market and differentiate.
Is there a tutorial on how to get that setup running step-by-step? I only found a GitHub issue (https://github.com/ggerganov/llama.cpp/issues/4439) that mentions that mainline llama.cpp isn't working for the model.
Bonus question if you have the time: there's a release by TheBloke for this on HuggingFace (TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF); but I thought his models were "quantised" usually - does that kneecap any of the performance?
If you're new to this then just download an app like LMStudio (which unfortunately is closed source, but it is free) which basically just uses llama.cpp under the hood. It's simple enough to get started with local LLMs. If you want something open source ollama is probably a good place to look too, it's just a CLI tool but several GUIs integrate with ollama specifically.
As for your bonus question, that is the model you want. In general I'd choose the largest quantized version that you can fit based on your system. I'm personally running the 8bit version on my M3 Max MacBook Pro and it runs great! Performance is unfortunately a loaded word when it comes to LLMs because it can mean tokens per second or it can mean perplexity (i.e. how well the LLM responds). In terms of tokens per second, quantized models usually run a little faster because memory bandwidth is a constraint, so you're moving less memory around. In terms of perplexity there are different quantization strategies that work better and worse. I really don't think there's much of a reason for anyone to use a full 16fp model for inference, you're not really gaining much there. I think most people use the 4bit quants because it's a nice balance. But really it's just a matter of playing with the models and seeing how well it works. For example, some models perform okay when quantized down to 2 bits (I'm shocked that's the case, but I've heard people say that's the case in their testing), but Mixtral is not one of those models.
I'm working on a blog post documenting what I've been doing as a newcomer to llama.cpp and the mixtral model. The steps can apply to any model really. Its mostly about optimization steps I'm experimenting with. Be warned its all new to me and my explanations may not be entirely accurate yet, as I'm still learning the lingo so to speak.
The blog is at https://geuis.com. I'll try to wrap it up today or tomorrow and get the post out.
Ollama is great, and they just added (are still adding) OpenAPI API compatible endpoints, thus opening up access to many other toolchain possibilities than previously available to it. It also has some support for some multi-modal (vision and text) models. Easy to use, easy to install, does the job it's designed to do (rather well, even)... Highly recommended!
There's walkthroughs on reddit.com/r/localllama. You can download one click installers for oobabooga, then it's just a matter of getting the model you want and making sure the config is correct.
I’m an immigrant. I’ll be the first in line to agree that all the demands (and the specifics of those demands: personal space, hygiene, clothing, noise etc) for assimilation by western societies that I’ve ever heard re: immigrants, are reasonable - assuming such luxuries can be afforded (nobody crams themselves 8 to a room by choice).
But, people aren’t robots whose movements are controlled by an on-off switch. The government introduced the means for people to arrive and work, and so the people arrived. They are continuing to arrive because the policies have not been updated yet. How is it the immigrants’ fault? Why the hate and the attacks on their dignity & humanity?
The nonstop online vitriol hurts me deeply to read - nowhere is “safe” - Reddit, HN, Instagram… the hate spewers seemingly spend all their time spewing on these platforms to manipulate opinions and tap into the fundamental atavistic psychological flaws of the human mind.
If you are a 19 year old Canadian who is priced out of everything and is in a line at the local movie theater 100 long with desperate foreign students trying to get your first job you're going to feel cheated and pissed off.
If you are a 37 yo millennial who still doesn't own a house, can't afford to have kids moving back into the shitty style of apartment you lived in your early 20's but now it costs $2700 a month, you are going to be fucking pissed off.
If you are a new parent waiting 9 hours overnight in an packed emergency room, you are going to feel pissed off.
It is hard to tell Canadians this is good for them when everything has gotten worse.
I think this is because immigration is mostly pushed by people with heavy ideological reasons (far left, no border militants, etc.) but also rich employers who just want cheap and malleable workers.
Now you have the common people to whom you never asked their opinion about mass immigration: they now understand that instead of increasing wages, giving better benefits and improve working conditions to attract potential workers where it is lacking (supply and demand), the rich employers cheated the game with the help of elected politicians, by bringing hundreds of thousands of people who are willing to do it for less.
Add to this difficulties to buy a home, wages stagnation, rent going up, general inflation... The rich get richer, the rest gets poorer. This is how hatred is born unfortunately.
i recently saw a refreshing perspective on canadian immigration that addressed these issues at their origins: failures in housing policy and complacent governance. it's the perspective i wish more canadians held so we could steer the government towards policy that's better for all. maybe nativist nimbyism isn't inevitable.
That video gets cited regarding every discussion about the current Canadian immigration disaster. It pushes two basic ideas-
1) If Canada had housing, services and infrastructure that met any growth in population, there would be no problem!
2) Look at all these problems occurring, isn't it wonderful‽‽‽ It's going to force a revelation and everything will be super fixed and it'll be better than ever! The future is now! Things that don't kill you make you stronger or something.
That covers 100% of the video, and the delusional, pie in the sky notion pushed by the creator. If that video were presented as farce it would actually be funny. Instead it's depressing and sounds like the dope-fueled meanderings of someone absolutely decoupled from any semblance of reality.
Nothing is being fixed. Housing starts have dropped. Housing projects are moving slower than they did before this influx (but wait, isn't the video holding that there has been some amazing, wonderful transformation? Where?). The health system is in shambles. Demographics ghettos grow. A whole generation are going to be living in their parent's basements indefinitely, and anyone looking for a future is moving out of Canada.
Canada is poised for a very unfortunate hard turn to the right due to the policies of the very minister quoted and his peers who pushed the same "no problem here!" dismissals to growing concern. Busy making excuses as problem because disaster became catastrophe.
It is very unfortunate that some of the anger and outrage about government policy is reflected in racist or xenophobic outbursts as described in the root post, but it was basically inevitable: This government basically declared war on a demographic of Canadians and they're "fighting back". I'm lucky to work in a high skill position and live in an affluent area, but I can recognize the absolute disaster that is occurring: When coffee shop chains are buying up apartments and evicting the residents to bring in an army of TFWs, something has gone profoundly awry. When international students don't actually go to classes, get a worthless diploma, and are buying a work permit and a path to residency, something has gone wrong.
Canada needs immigration. Canada was built on immigration. Canada is a multicultural nation. What has been happening is a perversion of that and has put a country on edge.
Would be cool if there was a single site with an easy-to-use but comprehensive index of all such materials found in and around an average home that would make a chemist anxious, along with the safe alternative.
If an individual does this, then they should be prepared to be sued. For example, say that Teflon in toxic and you will surely be sued.
There are federal agencies that are supposed to monitor thse things, but in some countries these institutions are broken. IMHO Europe does the best job of regulating toxic substances. Even in other jurisdictions, there are manufacturers guides that are often not read or adhered to.
I think this info is mostly out there and is able to be found.
Yes, I agree that a comprehensive global index would be a great thing - so start with checking there (I forget where their index is!)
And don't forget, alcohol, tobacco, driving and sugar are by far the biggest killers in our modern societies!
Besides, chemists themselves have varying ideas of personal safety, depending on what they work with and how flammable/explosive/toxic/all of the above it is. The only real solution, unfortunately, is for the regulatory authorities to make and publish clear evidence based decisions.
There are tools like EPA Safer Choice and various apps built around helping people find safer choices when shopping, for what it's worth. https://en.wikipedia.org/wiki/EPA_Safer_Choice
Young people also aren't rich, and live amongst the awful rent seeking economy built by those who came before. So it's not hypocritical to want to buy cheap, it's an unfortunate necessity. Also, the whole point of the push to make sustainability a mainstream concern is so that 'all those awful young people' can afford to buy sustainably produced products. Right now, unsustainable enterprises get to externalize the environmental costs on everyone else, and thus make sustainable & quality goods extremely pricey. It's a tax on morality - so let's eliminate that first and then point fingers about hypocrisy.
