Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist:
- Easy plug & play model installation, and trivial to change which model once installed.
- Runs a local web server, so I can interact with it via any browser
- Ability to feed a model a document or multiple documents and be able to ask questions about them (or build a database of some kind?).
- Absolute privacy guarantees. Nothing goes off-machine from my prompt/responses (USP over existing cloud/online ones). Routine license/update checks are fine though.
I'm not trying to throw shade at the existing ways to running LLMs locally, just saying there may be room for an OPTIONAL commercial piece of software in this space. Most of them are designed for academics to do academic things. I am talking about a turn-key piece of software for everyone else that can give you an "almost" ChatGPT or "almost" CoPilot-like experience for a one time fee that you can feed sensitive private information to.
I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission.
The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property.
What you are describing would help alleviate the concern about issue 2, but I’m not sure if it would help alleviate the concerns with issue 1.
It's basically the same thing in our company, too. They basically put a similar rule in place that prevents anyone from using e.g. Chat GPT. Little do they know that all software devs within the company are using co-pilot and the company is even paying for it. It's quite a funny situation tbh..
> Little do they know that all software devs within the company are using co-pilot and the company is even paying for it.
Just like annual sexual harassment training - it's mostly corporate CYA on liability. If it ever goes to court, they'll plead ignorance and blame the employees who should have known better as they were trained/informed on what they ought not to do.
Paying for co-pilot could bite them though, so I suspect it's a case were the one part of the organization isn't aware of what the other is doing
All of your assumptions are exactly right. They (mostly managers with little to no IT background) want to cover their own asses in case shit hits the fan (unlikely scenario if you ask me, because the company is just overrating the value of their data. Nobody gives a fuck about us anyway...) and many parts of this company have developed their own habits... The company is just very big and I can understand why they might be afraid, but come on, nobody will take that policy seriously forever. You need to eventually put some reasonable rules in place that allow you to make use of such Innovations...
Except, at my company they block software like that. Not only do they block it, but if you try to go to it a security person will immediately call your manager and ask what you are doing.
we try to eliminate this problem by using code models trained only on permissevely licensed code, then you can run them locally without sending code anywhere
Change company. Honestly. If you go as far as to forbid your partners in crime (workers sigh..) to explore new uncharted territory at all - well ya know someone will/might just win by not doing that.
This is particular, specifically problematic territory. I cannot imagine handing over proprietary data to a third party without a contract in place for how that data is stored and used. It’s not about innovation, it’s about using someone else’s tools without ownership. For the other case, it’s both about integrity in owning your own work, and a shield from legal consequences. These things should be very relevant to any business.
I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. I am more bullish on LLMs than most of my coworkers. I think there is a lot of potential there. I do not see that potential in the current commercial offerings, and the financial outlay to fine-tune an open-source model and run it at scale is…prohibitive.
> I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy.
Really? I'm academic now but I find Copilot at least moderately helpful when I'm writing a library. It's pretty good a lot of boilerplate functions, docstrings, regex, etc. I certainly don't want to go back to not using it, my code is a lot closer to production quality now and looks nicer.
Thinking back to my days in back-end it seems like it would have been very helpful/sped things up so I'm surprised to hear it's just a toy but I've been out of the professional game for a while now. What's the main criticism?
Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now.
That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.
I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup.
I've not yet been able to solve the challenge of needing CUDA etc for some models though!
Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.
Oobabooga is a great tool but it still has a long way to go in term of user-friendliness. It's absolutely not plug and play the way that chatgpt is; It requires research, trial and error, and knowledge of the tech to make the model work to its full potential. It's great once you finish setting it up, but it does not compare to what you would expect from a commercial product aimed at normal end-users.
Things like bad default values, no tooltips, an no curated model list to one-click download is what separates a tool like Oobabooga from a paid commercial product. These things require time/money and it would be very unlikely that an open source tool could find resources for all the testing and R&D.
I think there is a big market for products where you pay and can just start chatting with the model without having to ever go to the settings tab or google anything unless you need to do something out of the ordinary.
Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?
Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.
we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE
- Easy plug & play model installation, and trivial to change which model once installed.
- Runs a local web server, so I can interact with it via any browser
- Ability to feed a model a document or multiple documents and be able to ask questions about them (or build a database of some kind?).
- Absolute privacy guarantees. Nothing goes off-machine from my prompt/responses (USP over existing cloud/online ones). Routine license/update checks are fine though.
I'm not trying to throw shade at the existing ways to running LLMs locally, just saying there may be room for an OPTIONAL commercial piece of software in this space. Most of them are designed for academics to do academic things. I am talking about a turn-key piece of software for everyone else that can give you an "almost" ChatGPT or "almost" CoPilot-like experience for a one time fee that you can feed sensitive private information to.