Hacker News new | past | comments | ask | show | jobs | submit login
Fine-tune your own Llama 2 to replace GPT-3.5/4
954 points by kcorbitt 9 days ago | hide | past | favorite | 181 comments
There has been a lot of interest on HN in fine-tuning open-source LLMs recently (eg. Anyscale's post at https://news.ycombinator.com/item?id=37090632). I've been playing around with fine-tuning models for a couple of years, and wanted to share some insights and practical code. I’ve condensed what I’ve learned into a small set of notebooks at https://github.com/OpenPipe/OpenPipe/tree/main/examples/clas..., covering labeling data, fine-tuning, running efficient inference, and evaluating costs/performance. The 7B model we train here matches GPT-4’s labels 95% of the time on the test set, and for the 5% of cases where they disagree it’s often because the correct answer is genuinely ambiguous.

What is fine-tuning? You can think of it as a more-powerful form of prompting, where instead of writing your instructions in text you actually encode them in the weights of the model itself. You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn. Fine-tuning can work with as few as 50 examples but I usually try to get 1000+ if possible.

Prompting still has some big advantages over fine-tuning. It's way easier/faster to iterate on your instructions than label data and re-train a model. And operationally it's easier to deploy one big model and just adjust its behavior as necessary vs deploying many small fine-tuned models that will likely each get lower utilization.

Fine-tuning has one huge advantage though: it is far more effective at guiding a model's behavior than prompting, so you can often get away with a much smaller model. That gets you faster responses and lower inference costs. A fine-tuned Llama 7B model is 50x cheaper than GPT-3.5 on a per-token basis, and for many use cases can produce results that are as good or better!

For example, classifying the 2M recipes at https://huggingface.co/datasets/corbt/all-recipes with GPT-4 would cost $23k. Even with GPT-3.5 it would cost over $1k. The model we fine-tuned performs similarly to GPT-4 and costs just $19 to run over the entire dataset.

Disclaimer: My brother David and I are working on an open-source product called OpenPipe (https://github.com/openpipe/openpipe) to help engineers adopt fine-tuning as simply as possible. But none of the information above depends on our startup. The current post is just about sharing information that we’ve learned about fine-tuning. I hope it’s useful!

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5;

For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.

Llama 7B wasn't up to the task fyi, producing very poor translations.

I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on other vendors (even open source models).

I'm curious to see if others have gotten different results?

Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many classes of task.

Also, it's worth noting that Replicate started out with a focus on image generation, and their current inference stack for LLMs is extremely inefficient. A significant fraction of the 100x cost difference you mentioned can be made up by using an optimized inference server like vLLM. Replicate knows about this and is working hard on improving their stack, it's just really early for all of us. :)

Founder of Replicate here. It's early indeed.

OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B.

Running a fine-tuned GPT-3.5 is surprisingly expensive. That's where using Llama makes a ton of sense. Once we’ve optimized inference, it’ll be much cheaper to run a fine-tuned Llama.

We're working on LLM Engine (https://llm-engine.scale.com) at Scale, which is our open source, self-hostable framework for open source LLM inference and fine-tuning. We have similar findings to Replicate: Llama 2 70B can be comparable to GPT 3.5 price, etc. Would be great to discuss this further!

How heavy of a lift is it to optimize inference?

> Llama 7B wasn't up to the task fyi, producing very poor translations.

From what I've read and personally experimented with, none of the Llama 2 models are well-suited to translation in particular (they were mainly trained on English data). Still, there are a number of tasks that they're really good at if fine-tuned correctly, such as classification and data extraction.

> I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on other vendors (even open source models).

I think you're definitely right about that, and in most cases just using GPT 3.5 for one-off tasks makes the most sense. I think when you get into production workflows that scale, that's when using a small fine-tuned models starts making more sense. You can drop the system prompt and get data in the format you'd expect it in, and train on GPT-4's output to sometimes get better accuracy than 3.5 would give you right off the bat. And keep in mind, while you can do the same thing with a fine-tuned 3.5 model, it's going to cost 8x the base 3.5 price per token.

Is that because translation is typically an encoder-decoder task and llama is decoder only or is there something else about it that makes the last difficult for llama?

If you don't make it learn other-language texts, it won't be able to speak that language.

As I learned that 85% of its trainig data is English. Othere languanges composed of 15%.

Cost isn't the only incentive not to use an LLM service that resides in a foreign country. Around here, there are industries for which it's pretty much a no-brainer to avoid anything that sends data across the atlantic.

Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world.

(Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say that provisioning LLM capacity and GPU's is something we basically all have a tremendous amount of passion about.)

Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies?

Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last question, of course, but please think about it. It doesn't matter where the LLM is located but who controls it and who holds the resulting data.

I don't think this is a promise Microsoft can make. The US Cloud Act states that Microsoft falls under US jurisdiction and it's legally bound to share foreign data if asked by US law enforcement.

"The CLOUD Act asserts that U.S. data and communication companies must provide stored data for a customer or subscriber on any server they own and operate when requested by warrant, but provides mechanisms for the companies or the courts to reject or challenge these if they believe the request violates the privacy rights of the foreign country the data is stored in."


Worldwide big corps already utilized Microsoft 365 especially SharePoint. That's Microsoft's advantage.

I do think large tech companies do pretty well with customer data. As a former Googler I would be comfortable with my Gmail data residing in a foreign datacenter.

They do pretty well, except the Room_641A in the building which is allowed to do anything they what with production branch without it being visible to ordinary workers.


Azure GPT 4 is already available in: Australia East, Canada East, East US, East US 2, France Central, Japan East, Sweden Central, Switzerland North, UK South (https://learn.microsoft.com/en-us/azure/ai-services/openai/c...)

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for < $2K.

I built two such a systems after burning that much in a week on ChatGPT.

> I built two such a systems after burning that much in a week on ChatGPT.

What are you doing!?

Have a client with many thousands of csv, json, xml files detailing insurance prices. Fundimentally they all contained the same data but wildly different formats because they were produced by different companies and teams. I used ChatGPT to deduce their format so I could normalize them. Easily underbid their current contractor who was using humans for the work and now I have an easy quarterly billing. :)

TBC, I probably could have optimized tokens but contract was profitable and time critical.

Thanks for sharing!

Would you mind to share all your PC HW (mobo, casing, cooling, etc) for this dual GPU configuration? Thanks.

The one you could build for under 2K is last gen hardware.

* Chenbro Rackmount 4U Server Chassis RM42300-F (rack mount case Remove the air filter on 120mm fan. Put two decent 80mm exhaust at rear). * Two used air cooled 3090s. About $650 a piece on ebay. Check slot width and make sure everything will fit on your motherboard. Do a burn in when you get them cause used GPUs can be hit or miss. * 5950x CPU (overkill just had it) * 128GB DDR4 * Motherboard with x570 chipset and dual pcie x16. These will birificate to x8 pcie 4.0 lanes to each GPU. This is enough bandwidth to push GPUs to max IME * 1200W+ ATX power supply. * ebay "u.2 pcie 3.84TB" and adaptor for m.2 NVME slot. (again what I had & it is cheap)

If you're going to really beat the thing I would power limit the 3090s to 320w (from 350w). Perf change is not really notable and keeps temps better.

From people hosting image generation models on Stable Horde I've heard that you can pretty severely underclock/undervolt your GPUs and keep them stable, massively reducing heat output and energy cost without losing nearly as much performance. I'm not sure if this transfers into text generation or not, this was from image generation workers that have a few seconds downtime between requests; however it might be worth a bit of research if you happen to be running consumer GPUs.

----- From TheUnamusedFox, in August: > 3090 down to ~260-270 watts (from 400) with minimal gen speed impact. Same with a 3080ti. It seems to be more stable with image generation than gaming, at least on my two cards. If I try to game or benchmark with this undervolt it is an instant crash.

From another user:

> this undervolting stuff is pretty sweet. > undervolted_limits.png [1] > max_power_limits.png [2] > this is my before and after. > a solid 200 watt drop for only 9.2% loss of performance > not to mention the 30 degree drop in temps

[1]: https://cdn.discordapp.com/attachments/1143237412663869570/1... [2]: https://cdn.discordapp.com/attachments/1143237412663869570/1...

Thank you so much.

Are there any good resources related to expanding context windows, or even just the mechanics of how they actually work as properties of a model?

Lots. LLAMA 2 was trained on 4K context windows but can run on arbitrary length just the results become garbage as you go longer.

I refer you to https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-c... for an "easy" to digest summary

Edit: Nevermind, saw you posted elsewhere. Thank you!

Can you share your system specs? I was looking into something similar but my costs were closer to 6 to 8k for the whole system.

is the $2K you mentioned the total cost of ownership?

>For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.

You'll never get actual economics out of switching to open models without running your own hardware. That's the whole point. There's orders of magnitude difference in price, where a single V100/3090 instance can run llama2-70b inference for ~0.50$/hr.

No, they can't run it. llama 70 with 4 bit quantization takes ~50 GB VRAM for decent enough context size. You need A100, or 2-3 V100 or 4 3090 which all costs roughly roughly $3-5/h

Wrong. I am running 8bit GGML with 24GB VRAM on a single 4090 with 2048 context right now

Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?

Offloading 40% of layers to CPU, about 50t/s with 16 threads.

That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?

>Is there a secret to what you're doing?

Core speed and memory bandwidth matter a lot. This is on a Ryzen 7950 with DDR5.

Care to share your detailed stack and command to reach 50t/s? I also have a 7950 with DDR 5 and I don't even get 50 t/s on my two RTX 4090s....

TBH, Replicate is not a great way to run 7B beyond experimentation. You want a host with cheap consumer GPUs (like vast.ai) since the 4-bit requirements are so modest.

You either need a backend with good batching support (vLLM), or if you don't need much throughput, an extremely low end GPU or no GPU at all for exLlama/llama.cpp.

OpenAI benefits from quantization/batching, optimized kernels and very high utilization on their end, so the huge price gap vs a default HF Transformers instance is understandable. But even then, you are probably right about their aggressive pricing.

As for quality, you need a llama model finetunes on the target language (many already exist on Huggingface) and possibly custom grammar if your backend supports it.

It shouldn't be 100x. We've built an LLM API at Anyscale, and the price comparison works out as follows (per million tokens)

- Llama-2-70B: $1 (on Anyscale Endpoints [1]) - GPT-3.5-turbo: $1.50 - $2 (OpenAI [2])

[1] https://app.endpoints.anyscale.com/ [2] https://openai.com/pricing

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages).

You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on specific tasks (some form of extraction, summarization, generative tasks, etc.), or for general chatbot-like uses.

>You're better off using models specialized in translation

For a couple dozen languages, GPT-4 is by far the best translator you can get your hand on so basically no.

I will say that GPT-4 is just incredibly expensive. For my app I only use it for advanced translations/corrections, and usually a combination of GPT-3.5+Wiktionary is able to get the more simple stuff done

> GPT-3.5+Wiktionary

Can you share more about your app and what you're doing?

Sure! I'm building a personalized AI language learning tutor using Open AI's API and ElevenLabs (for Text to Speech).

Right now it's basically a chat bot that you can use to practice conversing with. It provides corrections for the things you type. Eventually I'd like to try adding Whisper as well to allow users to speak out loud.

When you hover over a word, you get a translation. Initially I thought using Open AI for every word translation would be too much, but I've been able to get it down to ~36-40 tokens/request. (3-4 cents/1000 requests). I also began parsing and uploading some of this [Wiktionary data](https://kaikki.org/dictionary/rawdata.html) and am working on a feature that integrates the GPT-3.5 translation with this Wiktionary data.

A lot of these features are still in the works but you can feel free to try it if you like (https://trytutor.app).

What would be the best local standalone solution for translation model? Personal use, mostly self-education. 2 popular languages both ways (like en-spa, fr-ger). Free, pretrained off the github would be the best. I can try and train say 100M params LLM on 4090 RTX. But I'm not sure satisfactory result are achievable.

There are plenty of examples in the literature of using LLMs for translation beating the metrics of non-LLM models, even for languages for which there isn't a lot of data. Transliterating non-Latin characters helps a lot with accuracy as well.

what models would you use for translation? I am working on a language learning tutor (trytutor.app, very early) and GPT-3.5 turbo has been working fine, for the most part.

For foreign language corrections ("correct this German sentence and give a reason for the correction"), GPT-3.5 doesn't quite have the horsepower so I use GPT-4

I’m actually replicate user. I have experimented with LLAMA2 on the replicate and I have similar experience

But you are totally correct about the pricing part it can get expensive

I’m running this photo service https://msdosimagetools.ngrok.dev/

Its doing 200+ photos every day and I’m using open source models behind the scene on replicate. My costs increasing day by day

Plus this is hosted locally

Llama and GPT are auto-regressive decoder only architectures which for pure translation jobs are not the optimal architectures. Training seq2seq models or encoder/decoder models on datasets of sentence pairs designed for translation will likely allow you to use much smaller models. You will not be wasting parameters on general “language understanding” capability that Llama and GPT have if pure translation is all you need. T5 or Flan-T5 might be good starting points.

Google Maps was also cheap. Initially. So it is aggressively cheap now, but would aggressively change later.

Google maps has always been free.

Their API, however, is not. (After a certain usage threshold)


Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years...

It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware.

I guess they are also getting a pretty good cache hit rate - there are only so many questions people ask at scale. But still, it's dumping.

Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as possible to optimize for inference. Also, Nvidia chips are way more efficient at inference than M1 Max. OpenAI also has the advantage of batching API calls which leads to better hardware utilization. I don't have definitive proof that they're not dumping, but economies of scale and optimization seem like better explanations to me.

What makes you think 3.5 is significantly smaller than 70B?

I also do not have proof of anything here, but can't it be both?

They have lots of money now and the market lead. They want to keep the lead and some extra electricity and hardware costs are surely worth it for them, if it keeps the competition from getting traction.

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

gpt3.5 turbo is a new model, not Curie. As others have stated, it probably uses Mixture of Experts which lowers inference cost.

Is there a source on that? I've never seen anyone think it's below even 70B

It still does a much better job at translation than llama 2 70b even, at 6.7b params

If it's MOE that may explain why it's faster and better...


I thought it was fairly well established that GPT 3.5 has something like 130B parameters and that GPT 4 is on the order of 600-1,000

I remember:

- gpt-3.5 175b params

- gpt-4 1800b params

You think they are caching? Even though one of the parameters is temperature? Can of worms, and should be reflected in the pricing if true, don't get me started if they are charging per token for cached responses.

I just don't see it.

You can keep around the KV cache from previous generations which lowers the cost of prompts significantly.

turbo is likely nowhere near 70b.

Together AI has new aggressive pricing where 70b are on par with gpt35 and everything smaller is fairly cheaper. The catch is the only 32k context length model as of today is their llama 7b which is fairly limited.

I thought Llama was opensource/free and you could run it yourself?

You (currently) need a GPU to run any of the useful models. I haven't really seen a business use-case that runs it on the user's computer, but given the hardware requirements it wouldn't be very feasible to expect.

So you'll have to figure out how to run/scale the model inference. Cloud GPU instances are generally very expensive, and once you start needing to horizontally scale it'll get messy fast.

At least at the moment it's expensive, especially if it's either very light usage or very intensive usage - you either need just a few seconds of compute occasionally, or lots of compute all the time requiring scaling.

The "lucky" ones in this scenario are small-medium businesses that can use one or a few cards on-site for their traffic. Even then when you take the cost of an A100 + maintaining it, etc. OpenAI's offering still looks attractive.

I know there's a few services that try to provide an api similar to what openai has, and some software to self orchestrate it, I'm curious how those compare...

> once you start needing to horizontally scale it'll get messy fast.

It gets expensive fast, but not messy, these things scale horizontally really well. All the state is encapsulated in the request, no replication, synchronisation, user data to worry about. I'd rather have the job of horizontally scaling llama2 than a relational database.

For sure, and yeah it wouldn't be terrible you're right. You'd just need the api servers + a load balancer.

My thing is that dynamically doing that is still a lot compared to just calling a single endpoint and all of that is handled for you.

But for sure this is a very decent horizontal use-case.

You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical.

So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

How do you fit Llama2-70b into V100? V100 is 16GB. Llama2-70b 4bit would require up to 40GB. Also, what do you use for inference to get 300+tokens/s?

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

I stumbled upon OpenRouter[0] a few days ago. Easiest I’ve seen by far (if you want SaaS, not hosting it yourself).

[0] https://openrouter.ai

>What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

That's an exercise left to the reader for now, and is where your value/moat lies.

> That's an exercise left to the reader for now, and is where your value/moat lies.

Hopefully more on-demand services enter the space. Currently where I am we don't have the resources for any type of self orchestration and our use case is so low/sporadic that we can't simply have a dedicated instance.

Last I saw the current services were rather expensive but I should recheck.

I bought an old server off ServerMonkey for like $700 with a stupid amount of RAM and CPUs and it runs Llama2-70b fine, if a little slowly. Good for experimenting

Unfortunately, Lama2 is not a fully open source license.

Compute costs money.

We provide per token based Llama 2 70B API at Deep Infra, $1/1M tokens, which is 25-50% cheaper than ChatGPT.

Can you provide a larger context length? Looking for a replacement of GPT-3.5 16k model. Might be interested for a higher-scale project.

Replicate has terrible pricing. Have you tried deepinfra?

For use cases well within the capabilities of an LLM from last year, fine-tuned LLaMa 2 13B should/will blow ChatGPT out of the water: think "rate the sentiment of this text from 0-10".

I believe this because LLaMa-2 13B is more than good enough to handle what I call "quick search", i.e.

``` User: "What's the weather in Milwaukee?"

System: Here's some docs, answer concisely in one sentence.

AI: It's 73 degrees Farenheit. ```

YMMV on cost still, depends on cloud vendor, and my intuition agrees with yours: GPT-3.5 is priced low enough that there isn't a case where it makes sense to use another model. It strikes me now that's there's a good reason for that intuition: OpenAI's $/GPU hour is likely <= any other vendor's and inference time of LLaMa 2 ~= GPT.

I do think this will change with local LLMs. They've been way over-hyped for months, but after LLaMa 2, the challenges remaining are more sociological than technical.

For months now it's been one-off $LATEST_BUZZY_MODEL.c stunts that run on desktop.

The vast majority of the _actual_ usage and progress is coming from porn-y stuff, and the investment occurs in one-off stunts.

That split of effort, and lack of engineering rigor, is stunting progress overall.

Microsoft has LLaMa-2 ONNX available on GitHub[1]. There's budding but very small projects in different languages to wrap ONNX. Once there's a genuine cross-platform[2] ONNX wrapper that makes running LLaMa-2 easy, there will be a step change. It'll be "free"[3] to run your fine-tuned model that does as well as GPT-4.

It's not clear to me exactly when this will occur. It's "difficult" now, but only because the _actual usage_ in the local LLM community doesn't have a reason to invest in ONNX, and it's extremely intimidating to figure out how exactly to get LLaMa-2 running in ONNX. Microsoft kinda threw it up on GitHub and moved on, the sample code even still needs a PyTorch model. I see at least one very small company on HuggingFace that _may_ have figured out full ONNX.

Funnily enough, ONNX is getting a spike in mindshare over the last month in the _Stable Diffusion_ community. There's decent cross-pollination between local art and local LLMs, ex. LoRA's were first a thing for Stable Diffusion. So I'm hoping we see this sooner rather than later.

[1] https://github.com/microsoft/Llama-2-Onnx

[2] Definition of cross-platform matters a ton here, what I mean is "I can import $ONNX_WRAPPER_LIB on iOS / Android / Mac / Windows and call Llama2.reply(String prompt, ...)"

[3] Runs on somebody else's computer, where "somebody else" is the user, instead of a cloud vendor.

My deepest thanks, I owe you one. Overlooked this completely. & spent dozens of hours learning way too much to still fall short of understanding how to make it work in ONNX.

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls.

My other thoughts to extend this are that you could make it seamless. To start, it'll simply pipe the user's requests to OpenAI or their existing model. So it'd be a drop in replacement. Then, it'll every so often offer to the user - "hey we think at this point there's enough data that a fine tune might save you approx $x/month based on your current calls, click the button to start the fine tune and we'll email you once we have the results" - and then the user gets the email "here are the results, based on that we recommend switching, click here to switch to calling your fine-tuned model" - Helicone and the other monitoring platforms could also offer something similar. (Side note I'm working on an "ai infra handbook" aimed at technical people in software orgs looking to deploy unspecified "AI" features and trying to figure out what to do and what resources they'll need - it's a 20+ page google doc, if anyone can help me review what I have so far please let me know and I'll add you.)

If it's latency/error/speed competitive, and cheaper, and equivalently accurate, then for anyone doing production scale LLM API usage it'd make sense to use something like this - either the fine-tune is worse so you keep using the regular API, or the fine tune has parity plus cost and/or speed advantage, so you switch. (It wouldn't make sense for prototyping scale, because the additional complexity of the switch wouldn't be worth it unless it could save you 4/5 or more figures a year in API costs I'd think.)

> My other thoughts to extend this are that you could make it seamless. To start, it'll simply pipe the user's requests to OpenAI or their existing model. So it'd be a drop in replacement. Then, it'll every so often offer to the user - "hey we think at this point there's enough data that a fine tune might save you approx $x/month based on your current calls, click the button to start the fine tune and we'll email you once we have the results" - and then the user gets the email "here are the results, based on that we recommend switching, click here to switch to calling your fine-tuned model"

You just described our short-term roadmap. :) Currently an OpenPipe user has to explicitly kick off a fine-tuning job, but they're so cheap to run we're planning on letting users opt in to running them proactively once they have enough data so we can provide exactly that experience.

>GPT and other LLMs don't allow you to use their output to train competing models

ToS is unenforceable and irrelevant to anyone that's in this space

That seems mostly right, particularly for internal models, but I wonder about adding some ringers to prove that copying happened:


Also, it seems sort of like how cryptocurrency folks assumed their transactions were anonymous? It's an API, so they could log the calls. (Maybe not the contents.)

> Side note I'm working on an "ai infra handbook" aimed at technical people in software orgs looking to deploy unspecified "AI" features and trying to figure out what to do and what resources they'll need - it's a 20+ page google doc, if anyone can help me review what I have so far please let me know and I'll add you.

Interested in helping out.

I would be interested in reviewing your handbook too. I am technical, but have not deployed any AI related tooling so far. keen to know if this is targeted to AI noobs as well.

I'm interested in your handbook as well. Is it the site in your bio?

I would be like to help in reviewing your handbook.

> GPT and other LLMs don't allow you to use their output to train competing models

I didn't allow them to use my output to train theirs either, so fuck 'em.

> Fine-tuning has one huge advantage though: it is far more effective at guiding a model's behavior than prompting, so you can often get away with a much smaller model. That gets you faster responses and lower inference costs. A fine-tuned Llama 7B model is 50x cheaper than GPT-3.5 on a per-token basis, and for many use cases can produce results that are as good or better!

These comparisons are reductive to the point of being misleading. Even with all the optimizations in the ecosystem, it's not trivial to get a finetuned 7B param model running at an acceptable inference latency. Even if you use a GPU such as an A100 for maximum speed, then you have scalability issues since A100s are scarce. Also, the "50% cheaper" assumes 100% utilization of a GPU which will never happen in production use cases.

Quality-wise, a finetuned Llama 2 is not necessairly better than ChatGPT. Finetuning requires a high-quality dataset which is not easy to construct. And in my own experience with finetuning Llama 2, qualitivately it caused more frustration to get outputs on par with just using ChatGPT.

The value of the ChatGPT API is more dependable scaling and not having to pay for an infra.

We're finding that when running Llama-2-7B with vLLM (https://github.com/vllm-project/vllm) on an A40 GPU we're getting consistently lower time-to-first-token and lower average token generation time than GPT-3.5, even when processing multiple requests in parallel. A40s are pretty easy to get your hands on these days (much easer than A100s anyway).

The 50x cheaper (that's 2% of the cost, not 50% of the cost) number does assume 100% GPU utilization, which may or may not be realistic for your use case. If you're doing batch processing as part of a data pipeline, which is not an unusual use case, you can run your GPU at 100% utilization and turn it off when the batch finishes.

If you've got a highly variable workload then you're right, you'll have much lower utilization numbers. But if you work with an aggregator that can quickly hot swap LoRA fine-tunes (as a disclaimer, my company OpenPipe works in this space) you can get back a lot of that lost efficiency since we can increase/decrease GPU capacity only when our aggregate usage changes, which smooths things out.

Doesn't this depend a lot on your application though? Not every workload needs low latency and massive horizontal scalability.

Take their example of running the llm over the 2 million recipes and saving $23k over GPT 4. That could easily be 2 million documents in some back end system running in a batch. Many people would wait a few days or weeks for a job like that to finish if it offered significant savings.

That's more of a fair use case.

It though also demonstrates why the economics are complicated and there's no one-size-fits-all.

We are talking about 7B models ? Those can run on consumer GPUs with lower latency than A100s AFAIK (because gaming GPUs are clocked different).

Not to mention OpenAI has shit latency and terrible reliability - you should be using Azure models if you care about that - but pricing is also higher.

I would say fixed costs and development time is on openai side but I've seen people post great practical comparisons for latency and cost using hostes fine-tuned small models.

"Running" and "acceptable inference speed and quality" are two different constraints, particularly at scale/production.

I don't understand what you're trying to say ?

From what I've read 4090 should blow A100 away if you can fit within 22GB VRAM, which a 7B model should comfortably.

And the latency (along with variability and availability) on OpenAI API is terrible because of the load they are getting.

When you say it can run on consumer gpus, do you mean pretty much just the 4090/3090 or can it run on lesser cards?

I was able to run the 4bit quantized LLAMA2 7B on a 2070 Super, though latency was so-so.

I was surprised by how fast it runs on an M2 MBP + llama.cpp; Way way faster than ChatGPT, and that's not even using the Apple neural engine.

It runs fantastically well on M2 Mac + llama.cpp, such a variety of factors in the Apple hardware making it possible. The ARM fp16 vector intrinsics, the Macbook's AMX co-processor, the unified memory architecture, etc.

It's more than fast enough for my experiments and the laptop doesn't seem to break a sweat.

Quantized 7B's can comfortably run with 8GB vram

This looks awesome! Tangential question - do you find GPT function calling to work consistently and without error, or do you get errors when using it? By errors I mostly mean incorrect function signatures/types or missing values...but if you see other unpredictable behavior that would help too.

I see wrong responses about 1% of the time, but I love it, considering parsing raw text output without function calling had a much higher error rate.

I haven't had much trouble with GPT 3.5 or 4 function calls returning in an undesirable format recently. I did get a few bad syntax responses when OpenAI first rolled it out, but not for the past few months.

Llama 2 can also pick the function call format up, given sufficient training data that contains function call responses, though you'll then have to parse the returned object out of the text-based response.

Has anyone done such fine tuning on llama though? Afaik most projects like llama.cpp use grammars instead.

Yep! The linked notebook includes an example of exactly that (fine-tuning a 7b model to match the syntax of GPT-4 function call responses): https://github.com/OpenPipe/OpenPipe/blob/main/examples/clas...


Can you clarify the 50x cheaper number? Is this for self-hosting, or if you're hosting on OpenPipe?

The pricing on OpenPipe says it's 0.0012 to 0.0016 per 1K tokens for Llama 7b. GPT-3.5 pricing is 0.0015 to 0.002, so not that different.

I'm assuming the 50x cost reductions are primarily from self-hosting?

Yep, the 50x cost reduction is if you self-host a fine-tuned model using the setup demonstrated in in the linked notebooks.

I think the cost calculation here does not reflect the actual scenario where most people face. In real world scenario, we don't get inputs queued up to millions and wait for the GPU to inference them continuously at 100% utilization. We need to ensure the user get their response in time, and assume that we get all the inputs spread out evenly within a month, we have to look at the cost of running GPU for a month vs using OpenAI API.

Is Llama 2 currently the way to go for fine-tuning your own models? Are there other open-source LLMs worth considering?

Depends on your use case. If you're doing pure classification then there are smaller encoder-only models like DeBERTa that might get you better performance with a much smaller model size (so cheaper inference).

But if you need text generation and are ok with a 7B+ parameter model, Llama 2 or one of its derivatives is what I'd strongly recommend. The community around it is much larger than any of the alternatives so the tooling is better, and it's either state of the art or close to it on all evals when compared to other similarly-sized open models.

If you're comfortable sharing more details of the task you're trying to do I might be able to give more specific advice.

The Huggingface Leaderboard is mostly dominated by Llama 2 variants: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

It depends a lot on what you're trying to do. If have a focused use case of the type of fine-tuning you want, you can probably get away with one of the smaller models.

Another thing to look out for is Retrieval Augmented Generation (RAG). I don't see it in wide use yet, but it may turn out to more useful than fine tuning for a lot of situations.

We've found Flan-T5 to be useful for text-to-text (mostly document QA). Haven't done a lot of testing on fine-tuning yet though.

It's one of widely fine tuned model for now. Take a look at this colab for fine tuning on your dataset https://github.com/mlabonne/llm-course/blob/main/Fine_tune_L...

I am a little bit confused whether I need fine-tuning or RAG for my use case? My use case is this: I have some private data (say 1000 word documents), I want a QA capability on those 1000 documents. What is the best approach? Any help is appreciated.

Look at it like this:

- Fine tuning: Difficult, time-consuming, slow, takes time to add new information, costs a lot more.

- RAG: Can be free if you use free options like Chrome, Weaviate, or Postgres with Vector Plugin. Really fast. Once you set it up, you just need to upload a document, and it's available for GPT to answer with.

I'm using RAG for a client right now, and it was a breeze. Really easy, especially if you use something like Langchain. Compared to fine-tuning, it's a lot easier, cheaper, and faster...

Very nice, thanks!

Check out what Matt Shumer put together as well: https://github.com/mshumer/gpt-llm-trainer.

I have used his trainer for auto distillation of GPT-4 into GPT3.5 fine tunes, but plan to do the same for Llama as well.


"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn."

Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

There's no rule that your fine-tuning dataset needs to be split into input/output pairs -- you can of course fine-tune a model to just continue a sequence.

As a practical matter though, most of the fine-tuning frameworks, including Axolotl (which this guide uses) and HuggingFace's SFTTrainer (the actual fine-tuning trainer most frameworks use under the hood) assume your data comes in input/output pairs, and automatically inserts a separator token to let the model know that the input has finished and it should start generating the output. In general most tasks can be formulated this way, including autocomplete tasks, so I'd probably recommend going that way unless you have a very strong reason not to.

Axolotl takes a lot of formats, not all of them are in the form of input/output.

"Completion" format only takes a single text value per dataset record. Some other formats are in the form of multiple choice answers, etc.

Take a look below (there are more formats in "see other formats") https://github.com/OpenAccess-AI-Collective/axolotl#dataset

“most tasks can be formulated this way, including autocomplete tasks”

For autocomplete tasks, with a corpus of unlabeled documents, would you insert a separator token at an arbitrary space in each document, in order to form input/output pairs?

What you described is basically an input/output pair. The input is the sentence so far, and the output is the next token. You build your dataset by splitting the raw text corpus into sentences, paragraphs or documents, and for each of these chunks generate input/target pairs by taking the sentence up to the Nth token as input and that token as output. You do this for each token in your corpus' chunks.

For further reference you can lookup "next-token prediction objective".

What makes sense to fine-tune and what not?

You said 50-1000 examples.

Do I fine-tune when having specific q/a sets like from real customers and I want to add the right answer to the model?

Do I fine-tune facts or should I use some lookup?

Does adding some code and API docs for a current version of something I want more support make sense? Like chatgpt knows quarkus 2 but not quarkus 3

> What makes sense to fine-tune and what not?

In general, fine-tuning helps a model figure out how to do the exact task that is being done in the examples it's given. So fine-tuning it on 1000 examples of an API being used in the wild is likely to teach it to use that API really effectively, but fine-tuning it on just the API docs probably won't.

That said, there are a lot of interesting ideas floating around on how to most effectively teach a model purely from instructions like API docs. Powerful models like GPT-4 can figure it out from in-context learning (ie. if you paste in a page of API docs and ask GPT-4 to write something with the API it can usually do a decent job). I suspect the community will figure out techniques either through new training objectives or synthetic training data to do it for smaller fine-tuned models as well.

Generally speaking, fine-tuning a small model makes sense when the task that you want it to carry out is well-defined and doesn't vary too much from one prompt to another. Fine-tuning facts into a model doesn't seem to scale super well, but general textual style, output format, and evaluation criteria for example can all be instilled through the fine-tuning process. I would use lookup if you need your answers to include a wide array of information that the model you're basing off of wasn't initially trained on.

I have such use case: I have Java project I develop, I also used phind-codellama-36B-q8 with very satisfying results to aid the development.

Can I train it further using the project source to let the model "understand" the project context more?

I found this tutorial helpful for getting started with fine-tuning https://www.youtube.com/watch?v=74NSDMvYZ9Y

This guy used gradient.ai and he has a Google Collab to try it

If I paid $20 to fine-tune a model to do X, and you paid $20 to fine-tune a model to do Y, is there a way to merge models, aggregating X and Y training, without training from scratch again?

This looks very helpful! I'm just starting out in the ML/LLM space and have an opportunity to work on this at $dayjob, bookmarking as this looks like an excellent resource. Thank you!

Thank you for posting this. I had to go look for your HuggingFace data sets to find the labeled variety you produced with GPT-4, but other than that, everything was easy to follow.

To all those who are on this panel, which is the most comprehensive way a newbie can learn fine-tuning these models with or without the GPUs?

Are there any well directed courses available?

I wrote the notebooks in the post with the intention of them being a gentle introduction to fine-tuning. Would love any feedback on open questions you have as you go through them!

Thanks for sharing this! I think you're working on something amazing. I will include your links in my newsletter, I think it will help a lot of folks: https://www.theprompt.io/

This looks very interesting and looks like GPT3.5 is subsidized heavily. Given the advantage of scale economics for OpenAI its going to be difficult for a corporation to justify spending on their equipment and administration costs. This is where security of data and other non-functional requirements will justify training and running your own models.

What are hardware requirements for larger models? What can I fine-tune on Nvidia A100? Will it be possible to work with 70b for example?

Depending on what you're trying to accomplish, I'd highly recommend trying the 7B and 13B models first before jumping to the 70B. They're quite capable and I think lots of folks assume they need to jump to a 70B model when really a smaller one would work fine.

That said, you should be able to fine-tune a 70B model on an A100 using QLoRA. However, depending on the specifics of your dataset it might actually be cheaper to run on an 8xA100 machine since that way you don't have to swap any weights out to the machine's non-GPU memory, and you might get enough time savings from that that the more expensive machine pays for itself.

The plan was to do it in-house. And buying 8xA100 is a bit too much ;)

I'm in the exactly same boat. Targeting to fine tune llama 2 70b on 2xA100, with the hope of having one A100 run an 8bit quantized 70b model 24/7.

If you have an experiences to share, successes or failures, please do.

Fine-tuned low parameter LLM's are superficially good but the cracks are obvious if you test them on anything that isn't very strictly tied to the training data. IMO GPT-4 is really the first LLM that's broken out of the fake intelligence quality most LLM's seem to have, though only by a little.

If we assume this is true: https://iv.nboeck.de/watch?v=K5iDUZPx60E&t=2989

Then there isn't anything in particular which makes their model(s) stand out. On the contrary, they seem rather inefficient, which is probably reflected on the inference cost this gargantuan conglomerate takes to run.

Genuinely informative reply for those (few) of us on HN who don’t know the details of LLMs, thanks

Someone needs to make an LLM purpose-built for creating high-quality datasets for fine-tuning other LLMs.

This. The best use of the current llms is to create better Datasets.

Do you still use few-shot prompting with a fine-tune? Or does it make little difference?

Nope, no need for few-shot prompting in most cases once you've fine-tuned on your dataset, so you can save those tokens and get cheaper/faster responses!

Not only that, but in a lot of cases you won't have to fine-tune at all if an existing instruct model does a good enough job with unambiguous enough instructions.

In my experience, there is little need to do that. With completely unambiguous instructions that describe the exact output format, you can often get away with no examples whatsoever. Single examples might be helpful, but multi-shot prompting will be definitely unneeded (and may even harm the model's output quality).

Q: How did you arrive at the $23k figure for classifying 2M examples using GPT-4?

We ran 5K randomly selected recipes through GPT-4 and extrapolated based on the average cost per query.

Makes sense. Thank you!

Thanks! When it comes to choosing where to work with these models, which compute platform do you recommend (assuming locally doesn't really make sense with my resources)? Colab? AWS StudioLab?

Which is your go to?

A 7b model will work for very specific cases, but it will have a hard time drawing parallels between synonims, so you'll need to be extremely careful in building your fine tuning samples.

Do you think this would end up facilitating the diffusion of finetuned LLMs ckpt models, just like stable diffusion? What's missing is web-UI?

There are already many hundreds of finetunes on huggingface, and many excellent UIs to run them in, like KoboldCPP and Text-gen-ui: https://huggingface.co/models?sort=modified&search=13B

There is even a crowdsourced version of the UI like artbot: https://lite.koboldai.net/#

And there are some excellent extant finetuning frameworks, like Aoxotol, that run on consumer GPUs: https://github.com/OpenAccess-AI-Collective/axolotl

IIRC Text-gen-ui had a QLORA finetuning UI too.

What I am saying is that its already like Stable Diffusion, but the community is just somewhat under the radar, and finetuning will never be quite as turnkey as dreambooth/sd 1.5 LORA due to the nature of the training data.

I have been trying to figure out how to fine tune codellama. Will the llama2 examples work for codellama as well?

for startups I guess this means nail your use case with gpt-4, and when scaling cost becomes an issue consider fine tuning.

"to replace GPT-3.5/4"

Very inflated statement when it comes to GPT4 since it is a MoE model with 8 separate models each an expert in one area, and you can't replace all 8 models with one model trained for $19.

I call BS on this claim. Maybe it matches GPT4 in the narrow domain you fine-tune it for, and if that can be done for $19 then for $19*8 you can take OpenAI out of business. That doesn't add up.

Can you elaborate on your plans for OpenPipe? Sounds like a very interesting project

Currently OpenPipe allows you to capture input/output from a powerful model and use it to fine-tune a much smaller one, then offers you the option to host through OpenPipe or download it and host it elsewhere. Models hosted on OpenPipe enjoy a few benefits, like data drift detection and automatic reformatting of output to match the original model you trained against (think extraction "function call" responses from a purely textual Llama 2 response) through the sdk.

Longer-term, we'd love to expand the selection of base models to include specialized LLMs that are particularly good at a certain task, e.g. language translation, and let you train off of those as well. Providing a ton of specialized starting models will decrease the amount of training data you need, and increase the number of tasks at which fine-tuned models can excel.

Thanks! I need to dive into the project and learn more. Sounds exciting

Any compliance yet? HIPAA etc

What are your thoughts on fine tuning vs low rank adaptations?

Llama LORAs are very customizable, and range from "almost full finetuning" to "barely taking more VRAM than GPTQ inference"

This post made me think of human hierarchies. Line level ICs are cheap because they are specialized and fine tuned. Leet code is a way to roughly measure degree of fine-tuning even though it doesn't accurately measure how well the fine tuning is for the job.

As you go up the hierarchy what you want is higher quality answers to more and more abstract and general questions.

AGI, God, CEOs, and figures like Paul Graham, Elon Musk etc.. all answer to various degrees the ultimate abstract question of "What is the meaning of gestures wildly at everything"

Cost efficiency and commoditization basically increases "how" capacity at the cost of "why" capacity

> AGI, God, CEOs, and figures like Paul Graham, Elon Musk

hacker news pantheon just dropped

Can't we have something for the command line that takes the form of

    cat new_data.txt | finetune model.file > new_model.file

Sure, it would be trivial to turn the second notebook into a script that behaves this way.

just curious would it be possible to add a small network perhaps a books of study material like programming books. freeze the weights of the existing large network, and combined with the new network try to predict the book. The existing networks know language but not the content, the combined network will be trained on the content, and eventually toegther they score better, These "small" added networks might just be specific towards a certain topic (ea learn python or so). Then these small networks can be become modular. esesentially creating some kind of lora networks for LLM's.

Maybe start this way from the ground up, so you can get modular units, for health, finance, programming, education, writting assitance, phyloophy, ethics etc etc. If the modules can be changed, then one might be able to reduce their seize. Ea pick 2 or 3 chain them and one has a LLM for a specific area of interest. (reducing running cost)

This is part of what we're doing at Automorphic. Building shareable, stackable adapters that you can compose like lego bricks.

Applications are open for YC Winter 2024

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact