On: Fine-tune your own Llama 2 to replace GPT-3.5/4

[haxton]

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

[JackRumford → haxton]

These sites say 154B:

https://www.ankursnewsletter.com/p/gpt-4-gpt-3-and-gpt-35-tu...

https://blog.wordbot.io/ai-artificial-intelligence/gpt-3-5-t...

[why_only_15 → haxton]

gpt3.5 turbo is a new model, not Curie. As others have stated, it probably uses Mixture of Experts which lowers inference cost.

[csjh → haxton]

Is there a source on that? I've never seen anyone think it's below even 70B

[ronyfadel → haxton]

It still does a much better job at translation than llama 2 70b even, at 6.7b params

[two_in_one → ronyfadel]

If it's MOE that may explain why it's faster and better...

[yumraj → two_in_one]

MOE?

[sarthaksrinivas → yumraj]

Mixture of Experts Model - https://en.wikipedia.org/wiki/Mixture_of_experts

[jiggawatts → haxton]

I thought it was fairly well established that GPT 3.5 has something like 130B parameters and that GPT 4 is on the order of 600-1,000

[avion23 → jiggawatts]

I remember:

- gpt-3.5 175b params

- gpt-4 1800b params