On: Yi-34B-Chat

[TekMol]

    Model is too large to load onto the free Inference API.

I don't understand which models one can use for free on HuggingFace and which not.

For example this 180B model works:

https://huggingface.co/spaces/tiiuae/falcon-180b-demo

Why? Is someone paying for it so I can use it for free?

[nodja → TekMol]

Yes, an A100 costs $4.13/hour

https://huggingface.co/pricing#spaces

[TekMol → nodja]

So who is paying for the freely available models?

For example there is StableDiffusion XL here:

https://huggingface.co/spaces/latent-consistency/lcm-lora-fo...

[nodja → TekMol]

The owner of the space.

[nwoli → nodja]

Sometimes huggingface pays for them if they’re popular as a sort of ad for HF spaces

[amelius → nodja]

Makes me wonder how much money OpenAI is burning ...

[dragonwriter → TekMol]

SDXL is only a 6.6B parameter model, its not a big model compared to LLMs.

[milansuk → nodja]

Yeah, they probably don't want their 'demo' links on the HN home page.

[Der_Einzige → milansuk]

Yes they do want those links on HN home page. Clout/likes/stars are worth more than GPU credits to most.

[antirez → TekMol]

Just use llama.ccp and run the quantized model like yi-34b-chat.Q4_K_M.gguf (you will need a 32GB Macbook, likely).

[numpad0 → antirez]

Note also that 2x32GB DDR4 UDIMM kits are ~$120. It's still an overkill for most purposes, but a cheap one.

[Der_Einzige → antirez]

Highly quantized models are crippled versions of the original model. Don't evaluate a model for a production use case based on it's quantized versions.

[nacs → Der_Einzige]

4_k_m and 5_k_m are very good quant levels that don't sacrifice much.

[eigenvalue → Der_Einzige]

I find the larger Q5 versions from theBloke to be pretty good and allow you to run a larger model than you’d be able to otherwise. But there is definitely an impact, although it’s hard to quantify. It’s sort of like how you get less effective at coding when you’ve only slept 5 hours.

[antirez → Der_Einzige]

This depends on the quantization level. Sure, they lose part of the performance, but are still quite useful and cheap to run.

[BoorishBears → TekMol]

That 180B is also too large for the free inference API:

https://huggingface.co/tiiuae/falcon-180B-chat

HF has a spare capacity based free inference tier

[nulld3v → TekMol]

That link is a HuggingFace "space". The owner of the space selects what GPU is available to the space and pays for all GPU usage.