On: 100K Context Windows

[iknowstuff]

Even a small startup, a researcher or a tinkerer can get a cloud instance with a beefy GPU. Also of note, Apple's M1 Max/Ultra should be be able to run it on their GPUs given their 64/128GB of memory, right? That's an order of magnitude cheaper.

[mejutoco → iknowstuff]

I am confused. Those amounts are ram, not gpu ram, aren‘t they? Macs cpus are impressive, but not for ml. A most realistic one for a consumer is a 4090 rtx 24 GB. A lot of models do not fit in that, so A6000 48GB and over for some professional cards. That might be around 9000€ already.

[piperswe → mejutoco]

Apple Silicon has unified memory - all memory is accessible to both the CPU and GPU parts of the SoC.

[karmasimida → piperswe]

But they comes at max 32GB model?

[mkl → karmasimida]

Mac Studio (desktop) is up to 128GB, and Macbook Pro is up to 96GB.

[codedokode → mejutoco]

> Macs cpus are impressive, but not for ml

On Mac GPU has access to all memory.

[himlion → iknowstuff]

I overlooked the unified memory on those machines. Can it really run this performantly?

[ukuina → himlion]

I run Vicuna quite well with my M1 Pro, 32GB.