I don't really see the point in hosting a tiny fraction of raw Falcon 180B or whatever when I can fully host a small finetune, of my choosing, that will perform better for what users actually want (which is mostly RP/Chat at the moment). And I can serve dozens of users on a single 2060 with a Mistral finetune.
Now, if it was a finetuning cluster, I would throw my 3090 at it without even blinking, but my understanding is that Petals doesn't finetune yet.
I personally would do it in return for the electricity cost of the GPU's and machine. That's my marginal price to do so.
However, that offer is only valid if a project can guarantee my computer experience won't be degraded. So it needs to reliably run only during idle time, or, if it wants to run while I'm using the machine it can't make any application laggy or throw errors about not having enough GPU RAM.
I think there are probably millions of GPU's available out there with similar terms to mine.
https://lite.koboldai.net/#
I don't really see the point in hosting a tiny fraction of raw Falcon 180B or whatever when I can fully host a small finetune, of my choosing, that will perform better for what users actually want (which is mostly RP/Chat at the moment). And I can serve dozens of users on a single 2060 with a Mistral finetune.
Now, if it was a finetuning cluster, I would throw my 3090 at it without even blinking, but my understanding is that Petals doesn't finetune yet.