I’m sharing the beta of Hulse, which helps teams deploy open-source NLP models from the Hugging Face model hub with their unused personal computing power. I’m curious to hear your thoughts!
Hulse works by running “hosts” on personal computers (either through a macOS app built with py2app & rumps, or using the python CLI) and using the Hugging Face transformers library under the hood to load models from the model hub. Each user joins/creates a set of “clusters” which gives them access to the computing resources of other cluster members. When a user issues a query to process data for a given model, the query gets transferred to an available host belonging to a shared cluster for processing. Once results become available, they get forwarded back from the host to the user.
The relay server between users and hosts is currently implemented using Django channels, with a plugin called Django EventStream. It maintains an HTTP stream request open and enables server notifications without having to go through a p2p protocol.
I’d love to get your feedback and see if anyone has a use case/feature they’d like to see covered for future iterations. Let me know what you think!
My $0.02 is Cloud machines are easier to work with and give you control over your hardware. If my local cluster isn't powerful enough to train the data (e.g. high RAM requirements), then I would have to fallback to cloud computing anyways.
Plus, there might be data access risks. There might be data that I should only have access to that my colleagues machines shouldn't see. In the context of GDPR, how do I know geographically where the server are running?
Consider where the data is stored. Most company data lives in cloud s3 buckets and AWS charges hefty fees to access that data outside of their network. Downloading terrabytes of training data locally could leave a large AWS bill.
Hulse focuses on simplifying the deployment of pre-trained models at the moment, so you could expect to still run your training on a proper GPU cluster. On the ease of use side of things, I'd argue cloud VMs are not necessarily easier to work with: setting up your firewall & all the APIs to handle queries towards your pre-trained models is pretty cumbersome.
On average a PC is used at around 30% of its computing power, thus leaving a good chunk of CPU/RAM available which would be more expensive to rent in the cloud than use locally.
I completely agree with you about the data access with colleagues. Right now, the user right management is very simple: you share a cluster with people, who may come to analyze your data and vis-versa. This could be an interesting addition!
Right now, the relay servers are hosted in the US by AWS. Running things in a p2p fashion could also be done (thus removing the GDPR problem - if combined with the above point, could help give the desired granularity).
Considering the AWS S3 egress bills, I agree with you! I think vendor lock-in is a problem especially when it constrains you to rent more expensive cloud servers (here AWS) to avoid extra costs.
I’m sharing the beta of Hulse, which helps teams deploy open-source NLP models from the Hugging Face model hub with their unused personal computing power. I’m curious to hear your thoughts!
Hulse works by running “hosts” on personal computers (either through a macOS app built with py2app & rumps, or using the python CLI) and using the Hugging Face transformers library under the hood to load models from the model hub. Each user joins/creates a set of “clusters” which gives them access to the computing resources of other cluster members. When a user issues a query to process data for a given model, the query gets transferred to an available host belonging to a shared cluster for processing. Once results become available, they get forwarded back from the host to the user.
The relay server between users and hosts is currently implemented using Django channels, with a plugin called Django EventStream. It maintains an HTTP stream request open and enables server notifications without having to go through a p2p protocol.
I’d love to get your feedback and see if anyone has a use case/feature they’d like to see covered for future iterations. Let me know what you think!
Website: https://hulse.app Github: https://github.com/hulsedev/hulse-py