paulcjh's comments

On: Midjourney web experience is now open to everyone

[paulcjh]

We're one of the Black Forest Labs partners, you can try the models here: https://www.mystic.ai/black-forest-labs

On: Show HN: Turbo Registry – Rust Docker Registry for AI Cuts Cold Starts by 90%

[paulcjh]

Thanks!

On: Show HN: Turbo Registry – Rust Docker Registry for AI Cuts Cold Starts by 90%

[paulcjh]

Hey folks.

I’m Paul, the CEO of Mystic (YC W21). Today, we're launching our Turbo Registry, a new type of Docker registry built in Rust. It's designed specifically for AI/ML inference deployments, which typically have larger container sizes. Traditional Docker registries suffer from slow download times, which can be a significant bottleneck in AI inference workflows when going from 0->1. Our new registry has high bandwidth parallel downloading when connected with our new containerd adapter. This adapter changes the way images are mounted and downloaded and interfaces with a new V3 API specification we have introduced in our registry. We will be open-sourcing this later this year but will keep it closed-source as we develop it further for the moment.

With Turbo Registry, we achieved:

• 5GB Docker images loading in 10.23 seconds (down from 82.21 seconds)

• 10GB Docker images loading in 14.75 seconds (down from 147 seconds)

• 20GB Docker images loading in 23.72 seconds (down from 270.47 seconds)

You can use the registry today behind our serverless endpoints.

Information about the roadmap and current limitations can be found in the documentation. Check out our documentation here to get started with Turbo Registry and experience the speed boost in your AI projects. We’re looking forward to your feedback and contributions!

On: OpenAI – transformer debugger release

[paulcjh]

A very meagre attempt to look like they provide open source tools help the world safely make AGI

On: Show HN: Fractional GPUs for AI

[paulcjh]

There's not really a trade off, just less VRAM.

On: SambaNova Says Its First with Trillion-Parameter GenAI Model

[paulcjh]

The cost of their chips need to some down first, way too big and expensive

On: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s

[paulcjh]

Really cool!

On: VLLM with Mistral 7B guide and benchmarks (1.8k+ tokens/s)

[paulcjh]

Managed to get 1.8k tokens per second with a batch of 60 when running vLLM with Mistral 7B on an A100 40GB in bfloat16 mode. Pretty damn fast!

vllm==0.2.0 got released an hour or so ago, so it's pretty fresh. Let me know fi you'd like anything else in there.

[behnamoh → paulcjh]

I prefer GPT-4's low speed to any other model's fast speed because with these models, quality is the most important thing.

[purplecats → behnamoh]

i agree with your sentiment but keep in mind speed (slowness) could be a red herring. i find it plausible that while they degrade the quality of GPT4 in order to (presumably) lower their costs (while maintaining or increasing the price), they might add subtle slight delays to give the impression that the app is doing hard quality work.

kind of like that infamous android virus scanner app that just had a timer controlling the work in progress animation to give the impression of quality work being done.

On: Mistral AI 7B – beats Llama 2 7B and 13B

[paulcjh]

Amen to that!

On: Mistral AI 7B – beats Llama 2 7B and 13B

[paulcjh]

Their new 7B model beats the Llama 2 7B in all benchmarks that they provided, and in many cases the 13B variation. This is a basic demo for you to try it it, seeing the best results with the instruct variation, and JSON extraction seems good!