I wonder what other health measurements could be obtained with this technology? Sending radar through the wrist seems like a method that could observe much more information, although I am not sure which.
- Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat.
- Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up.
So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
Yes, this. I was skeptical and disgusted at a lot of what was being done or promised by using LLMs, but this was because I initially saw a lot of wholesale: "Make thing for me," being hyped or discussed.
In practice, I have found them to be good tools for getting going or un-stuck, and use them more like an inspiration engine, or brain kick-starter.
- Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat.
- Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up.
So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
This vulnerability isn't with the underlying OS though. They just installed a disabled application that has security concerns, but someone has to manually enable it for it to be a problem.
- Enable fall back controls where the pilot can input "traditional" controls when the more advanced system is degraded, even though it's all fly by wire their should still be inputs that fully mimic a traditional cockpit and ability to use this system should be allowed if the pilot requires it.
- Drop any focus or marketing on getting more people flying, if the ease of use of your aircraft is such that pilots who otherwise would not be qualified to fly now can fly, this is a recipe for disaster.
- Instead, focus on bringing a higher quality aircraft to market for pilots who want a more capable system, this cannot do any harm imo.
- Any system that lowers mental workload so that more focus can go into other areas of flying is welcome, just ensure there is always a method to fly the aircraft without the "auto" magic, there should always be controls that give raw control if needed even modern fly by wire commercial airliners have this fall back ability.
If you watch the video, they are advertising two inputs: a stick (which doesn't act like a stick) and a speed suggestion (not actually a throttle.) If the system were to suddenly degrade to standard inputs, it could be confusing and potentially lethal.
I don't think this is a system for pilots. This is a system for ... someone else.
> if the ease of use of your aircraft is such that pilots who otherwise would not be qualified to fly now can fly, this is a recipe for disaster.
How does that work? If they’re now qualified to fly then clearly they meet the standards set out for that. It could be argued that if they meet those standards but would be a danger to themselves and others in a different plane, it’s still a net win.
How would a server/workstation like this be setup?
I thought you could only use the vram on the GPU, so for 700GB you would need 8-9 A100 nodes as 2 only gives 160GB.
I've been trying to figure out how to build a local system to run inference and train on top of LLM models, I thought there was no way to add vram to a system outside of adding more and more GPU's or use system ram (DDR5) even though that would be considerably slower.
I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia?
If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone.
From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific GPU's.
Nvidia has decades of experience selling hardware to people with all the pains that entails, support, sales channels, customer acquisition, software, it's something you don't just do overnight, and it does cost money. Google's TPUs get some of their cost efficiency from not supporting COTS use cases and the overhead of selling to people, and the total wall clock time has to also include the total operational costs, which dominate at their size (e.g. if it's 30x slower but 1/50th the TCO then it's a win. I don't know how TPUv5 stacks up against the B200). It's not as simple as "just put it on a shelf and sell it and make a gajillion dollars like nvidia"
TPU v5p is ~2 times slower than H100 at larg(ish)-scale training (order of 10k chips) [1]. And they already have v6 [2]. I think it's safe to say that they are fairly close to Nvidia in terms of performance.
We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?
If you'd like to contribute some much-appreciated compute to university researchers all across the US, please email me (mhsiu at ucsd.edu).
It should be possible to hook up your idle devices to Nautilus (https://nationalresearchplatform.org/), which is a cluster set up to support researchers at a bunch of universities. I can't guarantee anything since I'm not involved in the cluster management itself, but I can put you in contact with those who are if you're interested.
I know of many important projects that need GPUs right now and aren’t getting any. You could help motivate the ponydiffusion folks to actually try finetuning of SD3!
Exactly and they are still about 1/18ths as good at training llms as a H100.
Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.
Given that Google invented Transformer architecture (and Google AI continues to do foundational R&D on ML architecture) — and that Google's TPUs don't even support the most common ML standards, but require their own training and inference frameworks — I would assume that "the point" of TPUs from Google's perspective, has less to do with running LLMs, and more to do with running weird experimental custom model architectures that don't even exist as journal papers yet.
I would bet money that TPUs are at least better at doing AI research than anything Nvidia will sell you. That alone might be enough for Google to keep getting some new ones fabbed each year. The TPUs you can rent on Google Cloud might very well just be hardware requisitioned by the AI team, for the AI team, that they aren't always using to capacity, and so is "earning out" its CapEx through public rentals.
TPUs are maybe also better at other things Google does internally, too. Running inference on YouTube's audio+video-input timecoded-captions-output model, say.
If you're interested in a peer reviewed scientific comparison, Google writes retrospective papers after contemporary TPUs and GPUs are deployed versus speculation about future products. The most recent compares TPU v4 and A100. (TPU v5 and H100 is for a future paper). Here is a quote from the abstract:
"Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized systems, it is ~4.3x--4.5x faster than the Graphcore IPU Bow and is 1.2x--1.7x faster and uses 1.3x--1.9x less power than the Nvidia A100. TPU v4s inside the energy-optimized warehouse scale computers of Google Cloud use ~2--6x less energy and produce ~20x less CO2e than contemporary DSAs in typical on-premise data centers."
That quote is referring to the A100... the H100 used ~75% more power to deliver "up to 9x faster AI training and up to 30x faster AI inference speedups on large language models compared to the prior generation A100."[0]
Which sure makes the H100 sound both faster and more efficient (per unit of compute) than the TPU v4, given what was in your quote. I don't think your quote does anything to support the position that TPUs are noticeably better than Nvidia's offerings for this task.
Complicating this is that the TPU v5 generation has already come out, and the Nvidia B100 generation is imminent within a couple of months. (So, no, a comparison of TPUv5 to H100 isn't for a future paper... that future paper should be comparing TPUv5 to B100, not H100.)
I'm sure it probably is faster for thier own workloads (which they are choosing to benchmark on), why bother making it if not. But that is clearly not universally true, a GPU is clearly more versatile. This means nothing to most if they can't for example train an LLM on them.
I don't see how you can evaluate better and worse for training without doing so on cost basis. If it costs less and eventually finishes then it's better.
This assumes that you can linearly scale up the number of TPUs to get equal performance to Nvidia cards for less cost. Like most things distributed, this is unlikely to be the case.
The repo mentiones a Karpathy tweet from Jan 2023. Andrej has recently created llm.c and the same model trained about 32x faster on the same NVidia hardware mentioned in the tweet. I dont think the perfomance estimate that the repo used (based on that early tweet) was accurate for the performance of the NVidia hardware itself.
Google claims the opposite in "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
" https://arxiv.org/abs/2304.01433
Despite various details I don't think that this is an area where Facebook is very different from Google. Both have terrifying amounts of datacenter to play with. Both have long experience making reliable products out of unreliable subsystems. Both have innovative orchestration and storage stacks. Meta hasn't published much or anything about things like reconfigurable optical switches, but that doesn't mean they don't have such a thing.
I'm only aware of Nvidia AI Enterprise and that isn't required to run the GPU.
I think it's aimed at medium to large corporations.
Massive corporations such as Meta and OpenAI would build their own cloud and not rely on this.
The GPU really is a shovel, and can be used without any subscription.
Don't get me wrong, I want there to be competition with Nvidia, I want more access for open source and small players to run and train AI on competitive hardware at our own sites.
But no one is competing, no one has any idea what they're doing. Nvidia has no competition whatsoever, no one is even close.
This lets Nvidia get away with adding more vram onto an AI specific GPU and increase the price by 10x.
This lets Nvidia remove NVLink from current gen consumer cards like the 4090.
This lets Nvidia use their driver licence to prevent cloud platforms from offering consumers cards as a choice in datacenters.
If Nvidia had a shred of competition things would be much better.
> If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone.
While I do not actually think Google's chips are better or close to being better, I don't think this actually holds?
If the upside of <better chip> is effectively unbounded, it would outweigh the short term benefit of selling them to others, I would think. At least for a company like Google.
They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs.
> I can see Google's custom chips are 15x to 30x slower to train AI
TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for inference to serve their billions of customers is far greater than training costs for models - even LLMs. They've been running model inference as a part of production traffic for years.
This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling
That’s not necessarily true. Many companies make chips they won’t sell to support lucrative, proprietary offerings. Mainframe processors are the classic example.
In AI, Google (TPU) and Intel (Gaudi) each have chips they push in cloud offerings. The cloud offerings have cross selling opportunities. That by itself would be a reason to keep it internal at their scale. It might also be easier to support one, or a small set, of deployments that are internal vs the variety that external customers would use.
Intel, AMD and others also have chips for training that perform close to or sometimes better than Nvidia's. These are already in the market. Two problems: the CUDA moat, and, "noone gets fired for buying green".
In the dark notice how much light comes from a phone and how much that illuminates the person holding the phone.
A car headlight shining your way from hundreds of meters even over 1km will often illuminate that person far more than their phone screen.
So if the phone screen is enough to undo the hour long eyes adjusting to the dark duration, then the car headlights at almost any realistic distance causes an undo.
a headlight a kilometer away may be spread over 36000 square meters including your face. (more detailed info on headlight antenna gain would be appreciated.) a 1200-lumen high beam over that area is 0.03 lux
a 500 nit phone screen emitting over about a steradian from an 80mm × 160mm area is 6.4 lumens, but at night you usually turn it down to minimum brightness, say 0.6 lumens. (i haven't measured.) at a distance of 300mm that steradian is 0.09m², so it's about 6 lux
so the cellphone is about 200 times brighter than the headlight at that distance, and that's also observable from looking at people using cellphones walking on the highway
but the person i was asking for more detail didn't specify that the headlight is pointed at your face; and in most situations where you can see headlights, they're not pointed at you. that's just a detail you filled in
reply