> it may be possible to achieve a 100× energy-efficiency advantage
Running the math on a machine with 8x A100 (enough to run today's LLMs), that would be 300w * 8gpus / 100 = 24w.
This is within striking distance of IOT and personal devices. I'm trying to imagine what a world would look like where generative text models are commodetised to the point where you can either generate text locally on your phone, or generate GBs of text in the cloud.
I have to admit it's very hard to make any sort of accurate prediction.
> 100× energy-efficiency advantage for running some of the largest current Transformer models, and that if both the models and the optical hardware are scaled to the quadrillion-parameter regime, optical computers could have a >8,000×
Maybe I interpreted that incorrectly but I thought it's saying a 100x advantage for current large Transformer models, and 8000x advantage for future quadrillion-parameter models? I didn't include those because I suppose that size of model is quite a few years away. Admittedly this is only based on the abstract...
Need to compare this with custom silicon like Apple will be shipping.
They already have the Neural Engine chip which can run Stable Diffusion, but eventually you could imagine casting a specific model instance to an ASIC (say GPT-3.5 or -4, today).
If most devices are replaced within a year or two then you get a pretty good cadence for updating your Siri model (and even more incentive for users to upgrade hardware).
You don't, because of the scaling law they say they've identified. If optical energy per MAC operation scales as 1/d, we know two things: 1) there is no electronic architecture possible that can catch it, and 2) bigger models give optical networks a bigger energy advantage.
It's possible to have a temporary lead because of constant factors, but as long as an electronic circuit has to expend a unit of energy per MAC, you'll always be able to specify a model big enough that an optical network will beat it.
1) this is a research device and a theoretical scaling law; it’s not been proven.
> We conclude that with well-engineered, large-scale optical hardware, it may be possible to achieve a 100× energy-efficiency advantage
Emphasis on may.
2) in the real world, constant factors matter (as you allude to). For example if an ASIC gets a 1000x speedup (optimistic; we saw this for BTC) it might be the better choice for this generation, but start to lose next gen and beyond. If an ASIC only gets 100x or lower then it’s not favorable this gen.
So sure, this tech might win in the long term, but I wasn’t making any categorical claims, just noting that there are multiple horses we need to track.
It would be quite foolish to dismiss custom silicon solutions based on this paper.
You can run it on either now (for example, MochiDiffusion allows you to pick https://github.com/godly-devotion/MochiDiffusion#compute-uni...). Anecdotally, the GPU seems to be faster for an M1 Max or up GPU, the ANE is a touch faster on anything smaller, and more power efficient in general.
It will be also at least 100x times (physically) larger, because optical wavelength is ~1000nm vs 10nm of electronic gate size. So much for personal devices.
True but I think the difference is smaller than one might expect from pure element size.
IIRC one reason we don't already have fully 3D chips is because of the heat dissipation. Reducing 2400 W to 24 W means the heat is much more tractable, which means it can be closer to volumetric than planar.
Consider a 1cm*1cm*1mm chip; with 1μ^3 elements, 1e11 per chip; with (10nm)^3 elements limited to one layer because of heat, 1e12.
Yes this is still a factor of x10, and chips are a few layers because while heat is a problem it's not a total blocker, but it's still much less than the 100^3 ratio a simple scale-up would result in.
Maybe, but then we need to make sure light does not seep into neighboring cells, will need metal shields for that, and then heat will dissipate ... maybe they could solve this in the future.
What will happen, such a thin tube will not be able to confine the electromagnetic wave within its boundary, so most of the wave's field will propagate outside of the tube and will quickly diffract on any bumps it encounters.
>I'm trying to imagine what a world would look like where generative text models are commodetised to the point where you can either generate text locally on your phone, or generate GBs of text in the cloud.
Dead internet theory for one. Scalable spear phishing and scams. Scalable automated offensive hacking. SEO far worse than anything possible today. Mass manipulation campaigns.
Social interaction would also be strange. Every messenger and dating app able to automatically reply and suggest sophisticated messages.
I/O is already meeting those performance levels on today's technology.
You can get a hundred dollar SSD that has a read speed around 7 gigabytes per second using just over five watts. That will fill up 8x80GB in a minute and a half of load time. If your energy budget is 24 watts then install four and make it 20-25 seconds.
As far as cost, I don't know what the proposed chip would be, but $400 is 0.5% of that pile of GPUs and SSDs will only get cheaper.
From the info in the article alone, it's impossible to tell if this is even a correlation at all. I don't know what the expected number of reports would be for a given year, and if 30k is higher or lower than that.
> If a link between vaccination and menstrual changes is confirmed
Even the wording in the article stops short of claiming there is evidence for any link. It's just a bad headline.
Reddit was funded by ycombinator in summer ‘05, which probably explains part of it. Then the rest I would say is down to similarities between the two sites (primarily acting as link aggregators, although reddit has gradually diverged from this).
Has anyone heard of good resources that teach you "when to use X" instead of "why you should use X"?
It's somewhat understandable that the official websites of frameworks/tools/libraries/languages/services tend towards espousing the advantages of the thing and leaving out most of the drawbacks. That's just how the incentives are aligned. But even the average blog article about a tool posted on HN rarely talks about what situations it is most effective, and what situations it should potentially be avoided. I suppose it's a natural consequence of blog authors only knowing so many tools, no single person knows them all.
I'm not saying this information isn't out there – just that it seems like the exception rather than the rule.
> Has anyone heard of good resources that teach you "when to use X" instead of "why you should use X"?
One of the goals of a podcast I started last year was to answer that type of question and more.
It's mainly a podcast where I talk to someone new every week to get a better understanding of why they chose a specific tech stack to build their app but it also goes into the "how" since we talk about how they build and deploy their app, lessons learned along the way, etc..
Each episode has a "Motivation for using ..." time stamp so you can jump to that part of the discussion. It's usually within the first 5 or 10 minutes.
The goal here is to have a bunch of real world examples of why and when someone used X and they usually spend a few minutes talking about why they chose that instead of something else too. There's episodes ranging from solo developers working on a side project to high traffic services handling billions of requests per month.
The idea is to have loads of practical examples instead of trying to theory craft hypothetical scenarios.
Around 4-5 months ago I implemented a custom player so now it's really easy to listen at 2x speed, and there's always been timestamped links to quickly jump around to specific topics.
I want that protocolization too, although I don't hold out much hope that the monopolies in place can be broken, outside of fairly radical regulation.
To add to your list, there is also Secure Scuttlebutt [1] which has had a decent userbase over the past few years, and Planetary [2] which is a funded iOS client for it.
I think in general they all suffer from the chicken-egg problem and will need some reason for enough people to switch to be able to build a userbase. There isn't really any "novel hook" like tiktok, twitter, whatsapp, instagram, snapchat, etc have had in the past.
Man, I love the idea of Scuttlebutt but I hate the developer UX. I'm writing some apps that I wanted to put on SSB but have all but given up on the idea. Something about SSB, as a dev, leaves me with a lot of questions and no idea where to even get answers from.
So I'll write my app outside of SSB, hopefully in a way that's mostly compatible, and possibly with future integration.
I may also toy with an SSB-like protocol myself, as the fundamentals of SSB is a work of art imo. I really enjoy what Gossip brings to the table, and how SSB focuses on human->human relationship to bring P2P to the table.
I've been doing more of a product-ish role in my dayjob, so it's nice to do something (relatively) well-defined, where it's easy to tell if things are working, but still challenging enough to be fun.
Running the math on a machine with 8x A100 (enough to run today's LLMs), that would be 300w * 8gpus / 100 = 24w.
This is within striking distance of IOT and personal devices. I'm trying to imagine what a world would look like where generative text models are commodetised to the point where you can either generate text locally on your phone, or generate GBs of text in the cloud.
I have to admit it's very hard to make any sort of accurate prediction.