If I operate a nuclear reactor or a hydroelectric dam there are regulators that tell me what i'm allowed to do, so as to keep my profit motive from overwhelming the public interest.
If we want AI to actually have some safety rails, this is what we would do.
If we were to take the nuclear analogy, what's happening in AI right now is where the people selling nuclear power make a ton of noise about how they need the power to regulate their competitors because nuclear bombs might set the atmosphere on fire, and their proof for this is in a report about how they didn't wear TLDs (and suffered related increased cancer risks) despite it being common practice to wear them in all related industries.
Some controls are justifiable, but none of the people involved in any of this can be trusted to develop sane controls. Most likely we're looking at draconian proposals similar to attempted regulations on 3d printers.
If we want AI to actually have some safety rails, this is what we would do.