> but that a system "perfectly designed for outcomes humans like" is equivalent to creating strong AI that happens to run on a market economy.
Isn't that assuming that "market economy" is something "perfectly designed for outcomes humans like"? I disagree with treating current economic system as something inherently optimal - it has a lot of problems, all of which we can see even today. I agree with the equivalence to creating strong AI, but I don't think it would run on anything close to a market economy we have today.
I think we agree. My intent was not to laud the market so much as to kill the hope of a perfectly-designed system by showing that the thing requires Elua to make it really "work out well" from our perspective.
We do a whole lot of stuff to satisfy Gnon---marketing, sales, firing people, etc. Sometimes that happens to also satisfy Elua, like when we get iPhones. But Gnon MUST be resisted sometimes, and I think philanthropy is one of those times.
Philanthropy is what happens when you manage to escape the reach of Gnon, even for a little while. People like Bill&Melinda are able to directly address the needs that neither markets nor governments can satisfy.
> I'm convinced that it is only the good faith of enough actors that allows any system to function at all.
I think that how many actors displaying good faith we accumulate in a single system is a degree to which we have control over Gnon. It's actually sad that as societies instead of working to increase trust in one another, we're trying to take it out of the equation. We're literally sacrificing our futures to Moloch in exchange for temporary security. The less a system is interruptable by humans enforcing human values, the more any error in encoding those values in the system is squeezing us (hence, FAI problem), and we're both aware of how such systems tend to evolve.
ETA
The way some people don't understand how we're screwing ourselves over with market economy reminds me of a thing I learned in high school physics class.
The equation for potential energy in a gravity field goes like E = -G m_a m_b / r, with r being the distance between the two attrated bodies. But when you're learning about gravity for the first time, you're being taught the E = mgh equation, where g is a constant acceleration and h is the height above the surface. In one case, energy is proportional to 1/r; in the other, it is proportional to r. So how do you reconcile those two views, and how is it that people didn't stumble upon the first equation before the second one, and that we can do a lot of good computations with it?
When you draw a graph of E(r) (first equation) you'll get a hyperbola. But if you zoom in hard enough, to the area of typical values of r we encounter daily, you'll see a straight line. Gravitational attraction linearizes pretty well in the range of parameters we're used to deal with every day.
In a similar way, there is a disconnect between human values and the market that is supposed to help optimize for them. They are aligned, but not perfectly. Many people seem to have zoomed in so far that they see a straight line where they should see a hyperbole, a growing deviation from desired course.
This is exactly what we're seeing today. To put it bluntly, market economy, or Moloch in general, has outlived its usefulness. It is optimizing values increasingly decoupled from our own, so it's high time to start thinking how to get rid of it.
Isn't that assuming that "market economy" is something "perfectly designed for outcomes humans like"? I disagree with treating current economic system as something inherently optimal - it has a lot of problems, all of which we can see even today. I agree with the equivalence to creating strong AI, but I don't think it would run on anything close to a market economy we have today.