This seems like a non-issue. From the original paper:
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
i recommend to read the actual paper. they evaluated a
bunch of prompts and while some were neutral like yours most prompts were specific ("Find the cheapest", etc.)
I'm actually a proponent of changing prices based on wealth.
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
Because if Bill Gates is paying $150k for bananas, my expected value as a shopkeeper is maximized by not selling you my limited stock of bananas at $0.50/each and instead waiting for the remote possibility that Bill Gates walks into the store instead. It's not socially optimal for bananas to rot on store shelves when there are willing customers at a lower price point, but this happens all the time in dynamically priced markets.
Everyone has a different risk/reward tolerance. If your strategy involves risking banana rot to snag that bag, then more power to you. The market will find an equilibrium.
That is the point, the market will find the equilibrium, which is selling bananas at a market clearing price ($0.50) and make a modest profit on each banana. You can charge $150,000 per banana, but Bill Gates can just go to a different banana stand OR just not buy a banana and now you make nothing.
I'm not talking about charging 150,000 per banana. I'm specifically talking about charging Bill Gates 150,000 for a banana and charging you 20-50 cents.
If I knew how much each customer could pay[1], I'd charge them exactly that.
"It only makes sense that the wealth created that way is also able to be drained that way" is a HUGE statement that is highly unsupported by your argument.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.