At its core, it is price fixing, and already technically illegal.
There is no sound argument that can be made that weights in AI software programed in don't have biases created by human's, and won't ever engage in illegal activity. They are just trying to make the argument, its not illegal because an AI does it.
We have already seen how Amazon uses their bait and switch algorithms (dynamic pricing) and there is more to learn about with regards to Project Nessy.
It is anti-consumer activity, which can only occur in concentrated business sectors (oligopoly/monopoly).
Price fixing has several types but is more normally called horizontal price-fixing, this is where a group of vendors collusively and collectively agree to a price schedule or range (indirectly or directly); to set minimum or maximum prices (indirectly or directly); to advertise prices cooperatively or to restrict price advertising. It is a violation of the Sherman Antitrust act.
There is a real estate company who used such software that is currently going through the courts for doing exactly this through their rental price algorithm in their software suite, to unjustly raise rents. They are now being taken to task in court.
The most optimal path to profits is through collusive market manipulation. It doesn't matter if an AI algorithm does this or a person, because there is always a person that was the agent that directed the training for the algorithm towards a specific outcome at the beginning, and they had a care of due dilligence testing that should have been done prior, to ensure outcomes remained within the bounds of law.
By choosing to use an algorithm to determine prices in your business that others use, you are actively engaging in price fixing because you are cooperatively advertising prices (based on the algorithm). This occurs in parallel as well where dynamic pricing (Amazon) actively engages in anti-consumer behavior known commonly as a bait and switch.
You came to buy something, but now you have to pay more because the price went up after they saw you walk in (or any number of other bots with regards to website traffic, a bot does not equal a person). You had cost getting there, and you need a product but the market didn't dictate that increase in price. The algorithm did, which may be based on flawed manipulatable inputs.
There of course will always be deceitful language and messaging around this because the people in charge of integrating these system's know this is illegal, but that won't stop them from testing it in the courts (as they've done time and time again), which is why fines don't work. Costs are passed on, and they will always profit more than they lose out in these areas.
AI has offered another dubious layer of indirection for all the things that have already been shown to be unlawful, people claim its not this outcome if an AI does it (which is almost certainly false in most cases).
You can program an AI to do anything, including evil, unethical, and illegal acts.
I think OP means "price gouging" rather than fixing. Or, if you're a libertarian and want a positive spin, "increasing prices to exactly what the market will bear".
No I said exactly what I mean, when you cooperate on prices you engage in price fixing, it doesn't matter if you direct an AI to do it for you, or not; what matters is you chose to cooperate on your pricing.
The FTC has literature posted on this exact subject.
Also that spin you put forth as libertarian isn't correct and misleads.
It takes economic verbiage in isolation, and overgeneralizes to falsity.
It neglects price inflexibility, and a number of other dynamics, instead assuming the market is static and that you can raise the prices to whatever the market will bear (which also neglects price fixing and other illegal activities such as bait and switching).
Value is subjective, and changes with each valuation done by an individual.
There is no sound argument that can be made that weights in AI software programed in don't have biases created by human's, and won't ever engage in illegal activity. They are just trying to make the argument, its not illegal because an AI does it.
We have already seen how Amazon uses their bait and switch algorithms (dynamic pricing) and there is more to learn about with regards to Project Nessy.
It is anti-consumer activity, which can only occur in concentrated business sectors (oligopoly/monopoly).