And the best way to determine the right value is exactly that what you wrote: the owner sets the price of the portfolio, the owner would pay the fees accordingly, and the licensing fee would be a percentage of the total value. And if refused a license, any potential licensee can buy the remaining years of the patent at the initial annual price (= portfolio value / portfolio lifetime).
So if the patent has value but the inventor has set the price too low to save on fees, someone can buy it and make better use of it. And setting the price too high would have to be a reflection of the inventor's trust that the patent is worth it since the inventor is going to have to pay for it each year.
If it sounds unfair between big companies and small companies and individuals, some relative pricing can be established using percentages of the inventor's turnover / annual income. Therefore, it will hurt you for 1% or something, no matter how wealthy you are or how big company you are.