Note that fairness and equality are not the same thing. For example, what is the most fair way to share one cake among two people? First person cuts, second person chooses. It's unlikely that the cake is equally divided, but neither can object the outcome, if they agreed with the mechanism beforehand.
It's interesting that folks are slowly inching towards genuine algorithmic justice, but we are still focused on defining fairness from the POV of a single judge, and not from the entire integrated awareness of every participant in the community. I suppose that the latter is uncomputable...
Maybe I am misinterpreting your want, but this could be because some definitions of fairness directly contradict each other in some situations. So some unified view of fairness is not possible (in the view that I think you want).
Impossibility of Fairness: https://arxiv.org/abs/1609.07236
There are some frameworks to trade to balance competing fairness definitions (https://www.cs.toronto.edu/~toni/Papers/icml-final.pdf).
Somewhat related, but you have a trade off between calibration of a binary classifier and metrics on the positive and negative classes when you don't have equal base rates and similar ratios of some characteristics in the two classes: https://arxiv.org/abs/1609.07236
Even the cake cutting problem you mentioned below takes into account both POVs.
I'll also echo the sibling: what is true fairness? If it matches individual POVs, then why bother? If it doesn't then what's the use of it?
Concretely, imagine some judge Judy who is judging some dispute between some Alice and Bob. The article explores how we can evaluate Judy for fairness. However, justice is when Judy, Alice, and Bob all agree on an outcome which none of the three would change, even if they were to be forced to swap bodies in some unrelated way after the outcome is assigned.