The price of fairness - A framework to explore trade-offs in algorithmic fairness

With the increase in automated decision making using predictive analytics, the aspect of fairness of the resulting predictions for specific groups is increasingly considered in research and practice. Currently, the actual trade-off to achieve fairness, or a certain level of fairness, is not well understood, other than that an increase in fairness typically decreases other predictive analytics metrics such as accuracy. To enable a systematic evaluation of potential trade-offs between fairness and other metrics, a framework for exploring Algorithmic Fairness is proposed. Using a combination of multi-objective optimization and Pareto fronts, the framework allows for the exploration of fairness-performance trade-offs and enables the systematic comparison of different algorithmic techniques to increase fairness. A case study compares several fairness metrics and different algorithmic techniques, provides insight into trade-offs found between metrics, and shows how the framework can be leveraged to find a 'best' level of fairness for a given scenario.