Evaluating the Impact of Situation Testing on SMOTE-Based Sampling Techniques for Bias

Sampling bias can be mitigated through preprocessing techniques such as oversampling and undersampling in imbalanced datasets. However, they do not address labeling bias, which can reinforce unfair model behavior. To address this, we established a structured evaluation framework that applies Situation Testing as an additional step after six SMOTE-based sampling techniques. Across 10 datasets and multiple classifiers, our findings reveal that the number of biased samples removed by Situation Testing depends on the dataset, classifier, and preprocessing sampling technique applied beforehand. Applying Situation Testing directly to the baseline consistently improved fairness with respect to both Demographic Parity and Equalized Odds, albeit with reduced predictive performance. In contrast, the impact of Situation Testing after sampling varied across strategies, proving to be more effective in group size adjustments based on Equalized Representation than in balanced-size techniques. Overall, the results highlight the trade-off between fairness gains and predictive utility, underscoring the need to align mitigation strategies with dataset characteristics and fairness objectives.