Beyond the Surface: Spurious Cues in Automatic Media Bias Detection

Overview

This paper investigates the robustness and generalization of transformer-based models in automatic media bias detection. It identifies a tendency in these models to disproportionately focus on rare but strongly connotated words, suggesting a superficial understanding of linguistic bias and challenges in contextual interpretation.

Table of Contents

• [Methodology] (#methodology)
• Data
• Reproducibility
• Results
• License
• Citation

Methodology

• Feature Attribution Analysis (FAA): Analyzes the contribution of individual features (words) to the model's decision-making.
• Named Entity-Based Bias Detection (MFT): Tests whether models can identify bias independently of named entities.
• Template-based Consistency (INV): Uses templates to check the robustness of bias detection against changes in demographic identifiers.
• Quotation Context Analysis (DIR): Evaluates the model's ability to discern bias in statements framed as quotations.

Data

The data used in the methodology are publicly available under /data.

Reproducibility

To reproduce our results or evaluate your own models, follow the instructions:

1. install dependencies

pip install -r requirements.txt

2. Run the feature attribution analysis

• extract the attributions

  python src/scripts/get_feature_attributions.py
  

• visualize the results

  python src/scripts/visualize_attributions.py
  

3. Stress-test the model

   chmod +x src/scripts/stress-test.sh ; ./src/scripts/stress-test.sh
   

Results

Category	F1-Score
Gender	0.70
Male	0.68
Female	0.75
Non-Binary	0.69
Origin	0.98
European	0.94
African	0.99
Asian	1.00
Religion	0.86
Christianity	0.89
Islam	0.89
Atheism	0.80
Disability	0.84
Physical	0.84
Sensory	0.77
Neurodevelopmental and Mental Health	0.86
Political Affiliation (Politician Names)	0.92
Conservatives	0.97
Liberals	0.91
Socialists	0.89
Political Affiliation	0.86
Left-wing (liberal/progressive)	0.91
Right-wing (conservative)	0.80
Centrist (Moderate)	0.88
Occupation	0.67
Services	0.70
Creative Arts and Media	0.68
Skilled Trades and Manual Labour	0.64

License

This project is licensed under the MIT License. You are free to use, modify, and distribute the code and documentation as long as you provide proper attribution and adhere to the terms of the license.

Citation

# anonymous period