Bias Detection and Analysis in Transformer-based Language Models using Semantic and Layer-wise Representations

The rapid adoption of large language models in real-world applications has raised critical concerns regarding embedded biases in their representations and outputs [1], [2]. These biases, often inherited from training data, can influence decision-making systems in subtle yet impactful ways. This study presents a comprehensive framework for detecting and analyzing bias in transformer-based language models using both semantic similarity measures and internal representation analysis. Multiple models, including BERT, RoBERTa, DistilBERT, Sentence-BERT, and selected large language models, are eval- uated across standardized bias benchmarks [3], [4], [5], [6], [7], [8]. The approach combines embedding-based techniques such as Word Embedding Association Tests with layer-wise inspection of hidden states to identify how bias manifests across different architectural depths [9], [10]. The results highlight that bias is not uniformly distributed across models or layers, with certain intermediate representations exhibiting stronger associations. Furthermore, differences beten encoder-based and generative models suggest distinct bias prop- agation mechanisms [11], [2]. This work contributes a unified pipeline for bias evaluation and offers insights into improving fairness in modern language systems.