Measuring Bias In Large Language Models: A Comparative Evaluation of LLaMA3.2-1B and LLaMA3.1-8B Across Indian Socio-Culture Dimension

In this paper we present a 30, 000-variant India-context bias audit to compare LLaMA3.2-1B and LLaMA3.1-8B across categories of Gender, Religion, Profession and Region that proposes the India Context Sensitivity Index (ICSI) as a category-weighted fairness metric. The larger model shows an improvement of the aggregate bias score that is statistically significant (Mann-Whitney p < 0.001) a biased-response rate that is significantly lower (χ² = 51.92, p < 0.001, 94.04% unbiased responses) at approximately double the inference latency. The breakdown of results by categories also shows that this overall gain is unevenly split: the 8B remains more sensitive on Gender-based prompts even as it improves on Religion, Profession and Region, underscoring the merit of reporting disaggregated, category-imbued fairness over a single number bias score for India deployed LLMs [9], [17], [23]. The future work will allow the scaling of the benchmark to be done to additional model sizes within the LLaMA family (for instance 3B, 70B) to instead fit a bias-versus-scale curve rather than a two-point comparison (14). Then, the extension of the category set to ‘caste-adjacent’ and intersectional categories (for instance gender × region) that are under-represented in this effort (18), (9). The next direction will be replacing the lexicon-based bias scorer with the LLM-as-judge scorer validated against human annotation, to measure scorer-induced bias in the evaluation pipeline itself (28). Finally, deploying the mitigation variants (few-shot, prompt-engineering) evaluated here as live runtime guardrails and measuring their effect on the ICSI in a closed deployment loop (29), (30).

[1]      S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, “Language (technology) is power: A critical survey of ‘bias’ in NLP,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2020, pp. 5454–5476. [2]      I. O. Gallegos et al., “Bias and fairness in large language models: A survey,” Computational Linguistics, vol. 50, no. 3, pp. 1097–1179, 2024. [3]       K. Khandelwal, M. Tonneau, A. M. Bean, H. R. Kirk, and S. A. Hale, “Casteist but not racist? Quantifying disparities in large language model bias between India and the West,” arXiv preprint arXiv:2309.08573, 2023. [4]       A. Ahmad and P. Bhattacharyya, “Bias in language models: A survey,” Tech. Rep., IIT Bombay CFILT, 2024. [5]      A. Urlana, C. V. Kumar, B. M. Garlapati, A. K. Singh, and R. Mishra, “No size fits all: The perils and pitfalls of leveraging LLMs vary with company size,” in Proc. 31st Int. Conf. Comput. Linguistics (COLING), Industry Track, 2025, pp. 187–203. [6]      N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman, “CrowS-Pairs: A challenge dataset for measuring social biases in masked language models,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2020, pp. 1953–1967. [7]       M. Nadeem, A. Bethke, and S. Reddy, “StereoSet: Measuring stereotypical bias in pretrained language models,” in Proc. 59th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2021, pp. 5356–5371. [8]       K. Khandelwal, M. Tonneau, A. M. Bean, H. R. Kirk, and S. A. Hale, “Indian-BhED: A dataset for measuring India-centric biases in large language models,” in Proc. Int. Conf. Inf. Technol. Social Good (GoodIT), 2024, pp. 231–239. [9]       N. Sahoo, et al., “IndiBias: A benchmark dataset to measure social biases in language models for Indian context,” in Proc. Conf. North American Chapter Assoc. Comput. Linguistics (NAACL), 2024, pp. 8786–8806. [10]     Meta AI, “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.