Modeling Temporal Evolution of Sentiment in IMDb Reviews Using Transformer-Based Time Series Architectures

We have presented a Transformer-based time series approach for temporal sentiment evolution modeling in IMDb movie reviews. By evaluating the effect of positional encoding, multi-head self-attention, layer normalization, and attention weight extraction, we demonstrated that a compact 2-block encoder trained from scratch achieves 85.08% accuracy on the IMDb benchmark. The temporal self-attention heatmaps and entropy metrics provide novel interpretable evidence of how sentiment evolves across the token sequence, with negative reviews exhibiting mean attention entropy of 1.7580. The complete pipeline — from preprocessing through training to custom prediction — runs on a local CPU environment using TensorFlow 2.16.1 without requiring GPU infrastructure or pre-trained model downloads, demonstrating significant accessibility advantages over BERT-scale approaches.

[1] B. Pang and L. Lee, "Opinion Mining and Sentiment Analysis," Foundations and Trends in Information Retrieval, vol. 2, no. 1-2, pp. 1-135, 2008. [2] A. L. Maas et al., "Learning Word Vectors for Sentiment Analysis," in Proc. ACL, 2011, pp. 142-150. [IMDb Dataset] [3] B. Pang, L. Lee, and S. Vaithyanathan, "Thumbs Up? Sentiment Classification Using Machine Learning Techniques," in Proc. EMNLP, 2002, pp. 79-86. [4] S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997. [5] A. Vaswani et al., "Attention Is All You Need," in Proc. NeurIPS, 2017, pp. 5998-6008. [6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proc. NAACL-HLT, 2019, pp. 4171-4186. [7] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, "What Does BERT Look At? An Analysis of BERT's Attention," in Proc. BlackboxNLP, 2019. [8] S. Jain and B. C. Wallace, "Attention Is Not Explanation," in Proc. NAACL-HLT, 2019, pp. 3543-3556. [9] P. K. Diederik and B. Jimmy, "Adam: A Method for Stochastic Optimization," CoRR, vol. abs/1412.6980, 2015. [10] T. Mikolov et al., "Distributed Representations of Words and Phrases," in Proc. NeurIPS, 2013, pp. 3111-3119.