Skeleton-Based Real-Time Gesture Recognition: A Review of Methods, Datasets, and Deployment Challenges

Skeleton-based gesture recognition has matured from recurrent networks over flat joint vectors to graph, convolutional and attention-based models that exceed 93% on NTU RGB+D 60 under the cross-subject protocol and approach 95% on SHREC’17, while self-supervised pre-training now produces encoders that rival supervised training once fine-tuned. Pose estimators that feed these models run on commodity hardware and deliver body and hand landmarks from a single RGB camera. The remaining distance to deployed interaction lies less in benchmark accuracy than in conditions benchmarks omit: noisy keypoints, mixed hand-and-body vocabularies, unseen users, honest end-to-end latency, continuous input and privacy. The consolidated results also show that leaderboard gains increasingly come from ensembling and from better input skele-tons. Lightweight single-stream models, explicit handling of missing joints, pre-training for new vocabularies, and evaluation protocols that measure robustness and runtime alongside accuracy are, in our assessment, the most productive directions for real-time gesture interfaces.

[1]                 Mitra, S., Acharya, T.: Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C 37(3), 311–324 (2007) [2]                 Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: A survey. Artificial Intelligence Review 43(1), 1–54 (2015) [3]                 Ren, B., Liu, M., Ding, R., Liu, H.: A survey on 3D skeleton-based action recognition using learning method. Cyborg and Bionic Systems 5, 0100 (2024) https://doi.org/10.34133/cbsystems.0100 [4]                 Meng, X., Li, S., Yuan, M., Wang, W., Zhang, Z., Song, C., Cao, F.: Action recognition based on human skeleton: Review and prospect. Information and Control 54(1), 1–27 (2025) [5]                 Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., Liu, J.: Human action recognition from various data modalities: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(3), 3200–3225 (2023) [6]                 Foteinos, K., Cani, J., Linardakis, M., Radoglou-Grammatikis, P., Argyriou, V., Sarigiannidis, P., Varlamis, I., Papadopoulos, G.T.: Visual hand gesture recogni-tion with deep learning: A comprehensive review of methods, datasets, challenges and future research directions. arXiv preprint arXiv:2507.04465 (2025) [7]                 Shahroudy, A., Liu, J., Ng, T.-T., Wang, G.: NTU RGB+D: A large scale dataset for 3D human activity analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1010–1019 (2016) [8]                 Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.-Y., Kot, A.C.: NTU RGB+D 120: A large-scale benchmark for 3D human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(10), 2684–2701 (2020) [9]                 Wang, H., Schmid, C.: Action recognition with improved trajectories. In: IEEE International Conference on Computer Vision (ICCV), pp. 3551–3558 (2013) [10]               Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 25, pp. 1097–1105 (2012)