Feedback Deep Neural Networks for Enhanced Multi-User Detection in MIMO-NOMA Systems

non-orthogonal multiple access (NOMA) integrated with multiple-input multiple-output (MIMO) technology is pivotal for enhancing spectral efficiency in fifth-generation (5G) and sixth-generation (6G) wireless networks. Conventional successive interference cancellation (SIC) receivers, however, are plagued by issues such as near-far effects and error propagation, resulting in high bit error rates (BER), particularly for users with minimal signal-to-noise ratio (SNR) disparities. This paper introduces a feedback deep neural network (FDNN) receiver that employs deep learning for nonlinear signal detection and incorporates iter- Through extensive simulations, the FDNN demonstrates a substantial BER reduction for the weaker user, from 9.8×10−2 to 2.4×10−2at 20 dB SNR, surpassing zero-forcing (ZF)-SIC baselines. The study extends evaluations with ablation studies on network architectures, batch sizes, increased user counts (up to K=3), modulation schemes (QPSK and 16-QAM), and performance under imperfect channel state information (CSI). Additional analyses include computational complexity, latency metrics, and robustness in dynamic channel environments. The FDNN’s scalable and interference-resilient design holds signif- icant promise for next-generation wireless systems, addressing key challenges in real-time deployment.