Enhancing MIMO-NOMA with Feedback-Driven Deep Learning: Performance, Scalability, and Robustness

Abstract

The integration of Non-Orthogonal Multiple Access (NOMA) with Multiple-Input Multiple-Output (MIMO) technology has emerged as a cornerstone for achieving the spectral efficiency requirements of fifth-generation (5G) and beyond wireless networks. However, conventional Successive Interference Cancellation (SIC) receivers employed in MIMO-NOMA systems suffer from critical limitations including error propagation, near-far effects, and inadequate handling of nonlinear interference, particularly when power differences between users are small. This paper proposes a Feedback Deep Neural Network (FDNN) receiver that leverages deep learning for nonlinear signal detection and incorporates iterative interference cancellation to address these challenges. The proposed architecture replaces traditional linear detection stages with deep neural networks that learn optimal decision boundaries directly from data, while the feedback mechanism enables progressive refinement of symbol estimates. Extensive simulations demonstrate that the FDNN receiver achieves 5× to 23× lower Bit Error Rate (BER) compared to conventional SIC across the SNR range of 10-30 dB, with particularly significant gains of 5.07× at low power differences (ΔSNR=3 dB). The proposed receiver maintains robust performance under imperfect Channel State Information (CSI), achieving BER below 10⁻² at CSI error variance of 0.15, compared to 4.8×10⁻² for SIC. Scalability analysis for K=3 and K=4 users show consistent 4-5× improvement across all users. Computational complexity analysis reveals that FDNN incurs 2.1× higher latency than SIC, representing an acceptable trade-off for the substantial performance gains. These results position the FDNN receiver as a promising candidate for next-generation wireless systems requiring reliable multi-user detection in challenging channel conditions.