COMMUNICATION-AWARE SCHEDULING FOR DISTRIBUTED LLM INFERENCE ON HETEROGENEOUS EDGE AND CONSUMER DEVICES: A SURVEY

Distributed LLM inference can pool memory and compute across devices, but introduces communication, scheduling, and reliability costs. Existing systems explore collaborative hosting, heterogeneous parallelism, phase disaggregation, communication scheduling, and memory offloading; their results cannot be ranked directly across different evaluations. This survey proposes a lightweight layer-wise execution-plan selector based on compute time, communication cost, availability, and reconfiguration overhead. Implementation and controlled experiments are required to validate its cost model and determine when adaptation improves performance.

[1] A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y. Belkada, A. Chumachenko, P. Samygin, and C. Raffel, ‒Petals: Collaborative Inference and Fine-tuning of Large Models,‒ in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Volume 3: System Demonstrations, pp. 558–568, 2023. doi: 10.18653/v1/2023.acl-demo.54. Available: https://aclanthology.org/2023.acl-demo.54/ [2] A. Borzunov, M. Ryabinin, A. Chumachenko, D. Baranchuk, T. Dettmers, Y. Belkada, P. Samygin, and C. Raffel, “Distributed Inference and Fine-tuning of Large Language Models Over The Internet,” Advances in Neural Information Processing Systems 36, 2023. doi: 10.52202/075280-0540. Available: https://papers.nips.cc/paper_files/paper/2023/hash/28bf1419b9a1f908c15f6195f58cb865-Abstract-Conference.html [3] Y. Jiang, R. Yan, X. Yao, Y. Zhou, B. Chen, and B. Yuan, ‒HexGen: Generative Inference of Large Language Model over Heterogeneous Environment,‒ in Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 21946–21961, 2024. Available: https://proceedings.mlr.press/v235/jiang24f.html [4] Y. Jiang, R. Yan, and B. Yuan, ‒HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment,‒ in International Conference on Learning Representations, 2025. Available: https://proceedings.iclr.cc/paper_files/paper/2025/hash/0b941a1e5fbce23fe46b049999d04ed0-Abstract-Conference.html [5] A. Habibi and E. Erçetin, ‒Edge-LLM Inference With Cost-Aware Layer Allocation and Adaptive Scheduling,‒ IEEE Access, 2025. doi: 10.1109/ACCESS.2025.3592308. Available: https://doi.org/10.1109/ACCESS.2025.3592308 [6] L. Jin, K. Zhang, Y. Chen, Y. Zhuo, R. Li, Y. Gao, B. Yang, Z. Cai, and W. Dong, ‒Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication,‒ in Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 17633–17642, 2025. doi: 10.18653/v1/2025.findings-emnlp.957. Available: https://aclanthology.org/2025.findings-emnlp.957/ [7] Y. Zhong et al., ‒DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,‒ in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin [8] A. Patel et al., ‒Splitwise: Efficient Generative LLM Inference Using Phase Splitting,‒ in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024. doi: 10.1109/ISCA59077.2024.00019. Available: https://doi.org/10.1109/ISCA59077.2024.00019 [9] W. Sheng et al., ‒FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU,‒ in Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023. Available: https://proceedings.mlr.press/v202/sheng23a.html