I’m a Post-doctoral Researcher at The Hong Kong University of Science and Technology under supervision of Prof. Bo Li and Prof. Xiaowen Chu. Previously, I obtained my PhD degree of Computer Science from Hong Kong Baptist University under the supervision of Prof. Xiaowen Chu and Prof. Amelie Chi Zhou, and co-supervised by Prof. Bo Han. Before that, I got my Bachelor degree from HUST.

Research Interests I am interested in undersanding how deep learning models are optimized and how they learn knowledge and do reasoning. In next at least one year, I will delve into studying LLM reasoning mechanisms, agent workflow, fast finetuning and inference, and some practical LLMs’ applications in CS or other areas in which knowledge can be conveniently represented digitally.

I’m open for academic collaborations. If you are interested, please feel free to contact me.

🔥 News

[2024.11] 🎉🎉 I’m selected as the Top Reviewer of NeurIPS 2024 for both main and D&B tracks (Link).
[2024.10] 🎉🎉 Our paper “Hot Pluggable Federated Learning.” has been selected by the FL@FM-NeurIPS’24 workshop to receive the Outstanding Student Paper Award!. Congratulations to all co-authors!
[2024.10] 🎉🎉 Our paper “FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models.” is accepted In ASPLOS 2025! In this paper, we design and implement a new training system modularizes various operators in the entire MoE model, providing more fine-grained computation and communication scheduling, and achieving better computation communication overlap through appropriate gradient segmentation.. (paper and codes will come soon…)
[2024.09] 🎉🎉 Our paper “Hot Pluggable Federated Learning.” is accepted at Workshop Federated Foundation Models@NeurIPS 2024 as an Oral paper!. In this paper, we propose a new method to regard model heads as pluggable modules appended after the model backbone. (paper and codes will come soon…)
[2024.09] 🎉🎉 Our paper “FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion.” is accepted at NeurIPS 2024 as a Spotlight paper ! This work identifies the cause of low performance of one-shot FL, and proposes FuseFL to progressively train and fuses DNN model following a bottom-up manner, reducing communication costs to an extremely low degree. (paper and codes will come soon…)
[2024.09] 🎉🎉 Our paper “Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.” is accepted at NeurIPS 2024. In this paper, we present a new method for optimizing layerwise sparsity allocation in large language models. (paper and codes will come soon…)
[2024.09] 🎉🎉 Our paper “Should We Really Edit Language Models? On the Evaluation of Edited Language Models.” is accepted at NeurIPS 2024. In this paper, we benchmark the methods of editing LLMs and see how they influence LLM performance. (paper and codes will come soon…)
[2024.09] 🎉🎉 Our paper “LPZero: Language Model Zero-cost Proxy Search from Zero.” is accepted at EMNLP 2024 (Findings). In this paper, we propose LPZero, which can automatically design zero-cost proxies for NLP tasks. It uses genetic programming to find optimal symbolic equations, outperforming human-designed proxies in ranking consistency. (paper and codes will come soon…)
[2024.06] 🎉🎉 Our paper “Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning.” is accepted at ICPP 2024. This work finds that the overlapness between indexes of compressed client model parameters demonstrates important information that can be utilized to adjust averging weights. (paper)
[2024.05] 🎉🎉 Our paper “Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models.” is accepted at ICML 2024. This work finds new pruning metric to prune LLMs to achieve SOTA performance under the same sparsity ratio. (paper)
[2024.03] 🎉🎉 VMRNN is available. This work proposes the VMRNN cell, a new recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM. We construct a network centered on VMRNN cells to tackle spatiotemporal prediction tasks effectively. (paper, code)
[2024.01] 🎉🎉 Our paper “FedImpro: Measuring and Improving Client Update in Federated Learning” is accepted at ICLR 2024. (paper,code). This work trains the high-level neural network on reconstructed feature distributions, to accelerate FL training and enhance the model performance.
[2024.01] 🎉🎉 Our paper “Towards Efficient and Reliable LLM Serving: A Real-World Workload Study” is available. (paper),code. This paper introduces the first real-world trace dataset of LLM serving workloads, detailing user, system, and LLM behaviors. Many new insights of GPT services are found in this work.
[2023.10] 🎉🎉 Our paper “Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining” is available. (paper). This work desings an in-memory fault-tolerance framework for large-scale LLM pretraining.
[2023.09] 🎉🎉 Our paper “FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs” is available. (paper). This paper envisions a decentralized system unlocking the potential vast untapped consumer-level GPUs in pre-training, inference and fine-tuning of LLMs with privacy protection.
[2023.02] 🎉🎉 Our paper “FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training” is available. (paper). This work aims to democratize simulating large-scale FL experiments. BTW, our open-source FL framework FedML has reached 2.6k stars at github.
[2022.12] 🎉🎉 Our paper “GossipFL: A Decentralized Federated Learning Framework with Sparsified and Adaptive Communication” has been accepted by IEEE TPDS. (paper).
[2022.10] 🎉🎉 Our paper “NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension” is accepted at AAAI 2023. (paper).
[2022.05] 🎉🎉 Our paper “Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning” is accepted at ICML 2022. (paper, code).
[2022.03] 🎉🎉 Our paper “FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks” is accepted to FL-AAAI workshop '22 as Oral Presentation. paper.
[2021.07] 🎉🎉 Our paper “Data Resampling for Federated Learning with Non-IID Labels.” is accepted to FTL-IJCAI workshop ‘21 paper.
[2020.10] 🎉🎉 Our survey paper “A Quantitative Survey of Communication Optimizations in Distributed Deep Learning” [code, paper] has been accepted by IEEE Network Magazine.

📖 Educations

2020.09 - 2024.08, Hong Kong Baptist University, PhD in Computer Science
2014.09 - 2018.06, Huazhong University of Science and Technology, Bachelor in Telecommunications Engineering。

💻 Work & Research Experience

09/2024-present: PostDoc Researcher, The Hong Kong University of Science and Technology, advised by Prof. Bo Li and Prof. Xiaowen Chu.
09/2023-08/2024: Visiting Researcher, The Hong Kong University of Science and Technology (Guangzhou), advised by Prof. Xiaowen Chu.
02/2023-05/2023: Visiting Researcher, National University of Singapore, advised by Prof. Bingsheng He.
06/2022-10/2022: Research Intern, FedML Inc, advised by Dr. Chaoyang He.
10/2018-09/2020: Research Assistant, Hong Kong Baptist University, advised by Prof. Xiaowen Chu.

📕 Teaching

Teaching Assistant at HKBU
- 2023 Spring Semester, COMP7940 Cloud Computing
- 2022 Fall Semester, COMP7015 Artiﬁcial Intelligence
- 2022 Spring Semester, COMP 7550 IT Project Management
- 2021 Fall Semester, COMP 7015, Artificial Intelligence
- 2021 Spring Semester, COMP 7930, Big Data Analytics

👔 Professional Activities

Invited Program Committee Member (Reviewer):
- Machine Learning: KDD’23, ICML’22,23,24, NeurIPS’22,23,24, ICLR’23,24,25, AAAI’23,25, AISTATS’23’25, UAI’22, IJCAI-ECAI’22
- Networking & Systems: HPCC’21, ICDCS’22,23, ICPADS’22, IWQOS’23,24
Invited Reviewer for Journals
- Machine Learning:
  - IEEE Transactions on Pattern Analysis and Machine Intelligence
  - Transactions on Machine Learning Research
  - IEEE Transactions on Neural Networks and Learning Systems
  - Journal of Artificial Intelligence Research
- Networking & Systems:
  - IEEE Transactions on Parallel and Distributed Systems
  - IEEE Journal on Selected Areas in Communications
  - IEEE Network Magzines
  - IEEE Transactions on Networking
  - IEEE Transactions on Network Science and Engineering
  - IEEE Transactions on Intelligent Systems and Technology
  - IEEE Computational Intelligence Magazine
  - Journal of Parallel and Distributed Computing
- General:
  - ACM Computing Surveys

🎖 Honors and Awards

2024, NeurIPS Top Reviewer (Link).
2024, Outstanding Student Paper Award of FL@FM-NeurIPS’24 workshop.
2024, NeurIPS Scholar Award.
2024, ICLR Scholar Award.
2023/24, Research Performance Award, HKBU CS Department (Link).
2022/23, Research Performance Award, HKBU CS Department (Link).
2022/23 Fall, Teaching Performance Award, HKBU CS Department (Link).
2021/22, Research Performance Award, HKBU CS Department (Link).
2021/22 Fall, Teaching Performance Award, HKBU CS Department (Link).
2020, Scholarship for Nominees of Hong Kong PhD Fellowship Scheme, HKBU CS Department (Link).
2018, Outstanding Graduate, HUST
2016, Scholarship of Academic Excellence, HUST

📝 Publications

The * represents the equal contribution, 📧 corresponding author, 💡 project lead.

X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, X. Chu. FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models. In ASPLOS 2025.
Z. Tang, Y. Zhang, P. Dong, Y. Cheung, A. C. Zhou, B. Han, X. Chu. FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion.” In NeurIPS 2024 (spotlight).
L. Shen*, Z. Tang*💡 , L. Wu, Y. Zhang, X. Chu, T. Qin, B. Han. “Hot Pluggable Federated Learning.” In Workshop of Federated Foundation Models@NeurIPS 2024 (Oral, Outstanding Student Paper Award).
L. Li, P. Dong, Z. Tang, X. Liu, Q. Wang, W. Luo, W. Xue, Q. Liu, X. Chu, Y. Guo. Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models. In NeurIPS 2024.
Q. Li, X. Liu, Z. Tang, P. Dong, Z. Li, X. Pan, X. Chu. Should We Really Edit Language Models? On the Evaluation of Edited Language Models. In NeurIPS 2024.
P. Dong, L. Li, X. Liu, Z. Tang, X. Liu, Q. Wang, X. Chu. LPZero: Language Model Zero-cost Proxy Search from Zero. In EMNLP 2024 Findings.
Z. Tang, J. Huang, R. Yan, Y. Wang, Z. Tang💡📧, S. Shi, A. C. Zhou, X. Chu📧. Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning. In ICPP 2024.
P. Dong, L. Li, Z. Tang, X. Liu, X. Pan, Q. Wang, X. Chu📧. Evolving Symbolic Pruning Metric From Scratch for Large Language Models. In ICML 2024.
Y. Tang, P. Dong, Z. Tang, X. Chu, J. Liang📧. VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting. In CVPR Workshop 2024.
Z. Tang, Y. Zhang, S. Shi, X. Tian, T. Liu, B. Han, X.📧 Chu. FedImpro: Measuring and Improving Client Update in Federated Learning. In ICLR 2024.
Y. Wang, Y. Chen, Z. Li, Z. Tang, R. Guo, X. Wang, Q. Wang, AC Zhou, X. Chu📧. Towards Efficient and Reliable LLM Serving: A Real-World Workload Study. arXiv preprint arXiv:2401.17644.
Y. Wang, S. Shi, X. He, Z. Tang, X. Pan, Y. Zheng, X. Wu, AC Zhou, B. He, X. Chu📧. Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining. arXiv preprint arXiv:2310.12670.
Z. Tang, Y. Wang, X. He, L. Zhang, X. Pan, Q. Wang, R. Zeng, K. Zhao, S. Shi📧, B. He, X. Chu📧. FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs. In IJCAI-LLM Workshop 2023.09.,
Z. Tang, S. Shi, B. Li, X. Chu📧. GossipFL: A Decentralized Federated Learning Framework with Sparsified and Adaptive Communication. In IEEE Transactions on Parallel and Distributed Systems, 2022.
X. He , J. Yao, Y. Wang, Z. Tang, C. K. Chun, S. Simon, B. Han, and X. Chu📧. NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension. In AAAI 2023.
Z. Tang, Y. Zhang*, S. Shi, X. He, B. Han, X. Chu📧. Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. In Proceedings of the 39th International Conference on Machine Learning, 2022.
C. He, A. D. Shah, Zhenheng Tang, D. Fan, A. N. Sivashunmugam, K. Bhogaraju, M. Shimpi, L. Shen, X. Chu, M. Soltanolkotabi and S. Avestimehr. FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks. In FL-AAAI-22 workshop, 2022. [Best Paper Award]
Z. Liao, H. Yan, Z. Tang, X. Chu, T. Tao📧. Deep learning identifies leak in water pipeline system using transient frequency response. In Process Safety and Environmental Protection 2021.
Z. Tang, Zhikai Hu, Shaohuai Shi, Yiu-ming Cheung, Yilun Jin, Zhenghang Ren, Xiaowen Chu📧. Data Resampling for Federated Learning with Non-IID Labels. In FTL-IJCAI workshop, 2021.
S. Shi, Z. Tang, X. Chu, C. Liu, W. Wang, and B. Li. A quantitative surveyof communication optimizations in distributed deep learning. IEEE Network,35(3):230–237, 2021.
S. Shi, Z. Tang, Q. Wang, K. Zhao, and X. Chu. Layer-wise adaptive gradientsparsification for distributed deep learning with convergence guarantees. In ECAI 2020 * 24th European Conference on Artificial Intelligence, pages 1467–1474. IOS Press, 2020.
Z. Tang, S. Shi, and X. Chu📧. Communication-efficient decentralized learning withsparsification and adaptive peer selection. In ICDCS 2020.
Y. Wang, Q. Wang, S. Shi, X. He, Z. Tang, K. Zhao, X. Chu. Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training. In CCGRID 2020.
Z. Tang, Y. Wang, Q. Wang, and X. Chu. The impact of gpu dvfs on the energy andperformance of deep learning: An empirical study. In Proceedings of the Tenth ACM International Conference on Future Energy Systems, e-Energy ’19.
S. Shi, K. Zhao, Q. Wang, Z. Tang, and X. Chu. A convergence analysis ofdistributed sgd with communication-efficient gradient sparsification. IJCAI-19.
S. Shi, Q. Wang, K. Zhao, Z. Tang, Y. Wang, X. Huang, and X. Chu. A distributedsynchronous sgd algorithm with global top-k sparsification for low bandwidthnetworks. In ICDCS 2019.
X. Zhou, Z. Tang, W. Xu, F. Meng, X. Chu, K. Xin, and G. Fu📧. Deep learningidentifies accurate burst locations in water distribution networks. Water Research,166:115058, 2019.
X. He, S. Wang, S. Shi, Z. Tang, Y. Wang, Z. Zhao, J. Dai, R. Ni, X. Zhang, X. Liu,Z. Wu, W. Yu, and X. Chu. Computer-aided clinical skin disease diagnosis usingcnn and object detection models. In 2019 IEEE International Conference on BigData (Big Data), pages 4839–4844, 2019.

Preprint

Z. Tang, X. Chu, R. Ran, S. Lee, S. Shi, Y. Zhang, Y. Wang, A. Liang, S. Avestimehr, C. He📧. FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training. arXiv preprint arXiv:2303.01778.
Z. Tang, S. Shi, W. Wang, B. Li and X. Chu📧. Communication-efficient distributeddeep learning: A comprehensive survey. CoRR, abs/2003.06307, 2020.

Zhenheng Tang