Zhenheng Tang

Zhenheng Tang

PostDoc Researcher in Computer Science, The Hong Kong University of Science and Technology

Full News List

  • [2025.01]  🎉🎉 Our paper “Hot-pluggable Federated Learning: Briding General and Personalized FL via Dynamic Selection” is selected at ICLR 2025. This paper proposes a selective federated learning approach to integate personalized modules into general federated learning. (Paper)
  • [2025.01]  🎉🎉 Our paper “STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs” is selected at ICLR 2025. This paper introduces STBLLM, a novel approach that breaks the 1-bit barrier in language models by leveraging Structured Binary LLMs. (Paper)
  • [2025.01]  🎉🎉 Our paper “The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?” is selected as ICLR Blogpost 2025. This blog proposes a lottery LLM hypothesis suggesting that for a given LLM and task, there exists a smaller lottery LLM capable of producing the same performance with the original LLM with the assistances of multi-step reasoning and external tools. (Paper)
  • [2025.01]  🎉🎉 Our paper “Can LLM Simulations Truly Reflect Humanity? A Deep Dive.” is selected as ICLR Blogpost 2025. This blog rethinks LLM-based simulations by emphasizing both their limitations and the necessities for advancing LLM simulations. It offer actionable insights and strategies for enhancing the applicability of LLM simulations in human society in the future. (Paper)
  • [2025.01]  🎉🎉 Our paper “ParZC: Parametric Zero-Cost Proxies for Efficient NAS. “ is selected at AAAI 2025 for Oral Presentation!
  • [2024.12]  🎉🎉 Our paper “ParZC: Parametric Zero-Cost Proxies for Efficient NAS. “ is accepted at AAAI 2025! Parametric Zero-Cost Proxies (ParZC) method improves zero-shot Neural Architecture Search by addressing unequal node importance and using novel techniques for uncertainty estimation and architecture ranking. (paper and codes will come soon…)
  • [2024.11]  🎉🎉 I’m selected as the Top Reviewer of NeurIPS 2024 for both main and D&B tracks (Link).
  • [2024.10]  🎉🎉 Our paper “Hot Pluggable Federated Learning.” has been selected by the FL@FM-NeurIPS’24 workshop to receive the Outstanding Student Paper Award!. Congratulations to all co-authors!
  • [2024.10]  🎉🎉 Our paper “FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models.” is accepted In ASPLOS 2025! In this paper, we design and implement a new training system modularizes various operators in the entire MoE model, providing more fine-grained computation and communication scheduling, and achieving better computation communication overlap through appropriate gradient segmentation.. (paper and codes will come soon…)
  • [2024.09]  🎉🎉 Our paper “Hot Pluggable Federated Learning.” is accepted at Workshop Federated Foundation Models@NeurIPS 2024 as an Oral paper!. In this paper, we propose a new method to regard model heads as pluggable modules appended after the model backbone. (paper and codes will come soon…)
  • [2024.09]  🎉🎉 Our paper “FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion.” is accepted at NeurIPS 2024 as a Spotlight paper ! This work identifies the cause of low performance of one-shot FL, and proposes FuseFL to progressively train and fuses DNN model following a bottom-up manner, reducing communication costs to an extremely low degree. (paper and codes will come soon…)
  • [2024.09]  🎉🎉 Our paper “Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.” is accepted at NeurIPS 2024</ strong>. In this paper, we present a new method for optimizing layerwise sparsity allocation in large language models. (paper and codes will come soon…)
  • [2024.09]  🎉🎉 Our paper “Should We Really Edit Language Models? On the Evaluation of Edited Language Models.” is accepted at NeurIPS 2024. In this paper, we benchmark the methods of editing LLMs and see how they influence LLM performance. (paper and codes will come soon…)
  • [2024.09]  🎉🎉 Our paper “LPZero: Language Model Zero-cost Proxy Search from Zero.” is accepted at EMNLP 2024 (Findings). In this paper, we propose LPZero, which can automatically design zero-cost proxies for NLP tasks. It uses genetic programming to find optimal symbolic equations, outperforming human-designed proxies in ranking consistency. (paper and codes will come soon…)
  • [2024.06]  🎉🎉 Our paper “Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning.” is accepted at ICPP 2024. This work finds that the overlapness between indexes of compressed client model parameters demonstrates important information that can be utilized to adjust averging weights. (paper)
  • [2024.05]  🎉🎉 Our paper “Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models.” is accepted at ICML 2024. This work finds new pruning metric to prune LLMs to achieve SOTA performance under the same sparsity ratio. (paper)
  • [2024.03]  🎉🎉 VMRNN is available. This work proposes the VMRNN cell, a new recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM. We construct a network centered on VMRNN cells to tackle spatiotemporal prediction tasks effectively. (paper, code)
  • [2024.01]  🎉🎉 Our paper “FedImpro: Measuring and Improving Client Update in Federated Learning” is accepted at ICLR 2024. (paper,code). This work trains the high-level neural network on reconstructed feature distributions, to accelerate FL training and enhance the model performance.
  • [2024.01]  🎉🎉 Our paper “Towards Efficient and Reliable LLM Serving: A Real-World Workload Study” is available. (paper), code. This paper introduces the first real-world trace dataset of LLM serving workloads, detailing user, system, and LLM behaviors. Many new insights of GPT services are found in this work.
  • [2023.10]  🎉🎉 Our paper “Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining” is available. (paper). This work desings an in-memory fault-tolerance framework for large-scale LLM pretraining.
  • [2023.09]  🎉🎉 Our paper “FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs” is available. (paper). This paper envisions a decentralized system unlocking the potential vast untapped consumer-level GPUs in pre-training, inference and fine-tuning of LLMs with privacy protection.
  • [2023.02]  🎉🎉 Our paper “FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training” is available. (paper). This work aims to democratize simulating large-scale FL experiments. BTW, our open-source FL framework FedML has reached 2.6k stars at github.
  • [2022.12]  🎉🎉 Our paper “GossipFL: A Decentralized Federated Learning Framework with Sparsified and Adaptive Communication” has been accepted by IEEE TPDS. (paper).
  • [2022.10]  🎉🎉 Our paper “NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension” is accepted at AAAI 2023</ strong>. (paper).
  • [2022.05]  🎉🎉 Our paper “Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning” is accepted at ICML 2022. (paper, [code] (https://github.com/wizard1203/VHL)).
  • [2022.03]  🎉🎉 Our paper “FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks” is accepted to FL-AAAI workshop '22 as Oral Presentation. paper.
  • [2021.07]  🎉🎉 Our paper “Data Resampling for Federated Learning with Non-IID Labels.” is accepted to FTL-IJCAI workshop ‘21 paper.
  • [2020.10]  🎉🎉 Our survey paper “A Quantitative Survey of Communication Optimizations in Distributed Deep Learning” [code, paper] has been accepted by IEEE Network Magazine.