Distributed training using a large number of devices has been widely adopted for learning large deep learning models. Improving distributed training efficiency is critical for time, resource and energy consumption of large model learning. In this talk, I will introduce recent research works in my group on optimizing distributed training parallelisms for effective training acceleration and maximal resource utilization. Especially, we have designed optimized strategies and systems for operator sharding, computation and communication scheduling for SPMD parallelism (e.g., in Mixture-of-Experts model training) in both homogeneous and heterogeneous AI clusters, as well as dynamic micro-batching and pipelining to tackle sequence length variation in multi-task model training (e.g., Large Language Model training).
Professor Chuan Wu received her B.Engr. and M.Engr. degrees in 2000 and 2002 from Tsinghua University and Ph.D. degree in 2008 from University of Toronto. Since September 2008, she has been with Department of Computer Science at the University of Hong Kong, and is currently a Professor. Her current main research is on distributed machine learning systems and algorithms. She has served as associate editor for ACM/IEEE Transactions on Networking, IEEE Transactions on Cloud Computing, etc. She has active collaborations with various AI cloud operators (AWS, ByteDance, Alibaba, etc.), and received an Amazon Research Award (ARA) on AWS AI in 2021.
Clean water and clean air are vital for public health. This project focuses on developing high-efficiency and environmentally sustainable filters for removing harmful air/water pollutants. The team has developed novel architectures and functionalities for the filters to achieve high permeance, high removal efficiency, and excellent reusability.
