TechTalk

In computer science, a graph is a network modeling objects and their unique interactions. The graph learning model is a specialized machine learning model that learns on graphs. Similar to traditional machine learning models, a well-performed graph learning model can capture the global data distribution with sufficient and unbiased training data. However, in a distributed subgraph system, most data owners only possess small amounts of the data (small subgraphs) in their local systems and can have unpredictable biases.
In this talk, the speaker will introduce this novel yet realistic setting – subgraph federated learning, which aims to let distributed data owners collaboratively train a powerful and generalized graph learning model without directly sharing their subgraphs. Towards this setting, two major techniques are proposed by the research team. (1) FedSage, which trains a GraphSage model based on FedAvg to integrate node features, link structures, and task labels on multiple local subgraphs; (2) FedSage+, which trains a missing neighbor generator along FedSage to deal with missing links across local subgraphs. Empirical results and theoretical analysis of proposed models respectively demonstrate the effectiveness and prove the generalization ability.

Distributed training using multiple devices (i.e., GPU servers) has been widely adopted for learning DNN models over large datasets. However, the performance of large-scale distributed training tends to be far from linear speed-up in practice. Given the complexity of distributed systems, it is challenging to identify the root cause(s) of inefficiency and exercise effective performance optimizations when unexpected low training speed occurs. To date, there exists no software tool which diagnoses performance issues and helps expedite distributed DNN training, while the training can be run using different machine learning frameworks. This paper proposes dPRO, a toolkit that includes: (1) an efficient profiler that collects runtime traces of distributed DNN training across multiple frameworks, especially fine-grained communication traces, and constructs global data flow graphs including detailed communication operations for accurate replay; (2) an optimizer that effectively identifies performance bottlenecks and explores optimization strategies (from computation, communication and memory aspects) for training acceleration. We implement dPRO on multiple deep learning frameworks (PyTorch, TensorFlow, MXNet) and representative communication schemes (AllReduce and Parameter Server architecture). Extensive experiments show that dPRO predicts performance of distributed training in various settings with<5% errors in most cases and finds optimization strategies with up to87.1%speed-up over the baselines.