Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Contribution List

1. Welcome and Introduction

11/6/23, 9:00 AM

• Meet the instructor.
• Create an account at courses.nvidia.com/join

2. Stochastic Gradient Descent and the Effects of Batch Size

11/6/23, 9:15 AM

Learn the significance of stochastic gradient descent when training on multiple GPUs

• Understand the issues with sequential single-thread data processing and the theory behind speeding up applications with parallel processing.
• Understand loss function, gradient descent, and stochastic gradient descent (SGD).
• Understand the effect of batch size on accuracy and training time with an eye...

3. Training on Multiple GPUs with PyTorch Distributed Data Parallel (DDP)

11/6/23, 12:15 PM

Learn to convert single GPU training to multiple GPUs using PyTorch Distributed Data Parallel

• Understand how DDP coordinates training among multiple GPUs.
• Refactor single-GPU training programs to run on multiple GPUs with DDP.

4. Maintaining Model Accuracy when Scaling to Multiple GPUs

11/6/23, 2:30 PM

Understand and apply key algorithmic considerations to retain accuracy when training on multiple GPUs

• Understand what might cause accuracy to decrease when parallelizing training on multiple GPUs.

• Learn and understand techniques for maintaining accuracy when scaling training to multiple GPUs.

5. Workshop Assessment

11/6/23, 4:00 PM

Use what you have learned during the workshop: complete the workshop assessment to earn a certificate of competency

6. Final Review

11/6/23, 4:30 PM

• Review key learnings and wrap up questions.
• Take the workshop survey.