SC23 Tutorial: Efficient Distributed GPU Programming for Exascale

Repository with talks and exercises of our Efficient GPU Programming for Exascale tutorial, to be held at SC23.

Coordinates

• Date: 13 November 2023
• Occasion: SC23 Tutorial
• Tutors: Simon Garcia (SNL), Andreas Herten (JSC), Markus Hrywniak (NVIDIA), Jiri Kraus (NVIDIA), Lena Oden (Uni Hagen)

Setup

The tutorial is an interactive tutorial with introducing lectures and practical exercises to apply knowledge. The exercises have been derived from the Jacobi solver implementations available in NVIDIA/multi-gpu-programming-models.

Curriculum:

1. Lecture: Tutorial Overview, Introduction to System + Onboarding Andreas
2. Lecture: MPI-Distributed Computing with GPUs Simon
3. Hands-on: Multi-GPU Parallelization
4. Lecture: Performance / Debugging Tools Markus
5. Lecture: Optimization Techniques for Multi-GPU Applications Simon
6. Hands-on: Overlap Communication and Computation with MPI
7. Lecture: Overview of NCCL and NVSHMEN in MPI Jiri
8. Hands-on: Using NCCL and NVSHMEM
9. Lecture: Device-initiated Communication with NVSHMEM Jiri
10. Hands-on: Using Device-Initiated Communication with NVSHMEM
11. Lecture: Conclusion and Outline of Advanced Topics Andreas