Parallel and Distributed Computing

Course Syllabus

Name of the Course: Parallel and Distributed Computing
LTP structure of the course: 2-1-1
Objective of the course: To introduce various Parallel and Distributed hardware architectures and programming models.
Outcome of the course: Upon completion, students will be able to
- Understand the basics of various parallel and distributed computing platforms
- Identify the models and frameworks best suited to various workloads.
- Provide solutions to parallel and distributed computing problems.
Course Plan:

Unit	Topics for Coverage
Unit 1	Introduction to PDC: Latency vs. Bandwidth, Applications and Challenges, Types of architecture, Flynn’s taxonomy, Basic concepts: cores, nodes, threads, processes, speedup, efficiency, overhead, strong and weak scaling (Amdahl’s law, Gustafson’s law), Cache, Principle of Locality, Programming Models.
Unit 2	Distributed Computing: Distributed Memory, Message Passing Interface, Asynchronous/Synchronous computation/communication, concurrency control, fault tolerance, Distributed Programming with OpenMPI.
Unit 3	Parallel Computing: shared memory, data and task parallelism, Synchronization, Concurrent Data Structures, Shared Memory Programming with available APIs: PThreads, OpenMP, TBB.r
Unit 4	GPU Programming: GPU Architecture, Programming Models: CUDA/OpenCL, Basic Concepts: Threads, Blocks, Grids, GPU memory hierarchy, Thread Scheduling, Warps and Control divergence, Memory Coalescing, Programming with CUDA, Using CUDA Libraries: CuBLAS, CuFFT.

Text Book:
- The Art of Multiprocessor Programming by Maurice Herlihy and NirShavit, Morgan Kaufmann Publishers.
- Principles of Parallel Programming, by Calvin Lin, Larry Snyder, Addison-Wesley.
- Introduction to Parallel Computing, by AnanthGrama, Anshul Gupta, George Karypis, Vipin Kumar, Second Edition.
- Wen-Mei W Hwu, David B Kirk, Programming Massively Parallel Processors A Hands-on Approach, Morgann Kaufmann, 3e.