CS 475 / 575 -- Parallel Programming

On-campus Edition

Spring Quarter 2020

http://cs.oregonstate.edu/~mjb/cs575c


Handouts Projects Grades VHR Piazza


This page was last updated: March 22, 2020



Under Construction:
this content is not final until you see this message go away!


Announcements:


Here are Some Background Resources for You

Over the last couple of months, various faculty have been putting together mini-tutorials on topics that we (apparently) expect you to magically know, without us ever actually showing you. Here they are. We hope they help.


What We Will Be Doing This Quarter

The goals of this course are to leave you "career-ready" (i.e., both work-ready and research-ready) for tasks that require desktop parallelism, both on a CPU and on a GPU.

CS 475/575 topics include:

Prerequisites

This course will use C/C++ for most of its programming. Already being comfortable with function calls, arrays, for loops, structures, arrays of structures, structures of arrays, pointers, stacks, queues, trees, and linked lists is important. It is strongly suggested that you not use this class as an opportunity to learn programming for the first time.

Many of the assignments can be done on Linux systems. It would be good if you already know how to use the Linux command line and know at least one Linux-based editor (vim is good).

Some knowledge about computer architecture (e.g., cores, cache) would be a plus, but not critical.

Learning Outcomes

On completion of the course, students will have demonstrated the ability to:

  1. Explain the clockspeed limitations of computing using physics and Moore's Law
  2. Explain the limitations of parallel computing using Amdahl's Law
  3. Demonstrate "parallel thinking" in program design
  4. Demonstrate the ability to program parallel algorithms in multicore, SIMD, and GPU.
  5. Characterize different parallel programming patterns and what types of problems they best address
  6. Characterize how cache issues affect parallel performance
  7. Demonstrate the proper use of synchronization to avoid race conditions and deadlock
  8. Characterize SIMD performance speed-ups and what kinds of problems they are good for
  9. Characterize the benefits of using a CPU versus using a GPU for parallel programming
  10. Characterize the benefits of using a GPU versus using a CPU for parallel programming

In addition, those taking this course as CS 575 will also have demonstrated the ability to:

  1. Read a parallel-programming-related research paper and write a 5-page analysis paper of it. (I will make some of these available for you, or you can propose your own. It has to be a real research paper, though.)

Professor

The class is being taught by Professor Mike Bailey.

Office: Kelley 2117 (2nd floor, south side)
Phone: 541-737-2542
E-mail: mjb@cs.oregonstate.edu

Prof. Bailey Office Hours:

For the time being, my Office Hours will all be on Zoom.
Zoom-Room https://oregonstate.zoom.us/j/8340727662

Once I have chosen OHs based on your alias form, I will post those days and times here.

TAs

TBD.

Piazza

This term we will be using Piazza for class discussion. You can send emails directly to me or the TA, you can use the Virtual Hand Raise, or you can use Piazza.

Find our class piazza page at: https://piazza.com/oregonstate/spring2020/cs475575/home

The Virtual Hand Raise (VHR)

I recognize that it sometimes takes a certain amount of courage to ask a question in class. But, the worst thing of all is to not ask! So, this class also uses something called the Virtual Hand Raise. Click here to get into it. It will allow you to send me a question or comment, completely anonymously. I will answer questions submitted this way by email to the class or in class.

Textbook

There is no required textbook for this course. We will use notes that will be made available on the class web site.

Class Schedule

To see an Academic Year calendar, click here.

Week Topics Notesets Projects
1 Introduction. Syllabus. What this course is ... and isn't.
The things we care about Parallel Programming for.
Project notes: timing, graphing. (This is really important.)
Simple OpenMP.
Parallel programming background information.
Von Neumann architecture.
Course Introduction
Project Notes
Simple OpenMP
Parallel Programming Background Information
0
2 Multithreading.
Multicore.
OpenMP: fork-join model
Pragmas, what they do for you, what they don't do for you,
OpenMP: parallelizing for-loops
OpenMP: variable sharing
OpenMP: Reduction. Not doing anything special vs. critical vs. atomic vs. reduction.
Trapezoid integration.
Timing. Speedup. Amdahl's Law. Parallel efficiency. Gustafson's observation.
Moore's Law. What holds, what doesn't.
Hyperthreading.
OpenMP
Trapezoid Integration with OpenMP
Speedups and Amdahl's Law
Moore's Law and Multicore:
Hyperthreading and "Almost Amdahl"
1
3 OpenMP: dynamic vs. static thread assignment.
Chunksize.
Mutexes.
Barriers.
Sections, tasks.
OpenMP: tree traversal using tasks.
Caches. Architecture. Hits. Misses. False sharing.
Functional Decomposition
Caching Issues in Multicore Performance
Functional Decomposition
2
3
4 Data Decomposition
SIMD Vectorization: what it is, how to use it.
Prefetching
Data Decomposition
SIMD Vector Parallel Programming
Prefetching
4
5 Test #1 review
GPU 101.
Architecture.
GPUs -- what they are good at.
GPUs -- what they are not good at.
Why.
GPU 101  
6 Go over the test answers.
CUDA: general concepts, program setup.
CUDA: array*array multiplication
CUDA: matrix*matrix multiplication
CUDA
CUDA Array Multiplication
CUDA Matrix Multiplication
5
6
7 CUDA: matrix*matrix block multiplication
OpenCL: What is it? Diagram.
How OpenCL is like CUDA and how it is different
OpenCL library. Querying configurations.
OpenCL
first.cpp
first,cl
7
8 OpenCL Events
OpenCL Reduction.
Looking at OpenCL Assembly Language.
OpenCL / OpenGL Interoperability
OpenCL Events
OpenCL reduction
Looking At OpenCL Assembly language
8a
or
8b
9 OpenCL / OpenGL Interoperability OpenCL / OpenGL Interoperability  
10 The Message Passing Interface (MPI)
More information: books, periodicals, professional organizations, conferences.
Class Evaluations.
Test #2 review.
Message Passing Interface
More Information
Class Evaluations
 
T2 June ??
Test #2 ???, June ??, ?:?? - ?:?? ??.
(You can confirm this for yourself by going here.)
   
G Friday, June 12, 3:30
EECS Graduation Celebration: Gill Coliseum
   
G Saturday, June 13
OSU Graduation: assemble at the MU Quad
   

Grading

CS 475/575 will be graded on a fill-the-bucket basis. There will be 8 projects, 10 quizzes, and two tests. You get to keep all the points you earn.

In addition, the CS 575 people have an extra 100-point paper review project.
The CS 475 people automatically get 100 points for this project, even though they didn't have to do it.

Your final grade will be based on your overall class point total. Based on an available point total of 1100, grade cutoffs will be no higher than this.


Project Turn-In Procedures

Bonus Days and Late Assignments

Projects are due at 23:59:59 on the listed due date, with the following exception:

Each of you has been granted five Bonus Days, which are no-questions-asked one-day extensions which may be applied to any project, subject to the following rules:

  1. No more than 2 Bonus Days may be applied to any one project
  2. Bonus Days cannot be applied to tests
We will grade project assignments when they are two days past due so that we know how to apply the Bonus Days, like this:
  1. If your project's turn-in date is less-than-or-equal-to the due date, all is OK.
  2. If your project is 1 or 2 days late, but you have enough Bonus Days left to spend on it, then we will just grade it as normal, and adjust your Bonus Days Left (BDL) number on your grade sheet.
  3. If your project is 1 or 2 days late, and you don't have enough Bonus Days Left to spend on it, your grade is a 0.
  4. If your project is 3 or more days late, your grade is a 0, no matter how many Bonus Days you have left.

Downloadable Files

None yet.

Class Rules

Link to the Code of Student Conduct

https://beav.es/codeofconduct

Students With Disabilities

Accommodations for students with disabilities are determined and approved by Disability Access Services (DAS). If you, as a student, believe you are eligible for accommodations but have not obtained approval please contact DAS immediately at 541-737-4098 or at http://ds.oregonstate.edu.. DAS notifies students and faculty members of approved academic accommodations and coordinates implementation of those accommodations. While not required, students and faculty members are encouraged to discuss details of the implementation of individual accommodations.

Religious Holidays

Oregon State University strives to respect all religious practices. If you have religious holidays that are in conflict with any of the requirements of this class, please see me immediately so that we can make alternative arrangements.

Life Events

As {John Lennon? Allen Saunders?} has said: "Life is what happens to you while you're busy making other plans". We care about you as a person. When life happens to you, send me an email and come see me. I might be able to help, I might not. But I surely can listen. You are not alone.

Reach Out for Success

University students encounter setbacks from time to time. If you encounter difficulties and need assistance, it's important to reach out. Consider discussing the situation with me or an academic advisor. Learn about resources that assist with wellness and academic success at http://oregonstate.edu/ReachOut. If you are in immediate crisis, please contact the Crisis Text Line by texting OREGON to 741-741 or call the National Suicide Prevention Lifeline at 1-800-273-TALK (8255)

Basic Needs

Any student who has difficulty affording groceries or accessing sufficient food to eat every day, or who lacks a safe and stable place to live, is urged to contact the Human Services Resource Center (HSRC) for support: hsrc@oregonstate.edu, 541-737-3747. The HSRC has a food pantry, a textbook lending program, and other resources to help. Furthermore, if you are comfortable doing so, please notify me. I will do everything I can do to help you.

University Housing & Dining Services has been piloting a meal program called Food for Thought. Using the extra food generated by campus dining locations, they create pre-packaged, balanced meal options available at no cost to Oregon State students who meet eligibility criteria determined by the Human Services Resource Center. For others, these packages are available for purchase at a reduced price.

Other Useful Online Parallel Programming Information