Building ChatGPT
Logistics
Class Information
- Lecture
- M / W / F 11:30 AM - 12:20 PM
- Location
- 214 DeBart
- Slack
- #cse-30124-fa26
- GitHub
- nd-cse-30124-homeworks
Instructor
- Professor
- Bill Theisen (wtheisen@nd.edu)
- I think many students just call me Theisen (pronounced "Tyson"), which is fine too.
- Office Hours
- T 1:00 PM - 4:00 PM , TR 1:00 PM - 4:00 PM , and by appointment
- Office Location
- 350 Fitz
Teaching Assistants
Office Hours
Schedule
Click on the next to a topic for additional resources. indicates the required resource for the topic.
| Unit | Date | Topics | Assignments |
|---|---|---|---|
| Pre-training | Mon 08/24 | Syllabus | |
| Wed 08/26 | High Dimensional Data | ||
| Fri 08/28 | Programming Day 01: Linear Transformations | ||
| Sun 08/30 | Homework 01 | Homework 01 | |
| Mon 08/31 | Loss Functions | ||
| Wed 09/02 | Optimization | ||
| Fri 09/04 | Programming Day 02: Neural Networks | ||
| Sun 09/06 | Homework 02 | Homework 02 | |
| Mon 09/07 | Tokenization | ||
| Wed 09/09 | BPE Tokenization | ||
| Fri 09/11 | Programming Day 03: GPT4 Tokenizer | ||
| Sun 09/13 | Lab 01 | Lab 01 | |
| Mon 09/14 | Text Embeddings | ||
| Wed 09/16 | Anthropic Guest Lecture | ||
| Fri 09/18 | Programming Day 04: Embedding Layer | ||
| Sun 09/20 | Homework 03 | Homework 03 | |
| Mon 09/21 | Recurrent Neural Networks 01 | ||
| Wed 09/23 | Recurrent Neural Networks 02 | ||
| Fri 09/25 | Programming Day 05: Generative RNNs | ||
| Sun 09/27 | Lab 02, Homework 04 | Lab 02 Homework 04 | |
| Mon 09/28 | Self-Attention 01 | ||
| Wed 09/30 | Self-Attention 02 | ||
| Fri 10/02 | Programming Day 06: Self-Attention | ||
| Sun 10/04 | Homework 05 | Homework 05 | |
| Mon 10/05 | Multi-Head Attention | ||
| Wed 10/07 | Transformer Blocks | ||
| Fri 10/09 | Programming Day 07: Transformer Blocks | ||
| Sun 10/11 | Homework 06 | Homework 06 | |
| Mon 10/12 | Programming Day 08-1: Building GPT2 | ||
| Wed 10/14 | Programming Day 08-2: Building GPT2 | ||
| Fri 10/16 | Programming Day 08-3: Building GPT2 | ||
| Sun 10/18 | Lab 03 | Lab 03 | |
| Mon 10/19 | Fall Break | ||
| Wed 10/21 | Fall Break | ||
| Fri 10/23 | Fall Break | ||
| Exam 01 | Mon 10/26 | Review 01 | |
| Wed 10/28 | Review 01 | ||
| Fri 10/30 | Exam 01 | Practice Packet 01 Practice Packet 01 Solutions Exam 01 Exam 01 Solutions | |
| Post Training | Mon 11/02 | Post Training | |
| Wed 11/04 | LLM Evals | ||
| Fri 11/6 | Programming Day 09: Supervised Fine-Tuning | ||
| Sun 11/8 | Homework 07 | Homework 07 | |
| Mon 11/9 | RLHF | ||
| Wed 11/11 | Distillation | ||
| Fri 11/13 | Programming Day 10: RAG | ||
| Sun 11/15 | Lab 04 | Lab 04 | |
| Mon 11/16 | Harness Engineering 01 | ||
| Wed 11/18 | Harness Engineering 02 | ||
| Fri 11/20 | Programming Day 11: Harness Engineering | ||
| Sun 11/22 | Homework 08 | Homework 08 | |
| Mon 11/23 | Multimodal Models | ||
| Wed 11/25 | Thanksgiving Break | ||
| Fri 11/27 | Thanksgiving Break | ||
| Sun 11/29 | Homework 09 | Homework 09 | |
| Mon 11/30 | Current LLMs & Future Directions (Guest Lecture) | ||
| Wed 12/02 | Current LLMs & Future Directions (Guest Lecture) | ||
| Fri 12/04 | Current LLMs & Future Directions (Guest Lecture) | ||
| Sun 12/06 | Lab 05, Homework 10 | Lab 05 Homework 10 | |
| Exam 02 | Mon 12/07 | Review 02 | |
| Wed 12/09 | Exam 02 | Practice Packet 02 Practice Packet 02 Solutions Exam 02 Exam 02 Solutions |
There's loads of additional resources out there! If you find one that particuarly resonates with you, I'd appreciate it if you were willing to share it with the rest of the class. You'll even have the option to tag it with your name so future students can see who to thank!
Submit any additional resources to this google form: Additional Resources
Requirements
Coursework
| Component | Points |
|---|---|
| Lab Labs | 5 × 10 |
| Homework Homeworks | 10 × 5 |
| Midterm Midterm Exam (15 points for turning in Practice Packet 01) | 1 × 75 |
| Exam Final Exam (25 points for turning in Practice Packet 02) | 1 × 125 |
| Total | 300 |
Grading
| Grade | Points | Grade | Points | Grade | Points |
|---|---|---|---|---|---|
| A | 279-300 | A- | 270-278 | ||
| B+ | 260-269 | B | 250-259 | B- | 240-249 |
| C+ | 230-239 | C | 220-229 | C- | 210-219 |
| D | 195-209 | F | 0-194 |
Due Dates
Labs and Homeworks are due by 11:59 PM on the Sunday of the due week.
Practice Packets are due by 12:30 PM (start of class) on the day of the corresponding exam.
Practice Packets and Exams
I do something a little different in this class than I've seen in other ones. Instead of giving you old versions of the exams with which to practice, I will release a "practice packet". If you turn in this practice packet on the day of the exam (either online or paper is fine) then you will receive points on the exam itself for actually studying. The practice packet is graded entirely on completion. There will be an entry in canvas worth 0 points for each practice packet, the points you receive for doing it will be reflected in the exam score itself.
Policies
Participation
Students are expected to attend and contribute regularly in class. This means answering questions in class, participating in discussions, and helping other students.
Foreseeable absences should be discussed with the instructor ahead of time.
Community
Recalling one of the tenets of the Hacker Ethic:
Hackers should be judged by their hacking, not criteria such as degrees, age, race, sex, or position.
Students are expected to be respectful of their fellow classmates and the instructional staff.
Students with Disabilities
Any student who has a documented disability and is registered with Disability Services should speak with the professor as soon as possible regarding accommodations. Students who are not registered should contact the Office of Disabilities.
Academic Honesty
Any academic misconduct in this course is considered a serious offense, and the strongest possible academic penalties will be pursued for such behavior. Students may discuss high-level ideas with other students, but at the time of implementation (i.e. programming), each person must do his/her own work. Use of the Internet as a reference is allowed but directly copying code or other information is cheating. It is cheating to copy, to allow another person to copy, all or part of an exam or a assignment, or to fake program output. It is also a violation of the Undergraduate Academic Code of Honor to observe and then fail to report academic dishonesty. You are responsible for the security and integrity of your own work.
Late Work
In the case of a serious illness or other excused absence, as defined by university policies, coursework submissions will be accepted late by the same number of days as the excused absence.
Otherwise, there is an automatic 25% late penalty for assignments turned in 12 hours pass the specified deadline.
Classroom Recording
This course will be recorded using Zoom and Panopto. This system allows us to automatically record and distribute lectures to you in a secure environment. You can watch these recordings on your computer, tablet, or smartphone. In the course in Sakai, look for the "Panopto" tool on the left hand side of the course.
Because we will be recording in the classroom, your questions and comments may be recorded. Recordings typically only capture the front of the classroom, but if you have any concerns about your voice or image being recorded please speak to me to discuss your concerns. Except for faculty and staff who require access, no content will be shared with individuals outside of your course without your permission.
These recordings are jointly copyrighted by the University of Notre Dame and your instructor. Posting them to other websites (including YouTube, Facebook, SnapChat, etc.) or elsewhere without express, written permission may result in disciplinary action and possible civil prosecution.
CSE Guide to the Honor Code
For the assignments in this class, you are allowed to consult printed and online resources and to discuss the class material with other students. You may also consult AI Tools such as CoPilot or ChatGPT for help explaining concepts, debugging problems, or as a reference. Viewing or consulting solutions, such as those from other students, previous semesters, or generated by AI Tools is never allowed.
Likewise, you may copy small and trivial snippets from books, online sources, and AI Tools as long as you cite them properly. However, you may not copy solutions or significant portions of code from other students or online sources, nor may you generate solutions via AI Tools.
Finally, when preparing for exams in this class, you may not access exams from previous semesters, nor may you look at or copy solutions from other current or former students.
| Resources | Solutions | |
|---|---|---|
| Consulting | Allowed | Not Allowed |
| Copying | Cite | Not Allowed |
See the CSE Guide to the Honor Code for definitions of the above terms and specific examples of what is allowed and not allowed when consulting resources.
If you are unclear about whether certain forms of consultation or common work are acceptable or what the standards for citation are, you responsible for consulting your instructor.
If an instructor sees behavior that is, in his judgement, academically dishonest, he is required to file either an Honor Code Violation Report or a formal report to the College of Engineering Honesty Committee.
Resources
Submit any questions or suggestions to this anonymous google form: Questions and Suggestions
Note: This form is genuinely anonymous but anonymity is a priviledge. Please don't misuse it.
One of the benefits of ML/AI being extremely popular is there are many online resources available for learning it. The best teachers of the topics release much of their material avaible for free online. If something in class seemed unclear, you're encouraged to seek out an explanation that makes the most sense to you! If you find one that you really like, please share it with the rest of the class. Below are links to books, blogposts, and lectures that I personally find very useful.