Overview

CSCI 3343 is an advanced undergraduate-level computer vision class covering fundamental image basics (week 1-3), state-of-the-art deep learning (week 4-6) methods, and a wide range of real-world applications (week 7-14).

Schedule

The course materials expect different levels of understanding. Please take a look at them before class.
  • [C]onceptual: key concepts and applications; when things work or fail
  • [P]roblem solving: maths/coding details; hands-on debugging
Theme Date Topic Materials Assignments
Image Basics
(Week 1-3)
Mon, Aug. 30 Lec. 1: Introduction
Course: logistics
Computer vision: goal, applications
Motivation: why computer vison is so cool ps1 out (prerequisite)
lab1 out
final project (ideas out)
Wed, Sep. 1 Lec. 2: Image Formation
Light: EM wave, metamer, gamma curve
Material: diffusion, specular, BRDF
Imaging: optics and human perception
Fri, Sep. 3 Lab 1: Linear Algebra for Digital Images
Links: [pdf], [Colab]
Representation: matrix (NumPy)
Transformation: patch-wise function
Maths review: linear algebra ps1 due
No Class (Labor Day)
Wed, Sep. 8 Lec. 3: Cameras
Basic: Pin-hole and perspective geometry
Improvement: lens, digital sensor
Imaging: a brief history of cameras ps2 out (formation)
lab2 out
Fri, Sep. 10 Lab 2: Calculus for Function Optimization
Links: [pdf], [Colab]
Optimization 101
Gradient: multivariate, gradient descent
Maths review: calculus survey
Mon, Sep. 13 Lec. 4: Image Processing I (Filtering)
Linear/non-linear filters
Edge, denoising
Filtering: simple image filters
Wed, Sep. 15 Review Session
Pset2, Lec. 4
Fri, Sep. 17 Lec. 5: Image Processing II (Decomposition)
Fourier transform: 1 layer, N filters
Filtering: filter basis (breadth) ps2 due
Mon, Sep. 20 Lec. 5 (part 2)
Image pyramid: N layers, 1 filter
Filtering: filter recursion (depth) ps3 out (filter)
Deep Learning
(Week 4-6)
Wed, Sep. 22 Lec. 6: Machine Learning I
Machine learning overview
AlexNet: linear layer
Fri, Sep. 24 Lec. 7: Machine Learning II
Linear regression
Logistic regression, loss function
AlexNet: loss layer
Mon, Sep. 27 Lab. 3: Machine Learning
Links: [pdf], [Colab]
Pytorch basics
Generalization error
AlexNet: Softmax layer
lab3 out
Wed, Sep. 29 Lec. 8: Neural Networks I
Handcrafted features
Activation functions
AlexNet: fully-connected layer
Fri, Oct. 1 Lec. 9: Neural Networks II
Computation graphs
Backpropagation
AlexNet: optimization
Mon, Oct. 4 Lab 4: Multilayer Perceptron
Links: [pdf], [Colab]
Backpropagation as dynamic programming
CIFAR10: dataloader, tensorboard
ps3 due
lab4 out
final project (team info due)
Wed, Oct. 6 Lec. 10: Convolutional Networks
Convolution and pooling layers
Normalization and dropout layers
AlexNet: layers for efficient learning from images
Fri, Oct. 8 Lec. 11: AlexNet and Beyond
AlexNet visualization
VGGNet, BatchNorm, ResNet
Beyond AlexNet: three tricks to crack ImageNet ps4 out (ML+NN)
Image Understanding
(Week 7-8)
No Class (Fall Break)
Tue, Oct. 12 Lec. 12: Image Prediction
Applications and challenges
MVP and design choices
Target domain: scalar label final project (guideline)
Wed, Oct. 13 Lec. 13: Object Detection I
Problem formulation
Two-stage and single-stage
Target domain: bounding boxes
Fri, Oct. 15 Lec. 14: Object Detection II
R-CNN-based, YOLO
Target domain: bounding boxes
Mon, Oct. 18 Lab 5: Image Prediction and Object Detection in PyTorch
Links: [pdf], [Colab]
ps4 due
lab5 out
Wed, Oct. 20 Lec. 15: Image Segmentation
Generic/Semantic/Instance segmentation
FCN, SegNet, U-Net
Target domain: grid label ps5 out (finetune)
Fri, Oct. 22 Lec. 16: Image Generation
Links: [video]
GAN and conditional GAN
Style transfer
Target domain: natural images final project (proposal due)
Mon, Oct. 25 Lab 6: Image Segmentation and Genereation in PyTorch
Links: [pdf], [Colab], [video]
lab6 out
Video Understanding
(Week 9-10)
Wed, Oct. 27 Lec. 17: Motion
Links: [video]
Motion representation
Optical flow
Target domain: grid label
Fri, Oct. 29 Lec. 18: Video Prediction
Links: [video]
Action recognition
3D CNN and RNNs
Target domain: scalar label ps5 due
Mon, Nov. 1 Lec. 19: Object Tracking I
Links: [video]
Single-object tracking, Siamese NN
Target domain: temporally-linked bounding boxes check-in sign-up
Wed, Nov. 3 Lec. 20: Object Tracking II
Links: [video]
Multi-object tracking, Graph NN
Target domain: temporally-linked bounding boxes
Fri, Nov. 5 Lab 7: Tips For The Final Project
Links: [pdf], [video]
Sarah Barbrow @ BC library
Sarah Castricum @ BC Teaching Center
3D Understanding
(Week 11-12)
Mon, Nov. 8 Lec. 21: Image Transformation
Links: [video]
Homogeneous coordinate, homography
Target domain: transformation matrix ps6 out (motion)
final project (check-in)
Wed, Nov. 10 Lec. 22: Image Alignment
Links: [video]
Feature matching, RANSAC
Target domain: transformation matrix
Fri, Nov. 12 Lec. 23: Image Feature
Links: [video]
Feature detection, SIFT
Target domain: invariant feature
Mon, Nov. 15 Lec. 24: Stereo Vision
Perspective geometry
Stereo matching, depth prediction
Target domain: grid label
Wed, Nov. 17 Lec. 25: Point Cloud
Links: [audio]
Epipolar geometry, Structure from motion
PointNet
Source domain: point cloud ps6 due
Frontiers in Vision
(Week 13-14)
Fri, Nov. 19 Guest Lecture: Andrea Vicini, S.J.
Professor @ BC
Ethics in AI/Biomedical Research
ps7 out (stitch)
Mon, Nov. 22 Guest Lecture: James Tompkin
Assistant Professor @ Brown University
computer vision for creative media
final project (draft slide due)
No Class (Thanksgiving Break)
Mon, Nov. 29 Lec. 26: Multimodal Learning
Vision and soud
Vision and text
Wed, Dec. 1 Guest Lecture: Yue Wang
Final-year PhD @ MIT
Point cloud recognition, autonomous driving
Fri, Dec. 3 Guest Lecture: Kristen Grauman
Professor @ UT Austin (BC alum!)
Vision and sound
ps7 due
Final Projects
(Week 15-16)
Mon, Dec. 6 Presentation I
final project (final slide due)
Wed, Dec. 8 Presentation II
Mon, Dec. 20 final project (report/code due)


Staff & Office Hours


Instructor
Name Office hours
Donglai Wei (W) 11 am-noon, (F) 3-4 pm @ S276/CS student lounge
David Shen (Tue) 3-4 pm, (Thu) 6-7 pm @ Fulton 160
Yufan Yang (Tue) 10-11 am, (Thu) 3-4 pm @ Fulton 160
  • Office hours will take place in person (or Zoom if needed).
  • Donglai will hold additional one-on-one AMA office hours M/W 4-5pm (15-min by appointment)


Course information (aka. How to do well)

1. Attend the course with preparation

  • Lectures (MW): Do the reading before the lecture.
  • Labs (F): Bring your laptop.
  • Textbook: We'll be using the following two online books:

2. Ask for help (besides office hours)

  • Dropbox: The lecture pdfs will be uploaded to Dropbox (follow the link) and you can ask questions there by making comments on the slides directly.
  • Discord: For labs/psets/final projects, we will create dedicated channels for you to ask public questions. If you cannot make your post public (e.g., due to revealing problem set solutions), please create a room with instructors and TAs separately, or come to office hours. Please note, however, that the course staff cannot provide help debugging code, and there is no guarantee that they'll be able to answer last-minute homework questions before the deadline. We also appreciate it when you respond to questions from other students! If you have an important question that you would prefer to discuss over email, you may email the course staff, or you can contact the instructor by email directly.
  • Support: The university counseling services center provides a variety of programs and activities.
  • Accommodations for students with disabilities: If you are a student with a documented disability seeking reasonable accommodations in this course, please contact Kathy Duggan, (617) 552-8093, dugganka@bc.edu, at the Connors Family Learning Center regarding learning disabilities and ADHD, or Rory Stein, (617) 552-3470, steinr@bc.edu, in the Disability Services Office regarding all other types of disabilities, including temporary disabilities. Advance notice and appropriate documentation are required for accommodations.

3. Submit Pset/Lab in time on Canvas
Grades will be computed as follows, with all homeworks equally weighted:
Pset 70 points
Final project 30 points
Lab 0.5 point extra credit each
  • Pset: There will be a problem set approximately every week. All programming assignments are to be completed in Python on Colab, using the starter code that we provide. Assignments will always be due at midnight (11:59 pm) on the due date. The assignments will all be worth approximately equal amounts. You may be asked to annotate your pdf (e.g. by selecting your solution to each problem).
  • Final project: (guideline) In lieu of a final exam, we'll have final project. This project will be completed in small groups during the last weeks of the class. The direction for this project is open-ended: you can either choose from a list of project ideas that we distribute, or you can propose a topic of your own. A short project proposal will be due approximately halfway through the course. During the final exam period, you'll turn in a final report and give a short presentation. You may use an ongoing research work for your final project, as long it meets the requirements.
  • Lab: The materials are expected to be finished in class. If you submit the correctly finished Lab file before the next Fri class (10 am), you will receive 0.5 point per lab as the extra credit to your final grade. No late policy for the lab. You will receive 0.5 point per lab as the extra credit to your final grade, if your results are correct. Deadline is Dec 13th 11:59 pm.
  • Late policy: You'll have 10 late days to use over the course of the semester. Each time you use one, you may submit a homework assignment one day late without penalty (weekends don't count as late days). You do not need to notify us when you use a late day; we'll deduct it automatically. If you run out of late days and still submit late, your assignment will be penalized at a rate of 10% per day until 30% left (before Dec 13th 11:59 pm). If you edit your assignment after the deadline, this will count as a late submission, and we'll use the revision time as the date of submission (after a short grace period of a few minutes). We will not provide additional late time, except under exceptional circumstances, and for these we'll require documentation (e.g., a doctor's note). Please note that the late days are provided to help you deal with minor setbacks, such as routine illness or injury, paper deadlines, interviews, and computer problems; these do not generally qualify for an additional extension.
  • Academic integrity: While you are encouraged to discuss homework assignments with other students, your programming work must be completed individually. You may not search for solutions online, or to use existing implementations of the algorithms in the assignments. Thus it is acceptable to learn from another student the general idea for writing program code to perform a particular task, or the technique for solving a mathematical problem, but unacceptable for two students to prepare their assignments together and submit what are essentially two copies of identical work. If you have any uncertainty about the application of this policy, please check with me. Failure to comply with these guidelines will be considered a violation of the University policies on academic integrity. Please make sure that you are familiar with these policies.

Acknowledgements: This course draws heavily from MIT's 6.869: Advances in Computer Vision by Antonio Torralba, William Freeman, and Phillip Isola. It also includes lecture slides from other researchers, including Andrew Owens , Svetlana Lazebnik, Alexei Efros, Fei-fei Li, Carl Vondrick, David Fouhey, Justin Johnson, and Noah Snavely (please see acknowledgments in the lecture slides). Special thanks to Ce Liu for the insightful and generous advice.

Selected Q&A for the First Week

Check out the full list! Welcome to drop new questions there.

Q. Err, I learned those prerequisites before, but the memory is blurry now. Do you have any refresher materials?
A. Linear algebra (Essence, Chap 1-4), Calculus (Essence, Chap 1, 3-4, 8-9), Python ("Basics of Python" section).
Q. Hmm, the refreshers don't look too hard, but I'm still not sure if I can handle the course?
A. Pset 1 and Lab 1 will be out on Mon (check out Colab tutorial) and your maths/coding should be good if you can tackle them. It's also good if you can struggle through them to show your learning ability, which in the end is the goal of education.
Q. Wow, the course seems really intense!
A. Computer vision (AI) is super cool, but there is no free lunch. The course materials focus on core concepts and learning-through-coding, so that you can later be quick-to-understand and ready-to-code about AI in your career.
Q. What, final project from day one?!?
A. Relax, nothing serious is due until Oct. 18. We hope to get the best out of you by kindling your passion to apply computer vision to areas you care about dearly. As an intro course, we will only cover the basics for the exciting real-world applications (week 7-12). The final project is an effective way to dive deep into one topic and gain the full-cycle research experience.

We are here to assist you in becoming the better version of yourselves! Check out office hours and Discord!