$30
Project 2: SIFT Local Feature Matching
CS 6476
Brief
• Project materials including report template: proj2.zip
• Additional scenes to test on extra data.zip
• Hand-in: through Gradescope
• Required files: <your_gt_username>.zip, <your_gt_username>_proj2.pdf
Figure 1: The top 100 most confident local feature matches from a baseline implementation of project 2. In
this case, 89 were correct (lines shown in green), and 11 were incorrect (lines shown in red).
1
Overview
The goal of this assignment is to create a local feature matching algorithm using techniques described in
Szeliski chapter 7.1. The pipeline we suggest is a simplified version of the famous SIFT pipeline. The
matching pipeline is intended to work for instance-level matching – multiple views of the same physical
scene.
Setup
1. Install Miniconda. It doesn’t matter whether you use Python 2 or 3 because we will create our own
environment that uses python3 anyways.
2. Download and extract the project starter code.
3. Create a conda environment using the appropriate command. On Windows, open the installed “Conda
prompt” to run the command. On MacOS and Linux, you can just use a terminal window to run
the command, Modify the command based on your OS (linux, mac, or win): conda env create -f
proj2_env_<OS>.yml
4. This will create an environment named “cs6476 proj2”. Activate it using the Windows command,
activate cs6476_proj2 or the MacOS / Linux command, conda activate cs6476_proj2 or source
activate cs6476_proj2
5. Install the project package, by running pip install -e . inside the repo folder. This might be unnecessary for every project, but is good practice when setting up a new conda environment that may have
pip requirements.
6. Run the notebook using jupyter notebook ./proj2_code/proj2.ipynb
7. After implementing all functions, ensure that all sanity checks are passing by running pytest proj2_
unit_tests inside the repo folder.
8. Generate the zip folder for the code portion of your submission once you’ve finished the project using
python zip_submission.py --gt_username <your_gt_username>
Details
For this project, you need to implement the three major steps of a local feature matching algorithm (detecting
interest points, creating local feature descriptors, and matching feature vectors). We’ll implement two
versions of the local feature descriptor, and the code is organized as follows:
• Interest point detection in part1_harris_corner.py (see Szeliski 7.1.1)
• Local feature description with a simple normalized patch feature in part2_patch_descriptor.py (see
Szeliski 7.1.2)
• Feature matching in part3_feature_matching.py (see Szeliski 7.1.3)
• Local feature description with the SIFT feature in part4_sift_descriptor.py (see Szeliski 7.1.2)
2
1 Interest point detection (part1_harris_corner.py)
You will implement the Harris corner detection as described in the lecture materials and Szeliski 7.1.1.
The auto-correlation matrix A can be computed as (Equation 7.8 of book, p. 404)
A = w ∗
I
2
x
IxIy
IxIy I
2
y
= w ∗
Ix
Iy
Ix Iy
(1)
where we have replaced the weighted summations with discrete convolutions with the weighting kernel w
(Equation 7.9, p. 405).
The Harris corner score R is derived from the auto-correlation matrix A as:
R = det(A) − α · trace(A)
2
(2)
with α = 0.06.
Algorithm 1: Harris Corner Detector
Compute the horizontal and vertical derivatives Ix and Iy of the image by convolving the original
image with a Sobel filter;
Compute the three images corresponding to the outer products of these gradients. (The matrix A is
symmetric, so only three entries are needed.);
Convolve each of these images with a larger Gaussian.;
Compute a scalar interest measure using the formulas (Equation 2) discussed above.;
Find local maxima above a certain threshold and report them as detected feature point locations.;
To implement the Harris corner detector, you will have to fill out the following methods in part1_harris_corner
.py:
• compute_image_gradients(): Computes image gradients using the Sobel filter.
• get_gaussian_kernel_2D_pytorch(): Creates a 2D Gaussian kernel (this is essentially the same as your
Gaussian kernel method from project 1).
• second_moments(): Computes the second moments of the input image. You will need to use your
get_gaussian_kernel_2D_pytorch() method.
• compute_harris_response_map(): Gets the raw corner responses over the entire image (the previously
implemented methods may be helpful).
• maxpool_numpy(): Performs the maxpooling operation using just NumPy. This manual implementation
will help you understand what’s happening in the next step.
• nms_maxpool_pytorch(): Performs non-maximum suppression using max-pooling. You can use PyTorch
max-pooling operations for this.
• remove_border_vals(): Removes values close to the border that we can’t create a useful SIFT window
around.
• get_harris_interest_points(): Gets interests points from the entire image (the previously implemented methods may be helpful).
The starter code gives some additional suggestions. You do not need to worry about scale invariance or
keypoint orientation estimation for your baseline Harris corner detector. The original paper by Chris Harris
and Mike Stephens describing their corner detector can be found here.
3