Transcript
CSE399b, Computer Vision Project 3, Automatic 2D image mosaic and Face Morphing Due Tuesday April 17, 4:30pm
Overview This project focuses on image feature detection, feature matching and image mosaic techniques. We will follow technique outlined in the following paper: “Multi-image Matching using Multi-scale image patches”, Brown, M.; Szeliski, R.; Winder, S. CVPR 2005 ”Shape Matching and Object Recognition Using Shape Contexts”,S. Belongie, J. Malik, and J. Puzicha. PAMI 2002: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/shape/
Task 1: CAPTURE IMAGES For this project, you need to capture multiple images of a scene, which you will use to create image mosaic. In general, you should limit your camera motion to purely translational, or purely rotational (around the camera center). Bonus points will be given for interesting selection of images.
Task 2: AUTOMATIC CORRESPONDENCES You need to implement the following steps. 1. Detecting corner features in an image. You could follow method outlined in lecture note on Interest Point Detector, or your own image corner detector(for example derived from edge detector). You can probably find free “harris” corner detector on-line, and you are allowed to use it. 1
2. Implement Adaptive Non-Maximal Suppression. The goal is to create an uniformly distributed points. See section 3 of the reference paper, as well as lecture notes. 3. Extracting a Feature Descriptor for each feature point. You should use the subsampled image around each point feature. Dont worry about rotation-invariance just extract axis-aligned 8x8 patches. Note that its extremely important to sample these patches from the larger 40x40 window to have a nice big blurred descriptor. Dont forget to bias/gain-normalize the descriptors. As a bonus, you can implement the Geometric Blur features. 4. Matching these feature descriptors between two images. Remember to use the trick to compare the best and second-best match (take their ratio).
TASK 3: RANSAC and IMAGE MOSAIC Not all the matches computed in Task 2 will be correct. One way to remove incorrect matches is by implement the additional step of RANSAC (see lecture note on 2/21). 1. Use a robust method (RANSAC) to compute a homography. Use 4-point RANSAC as described in class to compute a robust homography estimate. Recall the RANSAC steps are: (a) Select four feature pairs (at random), pi , p1i (b) Compute homography H (exact). I will release the homography computation code shortly. (c) Compute inliers where SSD(p1i , Hpi ) < thresh (d) repeat step a-c (e) Keep largest set of inliers (f) Re-compute least-squares H estimate on all of the inliers 2. Proceed as in Project 1 on image mosaic to produce a mosaic. We will not worry about smooth image blending in this project.
2
TASK 4: AUTOMATIC FACE MORPHING We will use the procedures developed from Task 1 to Task 3 to automatically align face images, and create a face morphing video through a sequence of faces. You need to collect 12 or more face images. You are free to pick anyone, but it should include 6 celebrities (for example: http://www.galleryofcelebrities.com). You should allow variations in face pose and facial expression. You can manually crop out the face, center it, and resize to a standard size. Bonus for anyone who can automatically detect and crop out face images. Our tasks can be broken down to the following parts: 1. Feature detection. Using procedure from Task 2, compute the features for each face. For this task, it is important that you pick features at both coarse and fine image scale. 2. Pair-wise face alignment. For any pair of face images, we would like to find their best alignment. For this task, we need to generalize RANSAC from Task 3 to include deformable image transformation (Thin Plate Spline). You might need to experiment with number of control points needed for TPS. 3. Face similar measurement. Once we have aligned two face images, we need to compute a similar measure between them. We can compute the sum of feature matching errors, and the geometric distortion. This similar measure should be able to support query such as, who is the most similar looking celebrities to me? 4. Create an optimal sequence of face morphing. The goal is find a good ordering of the face images such that the resulting face morphing video is as smooth as possible. The best way is implement a dynamic programming algorithm for finding the optimal sequence, given pair-wise face similar measure. You can also implement a simpler version: (a) pick any image to start, (b) find the most similar image in the remaining images, (c) add it to the image sequence, and remove from the remaining images, (d) repeat step 2, until done. 5. Create the final video morphing. You can now create the video face morphing using your image morphing code, the given image ordering, and facial feature correspondences.
3
