Transcript
5/13/2008
Admin • Next week:
Depth and defocus Lecture 11
Obtaining depth • Multiple viewpoints (Stereo) • Special camera (last lecture on May 9th) • Active methods
– Anat Levin on Matting – Tuesday 11am on her SIGGRAPH 2008 paper
• Projects – Can everyone send me ½ page telling me where they are/what they’ve done – Projects due Friday 9th 4pm (last class)
Depth from Stereo • Huge amount of literature in Computer Vision • Good survey paper: A Taxonomy and Evaluation of Dense Two‐Frame Stereo Correspondence Algorithms. Daniel Scharstein & Richard Szeliski
– Change scene illumination
• Passive methods (require scene texture) – Defocus analysis – Multiple images – Single image
Active methods • Illuminate scene with light – Laser range scan – Binary pattern
http://www.cs.washington.edu/homes/curless/publications/cg99.pdf
Estimated Depth Map
1 of 2 input images
Ground truth Depth map
Active methods • Illuminate scene with light – Binary or colored patterns – Can use LCD projectors
Rapid Shape Acquisition Using Color Structured Light and Multi‐pass Dynamic Programming by Li Zhang, Brian Curless, Steven M. Seitz
1
5/13/2008
Active methods • Use shadows: – 3D Photography using shadows in dual‐space geometry (Bouguet, J.Y & Perona, P.)
Defocus & Depth of field Slides from Fredo Durand (MIT)
Circle of confusion
circle of confusion
From Basic Photographic Materials and Processes, Stroebel et al.
Depth of focus
Size of permissible circle? • Assumption on print size, viewing distance, human vision – Typically for 35mm film: diameter = 0.02mm
• Film/sensor resolution (8μ photosites for high-end SLR ) • Best lenses are around 60 lp/mm • Diffraction limit
From Basic Photographic Materials and Processes, Stroebel et al.
2
5/13/2008
Depth of field: Object space
Depth of field: more accurate view
• Simplistic view: double cone – Only tells you about the value of one pixel – Things are in fact a little more complicated to asses circles of confusion across the image – We're missing the magnification factor (proportional to 1/distance and focal length)
• Backproject the image onto the plane in focus – Backproject circle of confusion – Depends on magnification factor • Depth of field is slightly asymmetrical
sensor
Conjugate of circle of confusion
Point in focus lens
Point in focus lens
Object with texture
Depth of field
Depth of field: more accurate view
Deriving depth of field
• Backproject the image onto the plane in focus – Backproject circle of confusion – Depends on magnification factor ~ f/D
• • • •
Circle of confusion C, magnification m Simplification: m=f/D Focusing distance D, focal length f, aperture N As usual, similar triangles
D
~f
D
C
CD/f
f/N
CD/f
lens
d2
d1
Deriving depth of field
Deriving depth of field
D-d1
D
f/N
CD/f
d1
f/N
CD/f
d1
d2
3
5/13/2008
Deriving depth of field
Depth of field and aperture
N2C2D2 term can often be neglected when DoF is small (conjugate of circle of confusion is smaller than lens aperture)
• Linear: proportional to f number • Recall: big f number N means small physical aperture
D
f/N
f/N
CD/f
d2
d1
CD/f
d1
d2
DoF & aperture
Depth of field & focal length
• http://www.juzaphoto.com/eng/articles/depth_of_field.htm
• Recall that to get the same image size, we can double the focal length and the distance • Recall what happens to physical aperture size when we double the focal length for the same f number? – It is doubled
24mm f/2.8
50mm
f/32
Depth of field & focal length
DoF & Focal length
• Same image size (same magnification), same f number DoF • Same depth of field!
• http://www.juzaphoto.com/eng/articles/depth_of_fiel d.htm
Wide-angle g lens
DoF
50mm f/4.8 Telephoto lens (2x f), same aperture
200mm f/4.8 (from 4 times farther)
See also http://luminous-landscape.com/tutorials/dof2.shtml
4
5/13/2008
Important conclusion
Important conclusion
• For a given image size and a given f number, the depth of field (in object space) is the same. • Might be counter intuitive.
• For a given image size and a given f number, the depth of field (in object space) is the same. – The depth of acceptable sharpness is the same • But background far far away looks more blurry g more Because it ggets magnified • Plus, usually, you don't keep magnification constant
• Very useful for macro where DoF is critical. critical You can change your working distance without affecting depth of field • Now what happens to the background blur far far away?
Effect of parameters
Is depth of field a blur? aperture
focusing distance
focal length
From applied photographic optics
Bokeh
• Depth of field is NOT a convolution of the image • The circle of confusion varies with depth • There are interesting occlusion effects • (If you really want a convolution, there is one, but in 4D space… more about this in ten days)
From Macro Photography
Shape depends on aperture
• Pattern of out‐of‐focus blur
• Also on location within image
http://www.bobatkins.com/photo graphy/technical/bokeh.html
http://www.bobatkins.com/photography/technical/bokeh.html
5
5/13/2008
Comparison of lenses
Mirror Lens • Blur pattern
http://photo.net/learn/optics/mirrors/tamron500‐8a.jpg
http://www.bobatkins.com/photography/technical/bokeh.html
Confocal Stereo
Depth from defocus – Multiple images • Multiple sensors
S. Nayar
• Hasinoff & Kutulakos ECCV’06 • Vary aperture and focus of lens – Multiple images
• Time multiplex – Need static scene
Output #1: Depth map
Image and Depth from a Conventional Camera with a Coded Aperture
Single input image:
Anat Levin, Rob Fergus, Frédo Durand, William Freeman MIT CSAIL
6
5/13/2008
Output #1: Depth map
Single input image:
O Output #2: #2 All-focused All f d iimage
Lens and defocus Image of a point light source
Lens’ aperture
Lens
Camera sensor
Point spread function
Focal plane
Lens and defocus
Lens and defocus Image of a defocused point light source
Lens’ aperture
Object
Lens
Camera sensor
Object
Point spread function
Focal plane
Image of a defocused point light source
Lens’ aperture
Lens
Camera sensor
Point spread function
Focal plane
7
5/13/2008
Lens and defocus
Lens and defocus Image of a defocused point light source
Lens’ aperture
Object
Camera sensor
Lens
Image of a defocused point light source
Lens’ aperture
Object
Camera sensor
Lens
Point spread function
Point spread function
Focal plane
Focal plane
Depth and defocus
Challenges
Out of focus
• Hard to discriminate a smooth scene from defocus blur
?
Out of focus
Depth from defocus: Infer depth by analyzing local scale of defocus blur
• Hard to undo defocus blur
Input In focus
Ringing with conventional deblurring algorithm
Related Work
Key contributions • Exploit prior on natural images
• Depth from (de)focus
e.g.
• Plenoptic/ light field cameras
e.g.
Pentland, Chaudhuri, Favaro et al.
- Improve deconvolution
Adelson and Wang, Ng et al.
- Improve depth discrimination
• Wave front coding
e.g.
• Coded apertures for light gathering:
e.g.
• Blind Deconvolution
e.g.
Cathey & Dowski
Natural
• Coded aperture (mask inside lens) - make defocus patterns different from
Unnatural
F i Fenimore and dC Cannon
Kundur and Hatzinakos , Fergus et al, Levin
natural images and easier to discriminate
Never recover both depth AND full resolution image from a single image Except: Veeraraghavan, Raskar, Agrawal, Mohan, Tumblin SIGGRAPH07 optimize debluring while we optimize depth discrimination
8
5/13/2008
Defocus as local convolution
Defocus as local convolution
y = fk ⊗ x
Calibrated blur kernels at different depths Input defocused image
Local sub-window
Input defocused image
y = fk ⊗ x
Depth k=2:
y = fk ⊗ x
Depth k=3:
y = fk ⊗ x
Challenges
Try deconvolving local input windows with different scaled filters:
= ⊗ = ⊗
Sharp sub-window
Depth k=1:
Overview
= ⊗
Calibrated blur kernels at depth k
? ? ?
• Hard to deconvolve even when kernel is known Input
Larger scale
Correct scale
• H Hard d to identify id if correct scale:
Smaller scale
Somehow: select best scale.
Deconvolution is ill posed
? ? ?
Ringing with the traditional Richardson-Lucy deconvolution algorithm
= ⊗
Larger scale
= ⊗
Correct scale
= ⊗
Smaller scale
Deconvolution is ill posed
f ⊗ x = y
f ⊗ x = y Solution 1:
⊗
?
⊗
=
?
=
Solution 2:
⊗
?
=
9
5/13/2008
Idea 1: Natural images prior
Deconvolution with prior
Natural
| f ⊗ x − y |2 + λ ∑i ρ (∇xi )
x = arg min
What makes images special? Unnatural
Convolution error
Image
⊗
?
Derivatives prior
2
_
+ Low
Equal convolution error
gradient
⊗
?
Natural images have sparse gradients
2
_
+ High
put a penalty on gradients
Comparing deconvolution algorithms
Comparing deconvolution algorithms
(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/
Input
Richardson-Lucy
ρ (∇ x ) = ∇ x
ρ (∇ x ) = ∇ x
2
0 .8
“spread” gradients
“localizes” gradients
Gaussian prior
Sparse prior
Recall: Overview
(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/
Input
ρ (∇ x ) = ∇ x
Richardson-Lucy
2
ρ (∇ x ) = ∇ x
0 .8
“spread” gradients
“localizes” gradients
Gaussian prior
Sparse prior
Idea 2: Coded Aperture
Try deconvolving local input windows with different scaled filters:
• Mask (code) in aperture plane
= ⊗
Larger scale
= ⊗
C Correct t scale l
= ⊗
Smaller scale
? ? ?
Somehow: select best scale. Challenge: smaller scale not so different than correct
- make defocus patterns different from natural images and easier to discriminate
Conventional aperture
Our coded aperture
10
5/13/2008
Solution: lens with occluder
Solution: lens with occluder Image of a defocused point light source
Aperture pattern
Object
Lens
Camera sensor
Object
Lens with coded aperture
Point spread function
Point spread function
Focal plane
Focal plane
Solution: lens with occluder
Solution: lens with occluder Image of a defocused point light source
Aperture pattern
Object
Lens with coded aperture
Camera sensor
Image of a defocused point light source
Aperture pattern
Object
Lens with coded aperture
Point spread function
Focal plane
Point spread function
Solution: lens with occluder Image of a defocused point light source
Aperture pattern
Lens with coded aperture
Camera sensor
Image of a defocused point light source
Aperture pattern
Object
Point spread function
Focal plane
Camera sensor
Focal plane
Solution: lens with occluder
Object
Camera sensor
Lens with coded aperture
Camera sensor
Point spread function
Focal plane
11
5/13/2008
Convolution- frequency domain representation
spectrum
Larger scale
spectrum
Correct scale
scale
0
Frequency
0
Frequency
0
Frequency
Filter, 2nd scale
0
0
spectrum
Filter, correct scale
Frequency
Division by zero
⇒spatial ringing
ω
ω
Frequency
=
Frequency
Output spectrum has zeros
? Frequency
=
Observed image
0 0
spectrum
spectrum spectrum
Filter, wrong scale
?
ω
2nd observed image
⇔frequency multiplication where filter spectrum has zeros
0
0
Estimated image
Estimated image
Frequency
0
=
Division by zero with a conventional aperture?
spectrum
=
Observed image
Frequency
0
Estimated image
spectrum
spectrum spectrum
Filter, correct scale
spectrum
Frequency
1st observed image
Sharp Image
Coded aperture: Scale estimation and division by zero
0
=
0
Spatial convolution
?
spectrum
Filter,
Smaller scale
Estimated image
Frequency
0
1st
spectrum
Coded
spectrum
Conventional
Sharp Image
spectrum
Coded aperture- reduce uncertainty in scale identification
spectrum
Why coded?
Filter, wrong scale
ω
Frequency
0
0
Filter Design
Frequency
division of tiny value by zero ⇒ no spatial ringing
?
Frequency
ω
Frequency
ω
=
Frequency
Zero frequencies- pros and cons
Analytically search for a pattern maximizing discrimination between images at different defocus scales (KL-divergence) Previous talk:
Account for image prior and physical constraints
See paper for details
0
Score
Our solution:
0
0
No zero frequencies:
More discrimination between scales
+ -
Less discrimination between scales
Sampled aperture patterns
Conventional aperture
Filter can be easily inverted Weaker depth discrimination
Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Ashok Veeraraghavan Ramesh Raskar Amit Agrawal Mitsubishi Electric Research Labs (MERL)כ, Cambridge, MA Ankit Mohan† Jack Tumblin‡
0
Include zero frequencies:
+ +
Zeros improve depth discrimination Inversion difficult Inversion made possible with image priors
12
5/13/2008
Regularizing depth estimation Try deblurring with 10 different aperture scales
x = arg min | f ⊗ x − y |2 Convolution error
Derivatives prior
2
_
⊗
Depth results
+ λ ∑i ρ (∇xi )
+
Keep minimal error scale in each local window + regularization 200 235 245 255 265 275 285 295 305
Input
Regularizing depth estimation
Local depth estimation
Regularized depth
Sometimes, manual intervention 200
200
235
235
245
245
255
255
265
265 275
275
285
285
295
295 305
Input
305
Local depth estimation
Input
Local depth estimation
200
235
235
235
245
245
245
255
255
255
265
265
265
275
275
275
285
285
285
295
295
295
305
305
Regularized depth
305
After user corrections
Regularized depth
Input
All focused results
13
5/13/2008
Close-up
All-focused (deconvolved)
Original image
All-focus image
Input
All-focused (deconvolved)
Close-up
Comparison- conventional aperture result Ringing due to wrong scale estimation
Original image
All-focus image
Naïve sharpening
14
5/13/2008
Comparison- coded aperture result
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
15
5/13/2008
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Coded aperture: pros and cons
Deconvolution code available
+ Image AND depth at a single shot + No loss of image resolution + Simple modification to lens - Depth is coarse
http://groups.csail.mit.edu/graphics/CodedAperture/
unable to get depth at untextured areas, might g need manual corrections.
+ But depth is a pure bonus - Loss some light + But deconvolution increases depth of field
Admin • Fill in feedback forms – Can someone collect and return to 305WWH
50mm f/1.8:
$79.95
Cardboard:
$1
Tape:
$1
Depth acquisition:
priceless
16
