Graphics Programming
Cameras: Perspective Projection

• A pinhole camera consists of a camera body with a very small hole through which the light can enter
• The image is formed at the back of the camera body and is displayed upside-down

• A larger hole has the advantage that more light can enter the camera, resulting in shorter exposure times
• The disadvantage is that multiple projections overlap and the image is out of focus

object

camera

pinhole

image
of the object

larger pinhole

focal length

center of projection

image plane

focal length

image plane

$x$

$y$

$z$

$x$

$f$

$z$

$\tilde{\mathbf{P}}$

$\mathbf{P}$

$\tilde{\mathbf{P}}$

$\mathbf{P}$

$f \frac{p_x}{p_z}$

$f$

• The formula for mapping a 3D point $\mathbf{P}=(p_x,p_y,p_z)^\top$ to a point $\tilde{\mathbf{P}}= (\tilde{p}_x,\tilde{p}_y,\tilde{p}_z)^\top$ located at the image plane of the camera is given by:

  $\tilde{\mathbf{P}}= \left( f \frac{p_x}{p_z}, f \frac{p_y}{p_z}, f \right)^\top$

• This follows immediately from the figure by application of the intercept theorem, since

  $\frac{\tilde{p}_x}{f} = \frac{p_x}{p_z}$ and $\frac{\tilde{p}_y}{f} = \frac{p_y}{p_z}$

• Using homogeneous coordinates the perspective projection can be written as a linear mapping using a $4 \times 4$ matrix:

  $\tilde{\mathbf{P}}= \begin{pmatrix} \tilde{p}_x \\ \tilde{p}_y \\ \tilde{p}_z \end{pmatrix}= \begin{pmatrix} f \frac{p_x}{p_z}\\ f \frac{p_y}{p_z}\\ f \end{pmatrix} \in \mathbb{R}^3 \longmapsto \underline{\tilde{\mathbf{P}}}= \begin{pmatrix}f \, p_x \\f \, p_y \\ f \, p_z\\ p_z\end{pmatrix} \in \mathbb{H}^3$

  $\begin{align}\underline{\tilde{\mathbf{P}}} & = \begin{pmatrix}f \, p_x \\f \, p_y \\ f \, p_z\\ p_z\end{pmatrix} = \underbrace{\begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & f & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}}_{\mathtt{A}} \begin{pmatrix}p_x \\p_y \\ p_z\\ 1\end{pmatrix}\\ \underline{\tilde{\mathbf{P}}} &=\mathtt{A}\, \underline{\mathbf{P}} \end{align}$

$f$

image plane

$x$

$y$

$z$

• In OpenGL, the camera is pointing in the negative $z$-direction. Therefore, we have:

  $\begin{align}\underline{\tilde{\mathbf{P}}} & = \begin{pmatrix}f \, p_x \\f \, p_y \\ f \, p_z\\ -p_z\end{pmatrix} = \underbrace{\begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & f & 0 \\ 0 & 0 & -1 & 0 \end{bmatrix}}_{\mathtt{A}} \begin{pmatrix}p_x \\p_y \\ p_z\\ 1\end{pmatrix}\\ \underline{\tilde{\mathbf{P}}} &=\mathtt{A}\, \underline{\mathbf{P}} \end{align}$

near

far

$x$

$z$

$-z_n$

$-z_f$

displayed
area

• In OpenGL, there is a so-called near- and a far-clipping plane
• The near-plane and far-plane are located parallel to the image plane
• Points are only displayed if their $z$-coordinate lies within the range defined by the near- and far-plane
• To this end, a new linear mapping is defined, such that for points with a $z$-coordinate on the near-plane it holds:
  

  $p_z=-z_n \quad \mapsto \quad \tilde{p}_z=-1$

  and for points on the far-plane:

  $p_z=-z_f \quad \mapsto \quad \tilde{p}_z=1$

• In order to accomplish this, two new parameters $\alpha$ and $\beta$ are added to the linear transformation matrix

$\underline{\tilde{\mathbf{P}}} = \begin{pmatrix}f \, p_x \\f \, p_y \\ \alpha \, p_z + \beta \\ -p_z\end{pmatrix} = \begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & \alpha & \beta \\ 0 & 0 & -1 & 0 \end{bmatrix} \begin{pmatrix}p_x \\p_y \\ p_z\\ 1\end{pmatrix} \in \mathbb{H}^3$

• Thus, the projected point in Cartesian coordinates is:

  $\tilde{\mathbf{P}}=(\tilde{p}_x,\tilde{p}_y,\tilde{p}_z)^\top = \left( f \frac{p_x}{-p_z}, f \frac{p_y}{-p_z}, -\alpha \, + \frac{\beta}{-p_z} \right)^\top \in \mathbb{R}^3$

• Now $\alpha$ and $\beta$ can be determined from the conditions for the mapping of the $z$-coordinate:
  $\begin{align}p_z=-z_n \,&\mapsto \, \tilde{p}_z=-1 \quad \Rightarrow -\alpha \, + \frac{\beta}{z_n} = -1 \\ p_z=-z_f \, &\mapsto \, \tilde{p}_z=\ 1 \,\,\, \,\quad \Rightarrow -\alpha \, + \frac{\beta}{z_f} = 1\end{align}$

• Solving the equation system for $\alpha$ and $\beta$ provides:

  $\begin{align} \alpha &= \frac{z_f+z_n}{z_n-z_f}\\ \beta & = \frac{2 z_f \, z_n}{z_n-z_f}\end{align}$

• Thus, for the new projection matrix we have:

  $\begin{align} \underline{\tilde{\mathbf{P}}} & = \underbrace{\begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & \frac{z_f+z_n}{z_n-z_f} & \frac{2 z_f \, z_n}{z_n-z_f} \\ 0 & 0 & -1 & 0 \end{bmatrix}}_{\mathtt{A}} \begin{pmatrix}p_x \\p_y \\ p_z\\ 1\end{pmatrix}\\ \underline{\tilde{\mathbf{P}}} &=\mathtt{A}\, \underline{\mathbf{P}} \end{align}$

focal length A

image height A

focal length B

image height B

$y$

$z$

1

-1

$\Theta$

• Until now we have only defined the focal length $f$ as the distance between the image plane and the camera center, but nothing was stated about the size of the image plane in $x$- and $y$-direction
• In the end, only the ratio between the size of the image plane and the focal length is important, which is uniquely defined by the opening angle $\Theta$. All configurations with the same opening angle result in the same image (only with scaled $x$- and $y$-coordinates).
• In OpenGL, the size of the image plane is always chosen such that the resulting $x$- and $y$-coordinates are in the range $[-1; 1]$.
• For a given opening angle the focal length is therefore obtained by (compare figure):

  $\frac{f}{1} = \frac{\cos( 0.5 \, \Theta)}{\sin( 0.5 \, \Theta)} \Leftrightarrow f = \mathrm{cotan}( 0.5 \, \Theta)$

• The idea of the "Dolly Zoom" effect is to compensate a camera translation in $z$-direction ("Dolly") by a change in focal length ("Zoom")
• Mathematically it is easy to see from the projection equation that achieving the compensation is possible, because for the $y$-coordinate of a projected point we have:

  $\tilde{p}_y = f \frac{p_y}{-p_z}$

• Since there is only one focal length $f$ but typically many 3D points with different depth value $p_z$ in the scene, the compensation can only be achieved for a selected depth value. This creates an interesting perspective effect.
• A well-known movie is Vertigo (1958) by Alfred Hitchcock, who has used this effect to simulate dizziness of the protagonist

$\mathbf{C}_a$

$\mathbf{C}_b$

$\mathbf{e}_x$

$\mathbf{e}_y$

$\mathbf{e}_z$

$\tilde{\mathbf{a}}_x$

$\tilde{\mathbf{a}}_y$

$\tilde{\mathbf{a}}_z$

$\tilde{\mathbf{b}}_x$

$\tilde{\mathbf{b}}_y$

$\tilde{\mathbf{b}}_z$

World coordinate system

Local coordinate system

Camera coordinate system

• The transformation matrices are given by the basis vectors of the coordinate systems (as discussed in the chapters before)
• The transformation matrix $\mathtt{T}_{\mathrm{\small obj}}$ transforms a point from the local to the global coordinate system

  $ \mathtt{T}_{\mathrm{\small obj}} = \begin{bmatrix}\tilde{\mathbf{b}}_x & \tilde{\mathbf{b}}_y & \tilde{\mathbf{b}}_z & \mathbf{C}_b\\0 & 0 & 0 & 1\end{bmatrix}$

• The transformation matrix $\mathtt{T}_{\mathrm{\small cam}}$ transforms a point from the camera to the world coordinate system

  $\begin{align} \mathtt{T}_{\mathrm{\small cam}} & = \begin{bmatrix}\tilde{\mathbf{a}}_x & \tilde{\mathbf{a}}_y & \tilde{\mathbf{a}}_z & \mathbf{C}_a\\0 & 0 & 0 & 1\end{bmatrix} \\ & = \begin{bmatrix} \mathtt{R}_a & \mathbf{C}_a\\ \mathbf{0}^\top & 1\end{bmatrix} \end{align}$

• To simplify the definition of the matrix $\mathtt{T}_{\mathrm{\small cam}}^{-1}$ there is the GLU function

  gluLookAt(eyex, eyey, eyez, refx, refy, refz, upx, upy, upz);

• By setting up an eye point $\mathbf{C}_{\mathrm{\small eye}}$, a targeted reference point  $\mathbf{P}_{\mathrm{\small ref}}$, and a vector $\mathbf{v}_{\mathrm{\small up}}$ (which defines the direction in which the $y$-coordinate of the camera is pointing) the basis vectors of the camera coordinate system can be computed:
  

  eye point $\mathbf{C}_{\mathrm{\small eye}}$

  reference point $\mathbf{P}_{\mathrm{\small ref}}$

  up vector $\mathbf{v}_{\mathrm{\small up}}$

  $\tilde{\mathbf{a}}_x$

  $\tilde{\mathbf{a}}_y$

  $\tilde{\mathbf{a}}_z$

  $\begin{align} \mathbf{d} & = \mathbf{C}_{\mathrm{\small eye}} - \mathbf{P}_{\mathrm{\small ref}}\\ \tilde{\mathbf{a}}_z &= \frac{\mathbf{d}}{|\mathbf{d}|}, \mathbf{v}' = \frac{\mathbf{v}_{\mathrm{\small up}}}{|\mathbf{v}_{\mathrm{\small up}}|} \\ \tilde{\mathbf{a}}_x &= \mathbf{v}'\times \tilde{\mathbf{a}}_z \\ \tilde{\mathbf{a}}_y &= \tilde{\mathbf{a}}_z \times \tilde{\mathbf{a}}_x\\ \mathtt{R}_{a} & = \begin{bmatrix}\tilde{\mathbf{a}}_x & \tilde{\mathbf{a}}_y & \tilde{\mathbf{a}}_z \end{bmatrix} \\ \end{align}$

  This results in:

  $\mathtt{T}_{\mathrm{\small cam}}^{-1} = \begin{bmatrix} \mathtt{R}_a^{\top} & -\mathtt{R}_a^{\top} \mathbf{C}_{\mathrm{\small eye}}\\ \mathbf{0}^\top & 1\end{bmatrix}$

• Which transformations are applied to the vertices of the smaller cube?

  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  gluPerspective (...);
  
  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
  gluLookAt(...);
  
  
  glRotatef(...);
  glTranslatef(...);
  glScalef(...);
  

  $\mathtt{T}_{\mathrm{\small projection}}= \mathtt{I}$
  $\mathtt{T}_{\mathrm{\small projection}}= \mathtt{I} \, \mathtt{A}$
  
  $\mathtt{T}_{\mathrm{\small modelview}}= \mathtt{I}$
  $\mathtt{T}_{\mathrm{\small modelview}}= \mathtt{I}\,\mathtt{T}_{\mathrm{\small cam}}^{-1}$
  
  $\mathtt{T}_{\mathrm{\small modelview}}= \mathtt{I}\,\mathtt{T}_{\mathrm{\small cam}}^{-1} \,\mathtt{T}_r$
  $\mathtt{T}_{\mathrm{\small modelview}}= \mathtt{I}\,\mathtt{T}_{\mathrm{\small cam}}^{-1} \,\mathtt{T}_r\,\mathtt{T}_t$
  $\mathtt{T}_{\mathrm{\small modelview}}= \mathtt{I}\,\mathtt{T}_{\mathrm{\small cam}}^{-1} \,\mathtt{T}_r\,\mathtt{T}_t\,\mathtt{T}_s $
  
  

  $\begin{align} \underline{\tilde{\mathbf{P}}} &= \mathtt{T}_{\mathrm{\small projection}} \mathtt{T}_{\mathrm{\small modelview}} \, \underline{\mathbf{P}}\\ &= \mathtt{A} \, \mathtt{T}_{\mathrm{\small cam}}^{-1} \,\mathtt{T}_r\,\mathtt{T}_t\,\mathtt{T}_s \,\underline{\mathbf{P}} \end{align}$

When using the fixed-function pipeline the following transformations are applied on the vertex data

• The projection matrix was designed such that after projection and perspective division all $x$, $y$ and $z$-coordinates within the visible volume are mapped to the range $-1$ to $1$
• All primitives that are completely outside this range must not be drawn
• By testing for the range $[-1;1]$ it would be easy to implement clipping after the perspective division step
• In OpenGL, the clipping is carried out before the perspective division. Why?

• Instead of testing the range $[-1;1]$ the range $[-p_w;p_w]$ can be checked just as quickly

  $-p_w < p_x < p_w \quad \longmapsto \quad -1 < \frac{p_x}{p_w} < 1$

• This has the advantage that
  • for the case $p_w=0$ no special treatment is needed and
  • the division computation for clipped coordinates is no longer needed

• In the previous examples glEnable(GL_DEPTH_TEST) and glClear(GL_DEPTH_BUFFER_BIT) were used without discussing their functionality
• The function call glEnable(GL_DEPTH_TEST) is used to activated the depth test in OpenGL
• If the depth test is disabled, the primitives are written into the framebuffer in the order in which they are passed into the OpenGL pipeline
• This means that later drawn primitives are covering the ones drawn earlier
• This is typically not the desired behavior
• Instead, primitives that are closer to the camera should cover more distant ones, regardless of the order of drawing
• Ideally, the decision for each drawn pixel should be done in the framebuffer, because the individual primitives can penetrate each other
• In OpenGL the Z-Buffer method is employed

$x$

Normalized device coordinates

Camera coordinates

$\mathtt{A}$

$x$

$y$

$z$

$x$

$y$

$z$

• Although actually the $z$-coordinate in the camera coordinate system is the one to consider, the depth test can be carried out after the perspective division, since the depth relations are not changed
• However, when using "Normalized device coordinates" the $z$-axis is reversed with respect to the camera coordinate system, i.e., more distant points have a larger $z$
  (note, the left-handed coordinate here)
• For points on the near-plane in the camera coordinate system, now applies $\tilde{p}_z=-1$ and respectively for the ones on the far-plane $\tilde{p}_z=1$

• The Z-Buffer method requires (in addition to the usual framebuffer which contains the color information) a depth buffer of the same dimensions, which contains the depth values

Framebuffer        Depth Buffer

• At the beginning of the rendering process, the depth buffer is initialized with the z-values of the far-plane. This is done in OpenGL using the command glClear(GL_DEPTH_BUFFER_BIT)
• Writing a pixel in the frame- and depth-buffer occurs during the per-fragment operations in the OpenGL pipeline
• The depth value for each pixel is interpolated by the rasterizer using the transformed vertex information
• If the depth value for the pixel is smaller than the currently stored one in the depth buffer the color value is written into the framebuffer and the depth value into the depth buffer, otherwise both remain unchanged

FOR each primitiv
  FOR each pixel of primitive at position (x,y) with colour c and depth d
    IF d < depthbuffer(x,y) 
      framebuffer(x,y) = c
      depthbuffer(x,y) = d
    END IF
  END FOR
END FOR

depth resolution

$z$

• The depth buffer has only a certain accuracy. Typically an integer value with 16, 24 or 32 bits of precision
• The interval [-1.0; 1.0] is mapped to [0.0, 1.0] and then to [0, MAX_INT], e.g., [0, 65535] for 16 bits
• The value is rounded to the nearest integer
• Because the "Normalized device coordinates" have already been divided by $p_w$ the rounding errors for objects close to the camera are smaller (and consequently their depth accuracy is higher)
• Therefore, at distant primitives that are close together sometimes the so-called "Z-Fighting" can be observed, which is caused by random inaccuracies in the z-values where at times the one or the other primitive is shown.
• To resolve Z-fighting, it is important to choose the near- and far-plane with care, since these ultimately define the z-range onto which the possible integer depth value are spread
• Therefore, the near- and far-plane should be selected as close together as possible, such that they just enclose the depicted 3D scene