Image Synthesis
Image-based Lighting

• For image-based lighting, the emitted radiance $L_i$ of the environment is given by a spherical environment map

$s$

$t$

$0.0$

$1.0$

$1.0$

$0.0$

$0.0$

$\pi$

$\pi$

$0.0$

$-\pi$

$\theta$

$\phi$

$y$

$x$

$z$

$\mathbf{p}$

$\phi$

$\theta$

• Relationship between texture coordinates $(s, t)$ and spherical coordinates $(\phi, \theta):$
  $\begin{align} s &= 0.5 - \frac{\phi}{2\pi} \\ &= 0.5 - \frac{1}{2\pi} \operatorname{arctan2}(y, x)\\ t &= 1.0 - \frac{1}{\pi} \theta \\ &= 1.0 - \frac{1}{\pi} \arccos(z) = \frac{1}{\pi} \arccos(-z) \end{align}$
• Example: Envmap

  vec2 directionToSphericalEnvmap(vec3 dir) {
    float s = 0.5 - atan(dir.y, dir.x) / (2.0 * PI);
    float t = 1.0 / (PI) * acos(-dir.z);
    return vec2(s, t);
  }

• Using the theory of importance sampling, we can also approximate any integral by a sum:
  $\int\limits_a^b \mathrm{f}(x) \,dx \approx \frac{1}{N} \sum\limits_{n=1}^{N} \frac{\mathrm{f}(x_n)}{\mathrm{p}(x_n)}$
  where $p(x)$ is some arbitrary probability density function (PDF) that must fulfill the condition:
  $\int\limits_a^b \mathrm{p}(x) \, dx = 1$
• In theory, the best PDF (with the smallest variance) would be
  $\mathrm{p}(x) = \frac{\mathrm{f}(x)}{ \int_a^b \mathrm{f}(x)}$
  which means the PDF should follow the shape of the function (i.e., the sampling density should be higher if the function values are higher).

• Rendering equation (without the emissive term):
  $\begin{align}L_o(\mathbf{v}) &= \int\limits_{\Omega_h} \mathrm{f}_r(\mathbf{v}, \mathbf{l})\, \, L_i(\mathbf{l}) \cos(\check{\theta}) \, d\omega\\ & = \int\limits_{0}^{2\pi}\, \int\limits_{0}^{\pi/2} \mathrm{f}_r(\mathbf{v}, \mathbf{l})\, \, L_i(\mathbf{l}) \cos(\check{\theta}) \sin(\check{\theta})\, d\check{\theta} \, d\check{\phi}\\ & = \int\limits_{0}^{2\pi}\, \int\limits_{0}^{\pi} \underbrace{\mathrm{f}_r(\mathbf{v}, \mathbf{l})\, \, L_i(\mathbf{l}) \, (\cos(\check{\theta}))_{\small +} \sin(\theta)}_{\mathrm{f}( \phi, \theta)} \, d\theta \, d\phi\\ &\approx \ \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}( \phi_n, \theta_n)}{\mathrm{p}( \phi_n, \theta_n)}\\ &= \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}_r(\mathbf{v}, \mathbf{l})\,\,L_i(\mathbf{l}) \, (\cos(\check{\theta}))_{\small +} \sin(\theta_n)}{\mathrm{p}( \phi_n, \theta_n)}\end{align}$

$x$

$y$

$p(x, y)$

$p_x(x)$

$p_y(y)$

$p(x | y)$

• The inverse transform sampling can also be used for a two-dimensional PDF $p(x, y)$

1. Given a 2D PDF  $p(x, y)$,
   calculate the marginal PDF for one dimension, e.g. for $y\,$:
   $\mathrm{p}_y(y) = \int \mathrm{p}(x, y) \,dx$
2. Compute the conditional PDF for the other dimension:
   $\mathrm{p}(x | y) = \frac{\mathrm{p}(x, y)}{\mathrm{p}_y(y)}$
3. Use the known inverse transform sampling method for the two 1D PDFs $\mathrm{p}_y(y)$ and $\mathrm{p}(x | y)$

$\begin{align}\phi &= 2\pi \,(0.5 - s) \\ \theta &= \pi \, (1.0 - t) \end{align}$

$s$

$t$

$0.0$

$0.5$

$1.0$

$1.0$

$0.0$

$0.0$

$\pi$

$\pi$

$0.0$

$-\pi$

$\theta$

$\phi$

• If a multidimensional random variable $\mathbf{x} = (x_1, x_2, \dots, x_n)^\top$ is mapped with a multidimensional bijective (=biunique) function
  $\mathbf{y} = \mathrm{f}(\mathbf{x}) = (\mathrm{f}_1(\mathbf{x}), \mathrm{f}_2(\mathbf{x}), \dots, \mathrm{f}_n(\mathbf{x}))^\top$
  it follows for the PDF that
  $\mathrm{p}(\mathbf{y}) = \frac{1}{|\mathtt{J}_f(\mathbf{x})|} \mathrm{p}(\mathbf{x})$
  where $|\mathtt{J}_f(\mathbf{x})|$ is the determinate of the Jacobian matrix
  $\mathtt{J}_f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_n}{\partial x_1} & \dots & \frac{\partial f_n}{\partial x_n} \end{bmatrix}$

• Relationship between texture coordinates $(s, t)$ and spherical coordinates $(\phi, \theta)$:
  $\begin{align}\phi &= 2\pi \,(0.5 - s) \\ \theta &= \pi \, (1.0 - t) \end{align}$
• The Jacobian matrix is:
  $\mathtt{J} = \begin{bmatrix} \frac{\partial \phi}{\partial s} & \frac{\partial \phi}{\partial t} \\ \frac{\partial\theta}{\partial s} & \frac{\partial \theta}{\partial t} \end{bmatrix} = \begin{bmatrix} -2\pi & 0\\ 0 & -\pi \end{bmatrix} $
• Thus, the determinant of the Jacobian matrix is: $|\mathtt{J}| = 2 \pi^2$
• Rendering equation:
  $\begin{align}L_o(\mathbf{v}) &= \int\limits_{0}^{2\pi}\, \int\limits_{0}^{\pi}\underbrace{ \mathrm{f}_r(\mathbf{v}, \mathbf{l})\,L_i(\mathbf{l}) \, (\cos(\check{\theta}))_{\small +} \sin(\theta)}_{\mathrm{f}( \phi, \theta)} \, d\theta \, d\phi\\ &\approx \ \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}( \phi_n, \theta_n)}{\mathrm{p}( \phi_n, \theta_n)} = \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}( \phi_n, \theta_n)}{\mathrm{p}(s_n, t_n)} 2 \pi^2 \end{align}$

• The environment image is converted into a grayscale image $\mathrm{g}(s, t)$
• The image is parameterized in texture coordinates $(s, t)$
• The image has $S \times T$ pixels and square brackets $\mathrm{g}[x, y]$ denote access to the given pixel
• The integral over the whole grayscale image is:
  $\int\limits_0^1 \int\limits\limits_0^1 \mathrm{g}(s, t) \,ds \, dt \approx \sum\limits_{y=0}^{T-1} \sum\limits_{x=0}^{S-1} \mathrm{g}[x, y] \underbrace{\Delta s}_{\frac{1}{S}} \underbrace{\Delta t}_{\frac{1}{T}} = \underbrace{\frac{1}{S\,T}\sum\limits_{y=0}^{T-1} \sum\limits_{x=0}^{S-1} \mathrm{g}[x,y]}_{\tiny \mbox{image average}} = A$
• Thus, for the 2D probability density function, we get
  $\mathrm{p}(s, t) = \frac{\mathrm{g}(s, t)}{A}$
  with
  $A = \underbrace{\frac{1}{S\,T}\sum\limits_{y=0}^{T-1} \sum\limits_{x=0}^{S-1} \mathrm{g}[x,y]}_{\tiny \mbox{image average}} = \frac{1}{T}\sum\limits_{y=0}^{T-1} \underbrace{\frac{1}{S} \sum\limits_{x=0}^{S-1} \mathrm{g}[x,y]}_{\tiny \mbox{precomputed row averages}} $

• The marginal density with respect to $t$ is calculated by
  $\mathrm{p}_t(t) = \int\limits_0^1 \mathrm{p}(s, t) \,ds = \int\limits_0^1 \frac{\mathrm{g}(s, t)}{A} \,ds \approx \mathrm{p}_t[y] = \frac{1}{A} \underbrace{\frac{1}{S}\sum\limits_{x=0}^{S-1} \mathrm{g}[x, y]}_{\tiny \mbox{precomputed row averages}}$
• The corresponding cumulative distribution function (CDF) is:
  $\mathrm{P}_t(t) = \int\limits_{0}^t \mathrm{p}_t(\tilde{t}) \,d\tilde{t} \approx \mathrm{P}_t[y] = \frac{1}{T}\sum\limits_{\tilde{y}=0}^y\mathrm{p}_t[\tilde{y}]$
• The CDF cannot be inverted because the following function cannot be analytically solved for the uniformly distributed random variable $u \in [0, 1]$
  $u = \mathrm{P}_t[y] = \frac{1}{T}\sum\limits_{\tilde{y}=0}^y\mathrm{p}_t[\tilde{y}]$
  Therefore, the shader performs the addition over $\tilde{y}$ until $u$ is reached. The corresponding $y$ is the image row $Y$ at which the sample is taken

• Now that the image row $Y$ of the sample value was found, the conditional PDF $\mathrm{p}_s(s|t)$ must be used to find the image column
• Conditional probability density function is:
  $\mathrm{p}_s(s|t) = \frac{\mathrm{p}(s, t)}{\mathrm{p}_t(t)} \approx \mathrm{p}_s[x] = \frac{\frac{\mathrm{g}[x, Y]}{A}}{\frac{1}{A}\frac{1}{S}\sum\limits_{x=0}^{S-1} \mathrm{g}[x, Y]} = \frac{\mathrm{g}[x, Y]}{\frac{1}{S}\sum\limits_{x=0}^{S-1} \mathrm{g}[x, Y]} $
• The corresponding cumulative distribution function (CDF) is
  $\mathrm{P}_s(s) = \int\limits_{0}^s \mathrm{p}_s(\tilde{s})\,d\tilde{s} \approx \mathrm{P}_s[x] = \frac{1}{S}\sum\limits_{\tilde{x}=0}^x\mathrm{p}_s[\tilde{x}]$
• The CDF cannot be inverted because the following function cannot be analytically solved for the uniformly distributed random variable $v \in [0, 1]$
  $v = \mathrm{P}_s[x] = \frac{1}{S}\sum\limits_{\tilde{x}=0}^x\mathrm{p}_s[\tilde{x}]$
  Therefore, the shader performs the addition over $\tilde{x}$ until $v$ is reached. The corresponding $x$ is the image column $X$ at which the sample is taken