Image Synthesis
Image-based Lighting

Thorsten Thormählen
July 13, 2023
Part 3, Chapter 4

This is the print version of the slides.

Advance slides with the → key or
by clicking on the right border of the slide

Control Keys

→ move to next slide (also Enter or Spacebar).
← move to previous slide.
d enable/disable drawing on slides
p toggles between print and presentation view
CTRL + zoom in
CTRL - zoom out
CTRL 0 reset zoom

Slides can also be advanced by clicking on the left or right border of the slide.

Notation

Type	Font	Examples
Variables (scalars)	italics	$a, b, x, y$
Functions	upright	$\mathrm{f}, \mathrm{g}(x), \mathrm{max}(x)$
Vectors	bold, elements row-wise	$\mathbf{a}, \mathbf{b}= \begin{pmatrix}x\\y\end{pmatrix} = (x, y)^\top,$ $\mathbf{B}=(x, y, z)^\top$
Matrices	Typewriter	$\mathtt{A}, \mathtt{B}= \begin{bmatrix}a & b\\c & d\end{bmatrix}$
Sets	calligraphic	$\mathcal{A}, B=\{a, b\}, b \in \mathcal{B}$
Number systems, Coordinate spaces	double-struck	$\mathbb{N}, \mathbb{Z}, \mathbb{R}^2, \mathbb{R}^3$

Image-based Lighting

For image-based lighting, the emitted radiance $L_i$ of the environment is given by a spherical environment map

$s$

$t$

$0.0$

$1.0$

$0.0$

$\pi$

$0.0$

$-\pi$

$\theta$

$\phi$

Source: : HDR environment map from HDRI Haven, CC0

Environment Map

$y$

$x$

$z$

$\mathbf{p}$

$\phi$

$\theta$

Relationship between texture coordinates $(s, t)$ and spherical coordinates $(\phi, \theta):$
$\begin{align} s &= 0.5 - \frac{\phi}{2\pi} \\ &= 0.5 - \frac{1}{2\pi} \operatorname{arctan2}(y, x)\\ t &= 1.0 - \frac{1}{\pi} \theta \\ &= 1.0 - \frac{1}{\pi} \arccos(z) = \frac{1}{\pi} \arccos(-z) \end{align}$

Example: Envmap

vec2 directionToSphericalEnvmap(vec3 dir) {
  float s = 0.5 - atan(dir.y, dir.x) / (2.0 * PI);
  float t = 1.0 / (PI) * acos(-dir.z);
  return vec2(s, t);
}

Refresher: Importance Sampling

Using the theory of importance sampling, we can also approximate any integral by a sum:
$\int\limits_a^b \mathrm{f}(x) \,dx \approx \frac{1}{N} \sum\limits_{n=1}^{N} \frac{\mathrm{f}(x_n)}{\mathrm{p}(x_n)}$

where $p(x)$ is some arbitrary probability density function (PDF) that must fulfill the condition:

$\int\limits_a^b \mathrm{p}(x) \, dx = 1$
In theory, the best PDF (with the smallest variance) would be
$\mathrm{p}(x) = \frac{\mathrm{f}(x)}{ \int_a^b \mathrm{f}(x)}$

which means the PDF should follow the shape of the function (i.e., the sampling density should be higher if the function values are higher).

Refresher: Inverse Transform Sampling

The probability density function (PDF) $\mathrm{p}(x)$ can be chosen arbitrarily
In order to get the best possible result after few samples, the PDF should follow the shape of the function
But how can we generate sampling positions with a specific PDF?
- Many programming languages provide functions to generate uniform distributed samples
- With inverse transform sampling it is possible to transform uniformly generated random numbers to random numbers with other PDFs

Refresher: Inverse Transform Sampling

Compute the cumulative distribution function (CDF) from the probability density function (PDF):
$\mathrm{P}(x) = \int\limits_{-\infty}^x p(\tilde{x}) \,d\tilde{x}$
Compute the inverse of the CDF $\,\mathrm{P}^{-1}(u)$
Generate uniformly distributed random values $u \in [0, 1]$
Compute random numbers $x$ according to $\mathrm{p}(x)$ using the inverse of the CDF
$x = \mathrm{P}^{-1}_x(u)$

Refresher: Inverse Transform Sampling for a 2D PDF

The inverse transform sampling can also be used for a two-dimensional PDF $p(x, y)$
To do this, first calculate the marginal distribution for one dimension, e.g. for $x$:
$p(x) = \int p(x, y) \,dy$
If $x$ is generated according to this PDF, the conditional probability density function of $y$ is
$p(y | x) = \frac{p(x, y)}{p(x)}$
Now use the known inverse transform sampling for the two PDFs $p(x)$ and $p(y | x)$
Obviously, as an alternative, $p(y)$ and $p(x | y)$ can be used as well

Transformation of Probability Densities

If a multidimensional random variable $\mathbf{x} = (x_1, x_2, \dots, x_n)^\top$ is mapped with a multidimensional bijective (=biunique) function
$\mathbf{y} = \mathrm{f}(\mathbf{x}) = (\mathrm{f}_1(\mathbf{x}), \mathrm{f}_2(\mathbf{x}), \dots, \mathrm{f}_n(\mathbf{x}))^\top$

it follows for the PDF that

$\mathrm{p}(\mathbf{y}) = \frac{1}{|\mathtt{J}_f(\mathbf{x})|} \mathrm{p}(\mathbf{x})$

where $|\mathtt{J}_f(\mathbf{x})|$ is the determinate of the Jacobian matrix

$\mathtt{J}_f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_n}{\partial x_1} & \dots & \frac{\partial f_n}{\partial x_n} \end{bmatrix}$

Importance Sampling of Environment Maps

The environment image is converted into a grayscale image $\mathrm{g}(s, t)$
The image is parameterized in texture coordinates $(s, t)$
The texture coordinates $s$ and $t$ are both in the range [0, 1]
The image has $S \times T$ pixels and square brackets $\mathrm{g}[x, y]$ denote access to the given pixel
The integral over the whole grayscale image is:
$M = \int\limits_0^1 \int\limits\limits_0^1 \mathrm{g}(s, t) \,ds \, dt \approx \sum\limits_{y=0}^{T-1} \sum\limits_{x=0}^{S-1} \mathrm{g}[x, y] \underbrace{\Delta s}_{\frac{1}{S}} \underbrace{\Delta t}_{\frac{1}{T}} = \underbrace{\frac{1}{S\,T}\sum\limits_{y=0}^{T-1} \sum\limits_{x=0}^{S-1} \mathrm{g}[x,y]}_{\tiny \mbox{precomputed mean}}$
Thus, for the 2D probability density function, we get
$\mathrm{p}(s, t) = \frac{\mathrm{g}(s, t)}{M}$

Importance Sampling of Environment Maps

The marginal density with respect to $t$ is calculated by
$\mathrm{p}_t(t) = \int\limits_0^1 \mathrm{p}(s, t) \,ds \approx \mathrm{p}_t[y] = \frac{1}{M} \underbrace{\frac{1}{S}\sum\limits_{x=0}^{S-1} \mathrm{g}[x, y]}_{\tiny \mbox{precomputed row averages}}$
The corresponding cumulative distribution function (CDF) is:
$\mathrm{P}_t(x) = \int\limits_{0}^x \mathrm{p}_t(x) \,dx \approx \frac{1}{T}\sum\limits_{y=0}^Y\mathrm{p}_t[y]$
The CDF cannot be inverted because the following function cannot be solved for the uniformly distributed random variable $u \in [0, 1]$
$u = \frac{1}{T}\sum\limits_{y=0}^Y\mathrm{p}_t[y]$

Therefore, the shader iterates over $y$ until $u$ is reached. The corresponding $Y$ is the image row at which the sample is taken

Importance Sampling of Environment Maps

Now that the image row $Y$ of the sample value was found, the conditional probability distribution $\mathrm{p}_s(s|t)$ must be used to find the image column
Conditional probability density function is:
$\mathrm{p}_s(s|t) = \frac{\mathrm{p}(s, t)}{\mathrm{p}_t(t)} \approx \mathrm{p}_s[x] = \frac{\mathrm{g}[x, Y]}{\frac{1}{S}\sum\limits_{x=0}^{S-1} \mathrm{g}[x, Y]} $
The corresponding cumulative distribution function (CDF) is
$\mathrm{P}_s(x) = \int\limits_{0}^x \mathrm{p}_s(x)\,dx \approx \frac{1}{S}\sum\limits_{x=0}^X\mathrm{p}_s[x]$
This CDF cannot be inverted as well because the following function cannot be solved for the uniformly distributed random variable $v \in [0, 1]$
$v = \frac{1}{S}\sum\limits_{x=0}^X\mathrm{p}_s[x]$

Therefore, the shader iterates over $x$ until $v$ is reached. The corresponding $X$ is the image column at which the sample is taken

Example: Environment Map Sampling in GLSL

Example: IBL Sample
Pre-compute the row averages: BlaubeurenRowAvg

Example: Environment Map Sampling in GLSL

vec2 sampleEnvmap(sampler2D envmap, sampler2D rowAvg, 
        float totalAvg, vec2 random) {

  ivec2 texSize = textureSize(rowAvg, 0);
  float sumY = 0.0;
  float prevSumY = 0.0;
  float ra = 1.0;
  int y = 0;
  for(y = 0; y < texSize.y; y++) {
    ra = texelFetch(rowAvg, ivec2(0, y), 0).r;
    float pt = ra / totalAvg;
    sumY += pt / float(texSize.y);
    if(sumY >= random.y) {
      break;
    }
    prevSumY = sumY;
  }
  float subPixelY = (random.y - prevSumY) / (sumY - prevSumY);

  ivec2 envSize = textureSize(envmap, 0);
  float sumX = 0.0;
  float prevSumX = 0.0;
  int xx = 0;
  for(xx= 0; xx < envSize.x; xx++) {
    vec3 rgb = pow(texelFetch(envmap, ivec2(xx, y), 0).rgb, vec3(2.2));
    float gray = dot(vec3(0.2989, 0.5866, 0.1145), rgb);
    float ps = gray / ra;
    sumX += ps / float(envSize.x);
    if(sumX >= random.x) {
      break;
    }
    prevSumX = sumX;
  }
  float subPixelX = (random.x - prevSumX) / (sumX - prevSumX);

  return vec2((float(xx) + subPixelX) / float(envSize.x), 
              (float(y) + subPixelY) / float(texSize.y));
}

Image-based Lighting

When applied to the rendering equation, the result is:
$\begin{align}L_o(\mathbf{v}) &= L_e(\mathbf{v}) + \int\limits_\Omega \mathrm{f}_r(\mathbf{v}, \mathbf{l})\, L_i(\mathbf{l}) \cos(\theta) \, d\omega\\ &= L_e(\mathbf{v}) + \int\limits_{0}^{2\pi}\, \int\limits_{0}^{\pi/2}\underbrace{ \mathrm{f}_r(\mathbf{v}, \mathbf{l})\,L_i(\mathbf{l}) \cos(\theta) \sin(\theta)}_{\mathrm{f}(\theta_n, \phi_n)} \, d\theta \, d\phi\\ &\approx L_e(\mathbf{v}) + \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}(\theta_n, \phi_n)}{\mathrm{p}(\theta_n, \phi_n)} \end{align}$
Since only $\mathrm{p}(s, t)$ is known and not $\mathrm{p}(\theta, \phi)$, a conversion is required

Image-based Lighting

Relationship between texture coordinates $(s, t)$ and spherical coordinates $(\phi, \theta):$
$\begin{align} \phi &= 2\pi \, (0.5 - s)\\ \theta &= \pi \, (1.0 - t) \end{align} $
The Jacobian matrix is:
$\mathtt{J} = \begin{bmatrix} \frac{\partial \phi}{\partial s} & \frac{\partial \phi}{\partial t} \\ \frac{\partial\theta}{\partial s} & \frac{\partial \theta}{\partial t} \end{bmatrix} = \begin{bmatrix} -2\pi & 0\\ 0 & -\pi \end{bmatrix} $
Thus, the determinant of the Jacobian matrix is: $|\mathtt{J}| = 2 \pi^2$
Rendering equation:
$\begin{align}L_o(\mathbf{v}) &= L_e(\mathbf{v}) + \int\limits_{0}^{2\pi}\, \int\limits_{0}^{\pi/2}\underbrace{ \mathrm{f}_r(\mathbf{v}, \mathbf{l})\,L_i(\mathbf{l}) \cos(\theta) \sin(\theta)}_{\mathrm{f}(\theta_n, \phi_n)} \, d\theta \, d\phi\\ &\approx L_e(\mathbf{v}) + \frac{1}{N} \sum_{n=1}^{N} \frac{\mathrm{f}(\theta_n, \phi_n)}{\mathrm{p}(s_n, t_n)} 2 \pi^2 \end{align}$

Example: Image-based Lighting

Example: IBL

Are there any questions?

Please notify me by e-mail if you have questions, suggestions for improvement, or found typos: Contact

Image SynthesisImage-based Lighting

Control Keys

Notation

Image-based Lighting

Environment Map

Refresher: Importance Sampling

Refresher: Inverse Transform Sampling

Refresher: Inverse Transform Sampling

Refresher: Inverse Transform Sampling for a 2D PDF

Transformation of Probability Densities

Importance Sampling of Environment Maps

Importance Sampling of Environment Maps

Importance Sampling of Environment Maps

Example: Environment Map Sampling in GLSL

Example: Environment Map Sampling in GLSL

Image-based Lighting

Image-based Lighting

Example: Image-based Lighting

Are there any questions?

Image Synthesis
Image-based Lighting