Spatially Varying IBL Using Light Probe Sequences

Department of Science and Technology

Institutionen för teknik och naturvetenskap

Linköping University Linköpings Universitet

SE-601 74 Norrköping, Sweden

601 74 Norrköping

LiU-ITN-TEK-A--09/011--SE

Spatially Varying IBL Using

Light Probe Sequences

Richard Khoury

LiU-ITN-TEK-A--09/011--SE

Spatially Varying IBL Using

Light Probe Sequences

Examensarbete utfört i medieteknik

vid Tekniska Högskolan vid

Linköpings universitet

Richard Khoury

Handledare Jonas Unger

Examinator Stefan Gustavson

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –

under en längre tid från publiceringsdatum under förutsättning att inga

extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,

skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för

ickekommersiell forskning och för undervisning. Överföring av upphovsrätten

vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av

dokumentet kräver upphovsmannens medgivande. För att garantera äktheten,

säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ

art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i

den omfattning som god sed kräver vid användning av dokumentet på ovan

beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan

form eller i sådant sammanhang som är kränkande för upphovsmannens litterära

eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se

förlagets hemsida

http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet - or its possible

replacement - for a considerable time from the date of publication barring

exceptional circumstances.

The online availability of the document implies a permanent permission for

anyone to read, to download, to print out single copies for your own use and to

use it unchanged for any non-commercial research and educational purpose.

Subsequent transfers of copyright cannot revoke this permission. All other uses

of the document are conditional on the consent of the copyright owner. The

publisher has taken technical and administrative measures to assure authenticity,

security and accessibility.

According to intellectual property law the author has the right to be

mentioned when his/her work is accessed as described above and to be protected

against infringement.

For additional information about the Linköping University Electronic Press

and its procedures for publication and for assurance of document integrity,

please refer to its WWW home page:

http://www.ep.liu.se/

Abstract

Visual media, such as film and computer games, often require the need for the realistic rendering of synthetic objects. Image Based Lighting (IBL) techniques provide methods for applying measured real-world lighting to synthetic objects, making them appear believable within their environment. Given this ability, IBL techniques have drawn interest within many industries involved in visual effects, however its adoption has been mostly confined to implementations of its original method.

Traditional IBL, as it is now known, only requires the measurement of light at one position in space to provide the data for illuminating all synthetic objects in the scene. This single requirement places large constraints on the complexity of illumination within a scene by assuming there are negligible changes in lighting within the extent of the local environment. Due to this, lighting features such as shadows, that exhibit a spatial frequency greater than zero, cannot be represented within this limited model.

Modern research into IBL techniques aim to resolve this problem by presenting methods to capture, process, and render spatially varying illumination. This thesis builds upon recent research into densely sampled light probe sequences and con-siders its use in a production environment. Its objective is to present set of tools for processing and rendering this data for use with the commercial software pack-ages Maya, a modelling and animation application, and mental ray, a high-fidelity renderer.

Acknowledgments

I would like to thank everyone in the VITA department at Linköping University for providing an amazing environment for graphical inspiration. The pleasure I have had studying within this department has made me wish I could study there forever.

A special thanks goes to those whom I worked closely with during my the-sis term, especially Stefan Gustavason for his amazing insights and breadth of knowledge, Jonas Unger for his technical genius, and Per Larsson for his practical know-how.

And finally, I’d like to thank my lovely fiancé, Alison, who has been the most remarkable support during my two years in Sweden. I♥U

Contents

1 Introduction 3

2 Background 5

2.1 The Rendering Equation . . . 5

2.2 Traditional Image Based Lighting . . . 7

2.2.1 The General Method . . . 7

2.2.2 High Dynamic Range Imaging . . . 8

2.2.3 Rendering . . . 9

2.2.4 Limitations . . . 10

2.3 Modern Image Based Lighting techniques . . . 10

2.3.1 Improving Capture Time . . . 10

2.3.2 Spatially Varying Data . . . 11

2.3.3 Improving Rendering Efficiency . . . 14

3 Light Probe Sequences 15 3.1 Off-line Rendering . . . 15

3.1.1 Traditional IBL . . . 15

3.1.2 The Plenoptic Function . . . 16

3.1.3 Nearest Neighbour Sampling . . . 17

3.1.4 Ray Projection Sampling . . . 18

3.1.5 Single-Viewpoint Reprojection . . . 20

3.2 Real-time Diffuse Rendering . . . 20

3.2.1 Spherical Harmonics . . . 21

3.2.2 Down-Sampling . . . 27

4 Implementation and Usage 31 4.1 Ray Projection Algorithm . . . 31

4.2 Processing . . . 31

4.2.1 Spherical Harmonic Representation . . . 32

4.2.2 Down-Sampled Representation . . . 32

4.3 Maya Hardware Shader . . . 34

4.3.1 Limitations . . . 37

4.4 Mental ray Shaders . . . 37

4.4.1 Environment Shader . . . 37

4.4.2 Area Light Shader . . . 38 5

4.4.3 Issues . . . 38

5 Conclusion 41 5.1 Future Work . . . 41

A Remapping equations 45 B GLSL Code 47 B.1 Spherical Harmonic Rendering for Light Probe Sequences . . . 47

B.1.1 Vertex Shader . . . 47

B.1.2 Fragment Shader . . . 47

B.2 Down-sample Rendering for Light Probe Sequences . . . 49

B.2.1 Vertex Shader . . . 49

B.2.2 Fragment Shader . . . 49 C Ray Projection Equation 51

List of Figures

1.1 Scene composed of real and synthetic objects. The corresponding light probe image, bottom left, shows the radiance information used to illuminate the synthetic objects. Images taken from [1]. . . 3 2.1 Ray-tracing. A ray is cast from the eye/camera, through a

pixel-plane (the final image), and into the scene. The ray proceeds to bounce around the scene and returns the accumulation of colours/il-lumination found. . . 6 2.2 Debevec’s general method. Image taken from [1]. . . 7 2.3 The directions observed on a mirrored sphere from a parallel viewing

plane. . . 8 2.4 Photos of a mirror sphere taken at various exposures for generating

a HDR radiance map. Taken from [1]. . . 9 2.5 The Real Time Light Probe used in [2, 3, 4] . . . 11 2.6 Two capture devices presented by Unger et al. [5]. . . 12 2.7 1D spatial variance rendered through IBL techniques discussed

fur-ther in this thesis, and originally presented by Unger et al. [3] . . . 13 2.8 3D spatial variance rendered through IBL techniques discussed by

Unger et al. [4] . . . 13 3.1 Traditional IBL applied to a scene using two neighbouring light

probe samples. Image taken from [3]. . . 16 3.2 Nearest neighbour sampling with a 1D light probe sequence. The

point p is projected to the nearest point on the sample path. The nearest sample(s) are then used as normal radiance maps, indexed with the direction d. . . . 17 3.3 A rendering using nearest neighbour sampling with a 1D light probe

sequence. The vertical light bands are projected at an oblique angle to the sample path, though this method makes it appear as though they are orthogonal. This image also shows the noise that is gen-erated by undersampling the environment maps, even though over 16000 samples were chosen per environment map and some hours were taken for it to render. . . 18 3.4 Ray projection sampling with a 1D light probe sequence. The

di-rection d is projected to the sample path from point p, indicating the better samples to choose. . . 19 3.5 Ray projection sampling with a 1D light probe sequence. This

method displays better results for points diverging from the sample path. This image also shows the noise that is generated by under-sampling the environment maps, even though over 16000 samples were chosen per enviroment map and some hours were taken for it to render. . . 19

3.6 The single-viewpoint reprojection method. z0(green) is the position

of the mirror sphere at each frame, while the red points, z, are the projection points of each incident ray back to the sample path. R is the radius of the mirror sphere and r is the radial distance from the sample path to the ray’s intersection point. Image taken from [3]. 20 3.7 A scene lit bit a grill-covered spot light. The diffuse materials in

the bottom row reveal more about the spatially varying light than

the specular materials in the top row. . . 21

3.8 The first four bands of the spherical harmonic basis functions. The colour green indicates positive values, while red indicates negative. This image was taken from [6]. . . 23

3.9 An example of function projection (decomposition into basis func-tion coefficients) and the subsequent reconstrucfunc-tion. This image was taken from [7]. . . 24

3.10 Real-time rendering of a 1D light probe sequence using the spherical harmonic method. The same parallax errors occur as the nearest neighbour method presented in Section 3.1.3. . . 26

3.11 Rendering of a sphere showing the difference in regions between the spherical sampling schemes. The positive directions for the x(red), y(green), and z(blue) axes are shown. The green dots represent the primary sample direction for that region. Figures 3.11(a) and 3.11(b) show the non-uniform regions produced by uniform sam-pling in polar coordinate space. Figures 3.11(c) and 3.11(d) show the more uniform regions produced by using the vertices of an Icosa-hedron and its first subdivision respectively. . . 28

3.12 Real-time rendering of a 1D light probe sequence using the down-sampling method. The same four samples sets shown in Figure 3.11 are used. . . 29

4.1 The binary format of the spherical harmonic data processed from the light probe sequence, presented in terms of its use as a texture on the GPU. . . 33

4.2 The binary format of the down-sampled data processed from the light probe sequence, presented in terms of its use as textures on the GPU. . . 34

4.3 The hardware shader attribute editor. . . 36

4.4 The hardware shader attribute connections. . . 36

4.5 A screenshot of the Maya hardware shader in action. . . 37

4.6 Two scenes showing the scene rendered improperly when shadow tracing was enabled. . . 38

4.7 The skewing of spatial variance. This is a top-view of the xz-plane, the sample path (z-axis) located toward the middle of the image, as seen in 4.7(b). . . 39

Chapter 1

Introduction

The lighting within a scene is often the most important step in making computer graphics appear believable. In media applications where the aim is the combine both real and computer graphics content, the need for consistent and believable lighting is even more important as our visual perception is finely tuned to real-world phenomena and any irregularities are easily noticed.

Figure 1.1. Scene composed of real and synthetic objects. The corresponding light

probe image, bottom left, shows the radiance information used to illuminate the synthetic objects. Images taken from [1].

Modern techniques for rendering synthetic images mainly rely on the creation of synthetic light sources within the scene to provide the primary means of illumi-nation. While this can make purely synthetic scenes appear realistic, it often fails to adequately reproduce all the nuances of illumination within a real-world scene. This downfall becomes more obvious when real and computer generated content are composited together, a process used regularly in film and television, architec-tural renderings, augmented reality applications, and still media such as product brochures. This broad industrial demand, and the desire for more photo-realistic images, has largely driven the research in the field of Image Based Lighting (IBL);

a technique that relies on the measurement of real-world lighting (through image capture) to provide the illumination for a synthetic scene. An example of this can be seen in Figure 1.1 where the objects in the center of the image are all computer generated and illuminated by a single high dynamic range (HDR) image called a light a probe.

Though IBL techniques have been around for some time, they have still not managed to pervade the modern production pipeline. Many commercial software packages include the ability to perform traditional IBL (discussed in the following chapter), but the more modern research still remains as proof-of-concept software written mostly for research purposes.

This thesis will discuss some of the more modern research into IBL techniques, focusing specifically on research into 1-dimensional, densely sampled light probe sequences. It aims to present a set of tools for working with this data using two industry-level software packages: Maya and mental ray.

The following chapter will present a background into IBL techniques and touch on the more modern research into this field.

Chapter 3 will present, in more detail, the techniques for working with sequen-tial light probe data in both high-fidelity and real-time rendering applications.

Following from this, chapter 4 will detail the development of the tools, and how the relevant techniques are implemented within the Maya and mental ray frameworks.

Finally, chapter 5 will summarise the techniques and present an overview of how these tools can improve in future developments.

Chapter 2

Background

Rendering synthetic objects in a realistic way is a non-trivial task. The illumi-nation within the scene can often be quite complex, giving rise to many methods that attempt to solve its interaction with the environment. All these techniques provide some form of approximation of the illumination within a scene, with each having their own benefits and drawbacks. At the root of all of these methods is a mathematical model of light within the scene, for which modern rendering techniques are able to better approximate with extremely realistic results.

This chapter aims to introduce the reader to the problems faced by realistic rendering methods, and the direction IBL techniques take in attempting to solve them.

2.1

The Rendering Equation

Rendering methods such as ray-tracing and radiosity attempting to simulate the physical characteristics of light energy (radiant flux) within a closed system; the scene. These global illumination methods are used for high fidelity graphics as they better approximate the system of light energy that exists within a scene. Kajiya [8] were the first to describe the generalised form for calculating this system, referred to as the rendering equation:

B(x, ~ωo) = Lε(x, ~ωo) + Z Ωh(~n) L(x, ~ωi)ρ(x, ~ωi→ ~ωo)( ~ωi· ~n)d ~ωi (2.1) Where: 5

B(x, ~ωo) is the radiance at x in the outgoing direction ~ωo,

L(x, ~ωi) is the radiance at x from the incoming direction ~ωi,

Lε(x, ~ωo) is the self emission at x in the outgoing direction ~ωo, Ωh(~n) is the hemisphere centered on the normal ~n at point x,

ρ(x, ~ωi→ ~ωo) is the SBRDF (Spatially varying Bi-directional Reflectance Dis-tribution Function) at the point x,

( ~ωi· ~n) is the cosine weighting of the radiance based on the angle between the incoming direction and the normal.

This equation describes the total exitance radiance along a ray originating from a point, x, in the direction ~ωo, as an emission component plus the weighted sum of all radiance entering that point from the hemisphere that exits in the same outgoing direction. Solving this equation for each point on all surfaces becomes a recursive problem that cannot be analytically solved by a computer, so clever approximations must be applied to achieve a desirable result in a more practical time-frame (see [9] and [10]). The core difficulty in determining correct illumi-nation is knowing how light is interacting between all objects in the scene. In practice, this is generally done by casting rays through the viewing plane, finding which object it intersects, and determining how that point of intersection is illumi-nated by the whole scene by casting more secondary rays. Figure 2.1 demonstrates the ray-tracing method.

Figure 2.1. Ray-tracing. A ray is cast from the eye/camera, through a pixel-plane

(the final image), and into the scene. The ray proceeds to bounce around the scene and returns the accumulation of colours/illumination found.

This algorithm has the potential to be extremely slow in order to produce an adequate image. For applications that require a virtual object be placed in a real scene, this problem becomes even harder to solve by traditional means. A naïve

2.2 Traditional Image Based Lighting 7

solution using the methods described above would require the artist to produce accurate models for the materials, lighting, and geometry within the scene in order for the virtual objects to render correctly. But given a few constraints on the scene this can task can become a lot easier to solve.

2.2

Traditional Image Based Lighting

In 1998, Debevec [1] described a simple method for effectively rendering synthetic objects into a real scene. This paper extended upon ideas presented five years earlier by Fournier et al. [11] in which a synthetic object could be rendered into a real scene given various assumptions on the scene’s geometry and the common viewing parameters. Whilst the approach by Fournier et al. did not produce a seamless composition, they did show it was possible to render virtual objects within a real environment. The method given by Debevec, however, did not require the same assumptions that Fournier et al. imposed, and still produced visually pleasing results (see Figure 1.1).

2.2.1

The General Method

In order for traditional image based lighting to work, the scene must satisfy the general method described by Debevec in Figure 2.2. The most important notion within this diagram is that the distant scene is considered as a component that is affecting all objects within the local scene, but is not in turn being re-affected by any of those objects. This constraint is the basis for how the method works.

Figure 2.2. Debevec’s general method. Image taken from [1].

Given this scene constraint, it becomes possible to record the incident light at a point within the local scene and use it as a global illumination measurement

for illuminating all virtual objects close to that point. The tool Debevec used to capture the illumination from almost all directions is called a light probe and is ideally a completely reflective sphere. The probe represents all incident light onto a point in the local scene corresponding to the sphere’s centre point1. Figure 2.3 shows the principle of a light probe, and how the perfect observer2can see almost all of the scene from the range of directions offered by the reflections in the sphere, except those parts for which the sphere directly occludes.

0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 0000000000000 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 1111111111111 Dead Zone 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111

Figure 2.3. The directions observed on a mirrored sphere from a parallel viewing plane.

What Figure 2.3 also attempts to show is the non-linearity of the viewing angles as the viewing rays draw further away from the centre of the sphere. This can be easily observed between the blue and the red rays where the same small vertical difference from the view plane spans a larger angular range in the light probe. Because of this, care must be taken to maximise the resolution and quality of the light probe capture so that sufficient data exists towards the sphere’s edge.

2.2.2

High Dynamic Range Imaging

Since a light probe image captures the radiant flux from the global scene, the lighting information must be in a format that represents radiance values. Low dynamic range (LDR) images, as captured by commodity digital or film-based cameras, cannot measure all intensities of light that the scene contains within one exposure. This can be seen in over or under saturated images where light intensity information is clamped to values that the camera sensor or film can handle.

1_{This is a false assumption as mentioned in [3], but is a sufficient representation for traditional}

methods.

2_{The perfect observer is infinitely far away from the sphere such that all view rays are parallel.}

This would never happen in reality but allows the maths and conceptual understanding to remain more straight-forward.

2.2 Traditional Image Based Lighting 9

Debevec and Malik [12] describe a technique for recovering high dynamic range (HDR) radiance values from a set of LDR images. These LDR images must be taken with varying exposures so that each pixel is properly saturated in at least one of the images in the set. A knowledge of the film or sensor’s response to light, known as the response curve, and the exact exposure of each image is then used to generate a HDR image containing radiance values. Figure 2.4 shows a light probe captured with three different exposures, allowing the extraction of all radiance values including the low and high intensity areas within the scene.

Figure 2.4. Photos of a mirror sphere taken at various exposures for generating a HDR

radiance map. Taken from [1].

2.2.3

Rendering

Now that a HDR radiance map is obtained it can be used as an environment map for rendering an object within the local scene. Each pixel that covers the mirror sphere corresponds to the flux travelling along a particular ray direction. So for each desired direction we wish to calculate the light from, we can index into the radiance map at a particular pixel and retrieve the information there. The equations for mapping between image coordinates of a light probe and various other coordinate systems are presented in Appendix A.

Rendering mirror-like materials realistically becomes as simple as doing one lookup per intersection into the radiance map; much easier than computing the full global illumination within the scene. Diffuse materials are a little more intensive as the hemisphere around the surface normal of the object must be sampled. To improve the performance of rendering diffuse materials a diffuse convolution map can be preprocessed to allow very quick mapping from surface normal to a diffuse value. The same can be done for various other material types such as glossy surfaces. More advanced techniques for rendering diffuse materials are discussed in Section 3.2.

For a local scene that contains multiple virtual objects, a further global illu-mination step must be used to calculate object to object interactions for which traditional IBL cannot directly solve. Whilst the speed of rendering is now hin-dered by a global illumination step, the larger benefits of the light probe method are still not lost. Extensions for improving global illumination performance are

discussed later in this chapter.

2.2.4

Limitations

Two important limitations to traditional IBL are discussed below. While they are not the only ones that exist, they represent the most important factors for which the modern research this thesis is based on has chosen to solve.

Static scene The HDR reconstruction from LDR images places a lot of impor-tance on the scene being static so that no artefacts like ghosting will be present in the resulting radiance map. This problem is not limited to light probe capture, but rather all forms of HDR reconstruction using LDR image sets.

Spatial variance For most scenes there is going to be some spatial variance which one light probe alone cannot represent. If the object is expected to move within the scene or is static but on the border of spatially varying phenomena such as shadows, then multiple light probes must be taken to allow the virtual object to render correctly within the scene.

2.3

Modern Image Based Lighting techniques

In an environment such as a film set there is often a very limited time given to visual effects crew wishing to capture the lighting information necessary for post-production. Currently there is no commercially available device to conveniently capture many light field samples within a scene. Firstly, the camera’s position and orientation must be measured or made able to be tracked, normally by placing markers within the scene and using image-based tracking. The measuring and setup must be accurate and thus it requires a reasonable amount of time; not very well suited for a time pressured environment. Once the scene has been prepared the capture can take place. The acquisition is often done using commodity LDR digital cameras, so the many photos required to make one HDR radiance map take some time to capture. This reasserts the requirement that the scene remain static while the varying exposures are acquired.

In the scenario above it is easy to see that, given the current methods, not many light field samples can be acquired within the potentially short time-frame allocated. Solutions for improving capture time and complexity are necessary in order for better image-based lighting methods to fit inconspicuously into the film set environment.

2.3.1

Improving Capture Time

The static scene limitation is soon becoming a thing of the past as more ad-vanced camera technology is allowing the direct capture of HDR images, even at video frame rates. Beyond solving the static scene limitation, a fast capture de-vice becomes necessary for the acquisition of large data sets representing various illumination properties, such as high frequency spatial variations.

2.3 Modern Image Based Lighting techniques 11

Figure 2.5. The Real Time Light Probe used in [2, 3, 4]

Unger et al. [2, 3, 4] presents the design and use of a Real Time Light Probe, a custom three-camera rig that quickly captures HDR light probe sequences. Each camera captures monochrome HDR video at 512x512 resolution, 25fps, and at an effective dynamic range of 10,000,000:1. As shown in Figure 2.5, red, green, and blue colour filters are attached to the lenses, and each unit is aimed toward the centre of the light probe residing at the opposite end of the rig. After capture, the three colour channels must be aligned since they both have different views of the mirror sphere.

2.3.2

Spatially Varying Data

Capturing spatially varying data is another point of interest to those wishing to render synthetic objects within a real scene. Most real world scenes contain light that varies in the spatial domain, sometimes quite rapidly, and it is important to have the tools to capture these changes. In 2003, Unger et al. [5] experimented with two ways to capture incident light fields (ILF) on a plane: firstly using a mirror-sphere/light-probe array (2.6(a)), and secondly using a high-fidelity cap-turing device (2.6(b)).

Whilst mirror spheres are often a great way to capture a light field sample, the mirror sphere array suffers from two main problems. The first issue is resolution. Since the whole array is captured using one image, each light field sample only constitutes a smaller fraction of that image meaning that the directional resolution is quite limited. The second issue is the interreflection between adjacent spheres on the plane. This means that the parts of the captured data within each light field sample that are not directly reflecting the environment need to be discounted. Fortunately for their purposes, the unoccluded field of view of 159.2 degrees was sufficient enough given the requirement for capturing a hemisphere of light at each sample point.

The high fidelity capture device in [5] improves on both issues above at the expense of a greater capture time. The spatial resolution of the ILF is limited only by the motor’s step size and rig dimensions, and the directional resolution is much higher since each light field sample is a separate high-resolution image through a

(a) Mirror Sphere Array

(b) High Fidelity Capture Device

2.3 Modern Image Based Lighting techniques 13

185 degree fish-eye lens.

The two devices mentioned above only allow for spatially varying ILF capture within a predefined plane. Using the HDR video camera mentioned previously (shown in Figure 2.5), Unger et al. improved on this by allowing the capture of light field samples along a path[2, 3] and in a volume[4]. In [3], the tracking of light probe samples was done using measured markers on the camera rig, along with two external video cameras and commercial video tracking software. There is also men-tion of the possibility that image-based tracking methods could be used in place of the external video-based tracking. Since the rig in [4] is computer controlled, the tracking can be easily captured along with each light probe acquisition.

Figure 2.7. 1D spatial variance rendered through IBL techniques discussed further in

this thesis, and originally presented by Unger et al. [3]

Figure 2.7 shows the effect of rendering synthetic objects using densely sampled spatially varying data along a straight line path[3]. Figure 2.8 shows the result of rendering four synthetic objects using ILF data captured in a volume[4]. This image in particular shows some of the latest work in spatially varying data capture and rendering.

Figure 2.8. 3D spatial variance rendered through IBL techniques discussed by Unger

2.3.3

Improving Rendering Efficiency

As mentioned previously, calculating global illumination can be an extremely time consuming process. Extracting the position of a light source is a vital mechanism for improving the efficiency and accuracy of the global illumination step as it acts as a kind of importance sampling. Methods such as the median cut algorithm[13] are used to determine the positions of major light sources within a single HDR radiance map and substitute in appropriately valued point-light sources for which rendering is quick and noise doesn’t exist. For large data sets where it is impractical to look up individual radiance maps, new techniques had to be developed. Unger et al. [4] are able to generate an approximation for the global scene bounding box and extract the various light sources that may be contained within it. This reduces the overall amount of data that the ILFs represent and not only provides the ability to improve rendering times, it also allows for an extremely flexible post-production environment; allowing light sources to be altered or removed altogether.

Chapter 3

Light Probe Sequences

The pipeline for a production describes the sequence of events necessary for its completion. For visual effects, the pipeline is comprised of several stages, which include the modeling of objects, the lighting and animation of objects within a sequence, the rendering, and the compositing. Working with light probe sequences requires the creation of specialised tools that can fit into this pipeline. The tools presented in this thesis are structured into two main components: a real-time viewport preview, and an off-line shader for high-fidelity rendering. These are designed to fit into the production pipeline in the lighting/animation and rendering stages respectively.

This chapter will examine the techniques required by these tools to work with 1D light probe sequences. The following methods will be evaluated with the same dataset used by Unger et al. [3]. For simplicity, each light probe sample was considered uniformly spaced. This scene contained three vertical bars of light projected at an oblique angle to the sample path.

The high-fidelity render methods will be considered first as they provide the fundamental approach to working with this type of data. Following on from this, methods will be explored that process this data and allow for real-time interaction.

3.1

Off-line Rendering

3.1.1

Traditional IBL

As mentioned in the previous chapter, traditional IBL uses one light probe to illuminate the entire virtual scene. Unger et al. [2, 3] show how this technique fails to adequately mimic the real-world lighting when naïvely applied to light probe sequences, especially those exhibiting high frequency spatial variations.

Figure 3.1 shows a scene with two neighbouring light probe samples applied using traditional IBL techniques. The difference in lighting information between the two samples can be easily observed, and would result in an unpleasant flickering between frames of an animation adopting this approach.

Figure 3.1. Traditional IBL applied to a scene using two neighbouring light probe

samples. Image taken from [3].

During the course of experimentation for this thesis, an extension to the tradi-tional IBL method was explored with the goal of removing the flickering artefacts exhibited during animation. For each rendered frame, a variable number of sam-ples on either side of the main sample were also included in the IBL lookup. Each sample in this set was Gaussian weighted to allow for smoother transitions be-tween areas of high frequency lighting in the sequence. Though the flickering can be alleviated by using this method, the technique fundamentally lacks the ability to represent illumination that varies over objects of all sizes, which is object of this thesis. Further insight into this problem can be seen by examining what tra-ditional IBL is doing in terms of the plenoptic function[14]: a conceptual tool for describing the flow of radiance within a scene.

3.1.2

The Plenoptic Function

In its general form, the plenoptic function describes at any time, t, any frequency of light, λ, travelling through any point in 3D space, x, from any angular direction,

~

ω, within the scene1. In this form it is a 7D function denoted by:

P (x, ~ω, t, λ) (3.1)

This function is often reduced to 5 dimensions by assuming t is constant and considering only the red, green, and blue wavelengths of light. The equation form thus becomes:

P (x, ~ω) (3.2)

Understanding the concept of the plenoptic function allows us to analyse the capability of any IBL technique. Traditional IBL, when written in terms of the plenoptic function, reduces to 2 dimensions based solely on the angular direction of a ray at any point in space. When viewed in this way it becomes apparent that this method would fail to represent any spatial variation that a scene may

3.1 Off-line Rendering 17

exhibit. Based on this analysis, the techniques required to adequately represent these lighting effects must utilise a higher degree of the plenoptic function.

The densely sampled light probe sequences, such as those obtained by the Real

Time Light Probe (Figure 2.5), can be described by equation 3.2 if we assume

a static scene. This satisfies the requirements for representing spatial variation, allowing more complex sampling algorithms to be used when rendering with these datasets.

3.1.3

Nearest Neighbour Sampling

The most straight-forward method for using a 1D light probe sequence is to find the sample along the sample path that is closest to the point being rendered and perform the IBL lookup there. This method is referred to as nearest neighbour sampling. Figure 3.2 demonstrates this process in 2D by looking orthogonal to the sample path.

Sample Path

Light Probe Samples p

d

Figure 3.2. Nearest neighbour sampling with a 1D light probe sequence. The point p is

projected to the nearest point on the sample path. The nearest sample(s) are then used as normal radiance maps, indexed with the direction d.

Since this method relies on the position of a point in space as well as the direction of a ray originating from that point, it appears to satisfy the plenoptic dimensionality of the dataset. Figure 3.3 shows a scene rendered using nearest neighbour sampling of a 1D light probe sequence. Though the objects in the scene now display spatial variation over their surfaces, at closer inspection the method actually fails to fully replicate the directional properties of the illumination. Given a linear sample path, the scene can be transformed such that the sample path lies along the z-axis. The nearest neighbour projection simply ignores the x and y coordinates of any point in order to find the nearest sample along the sample path. This means that all spatial variations that aren’t orthogonal to the sample path are treated as thought they are exactly parallel to the sample path. In figure 3.3, the light bands within the scene are actually projected from an oblique angle to the sample path, as stated as the beginning of this chapter.

Figure 3.3. A rendering using nearest neighbour sampling with a 1D light probe

se-quence. The vertical light bands are projected at an oblique angle to the sample path, though this method makes it appear as though they are orthogonal. This image also shows the noise that is generated by undersampling the environment maps, even though over 16000 samples were chosen per environment map and some hours were taken for it to render.

Since all 1D paths can be rotated to the z-axis, and all objects in space rotated the same, the sampling method described above can be reduced to the 3D plenoptic function P (z, ~ω). This proves that the nearest neighbour sampling method doesn’t,

in fact, satisfy the plenoptic dimensionality of the dataset.

3.1.4

Ray Projection Sampling

The downfall of the nearest neighbour method is its inability to utilise the full 5 dimensions of information that the dataset has to offer. Having another look at Figure 3.2, it becomes obvious that the direction d, when translated down to the sample path, would almost certainly point toward lighting information that is not appropriate to the same direction at point p.

To fix this, ray projection sampling, as discussed by [3] and shown in Figure 3.4, would find a far more appropriate sample to use. The ray at each point, r(t) = p + td, should be extended toward the linear sample path to find the shortest distance between these two lines. This indicates a better sample region to use for IBL lookup in the direction d. The equations that do this for samples along the z-axis are presented in [3] and are listed below in an adapted form:

r(t) = p + td

∆ =

q

rx2+ ry2

pproj = p + (arg min∆(t))d

zproj = pproj,z (3.3)

If point of shortest distance falls between two light probe samples then an ap-propriate interpolation should be used which would mainly depend on how dense

3.1 Off-line Rendering 19

the data was sampled, and the size of the object being rendered. For densely sam-pled light probe data being rendered onto large objects, the interpolation scheme becomes less important since the distance between samples only occupies a rela-tively small region (possibly sub-pixel). The converse scenario would require more sophisticated interpolation to try and maintain sharpness between areas of high frequency variations in light. The tools created for this thesis always use a linear interpolation scheme for blending between colour values of neighbouring samples.

Sample Path

Light Probe Samples p

d

interpolate

Figure 3.4. Ray projection sampling with a 1D light probe sequence. The direction d

is projected to the sample path from point p, indicating the better samples to choose.

Figure 3.5 shows the ray projection method applied to a simple scene. In this example, the oblique lighting direction is now represented on all rendered surfaces. As the points being rendered diverge away from the sample path, the lighting continues to illuminate them from the same oblique angle.

Figure 3.5. Ray projection sampling with a 1D light probe sequence. This method

displays better results for points diverging from the sample path. This image also shows the noise that is generated by undersampling the environment maps, even though over 16000 samples were chosen per enviroment map and some hours were taken for it to render.

3.1.5

Single-Viewpoint Reprojection

Following on from ray projection sampling, a further adjustment must be made to be sure that the correct sample is used. This adjustment is based on the physical size of the mirror-sphere and, given a straight line sampling path along the view direction, the θ component of the projected ray. This is discussed by [3], in which the following equations were presented:

θ 2 = arcsin r R z = z0− r · cos θ 2 − r · tan(θ − π 2) (3.4) Where r, R, θ, and z0 are describe in Figure 3.6.

Figure 3.6. The single-viewpoint reprojection method. z0(green) is the position of the

mirror sphere at each frame, while the red points, z, are the projection points of each incident ray back to the sample path. R is the radius of the mirror sphere and r is the radial distance from the sample path to the ray’s intersection point. Image taken from [3].

The offset z must be used in conjunction with the zproj value, found using the ray projection method, to obtain the correct region for choosing which light probe samples to use. This technique is especially important for correct rendering using densely sampled light probe sequences; where the spatial sampling is much smaller relative to the size of the mirror sphere.

This technique is not used for this thesis in order to simplify the calculations during real-time rendering. For consistency, it was also not applied to the off-line rendering shaders.

3.2

Real-time Diffuse Rendering

Due to the potentially long rendering times for each frame, it may become imprac-tical to rely on a trial and error approach when working with ILF data. This can

3.2 Real-time Diffuse Rendering 21

be further accentuated in production environments where a queue system con-trols access to the render-farm which animators use. A real-time interface into a spatially varying dataset provides greater efficiency for animators and lighting directors, as it removes the need for costly trial and error rendering by providing an interactive approximation of the outcome.

Figure 3.7. A scene lit bit a grill-covered spot light. The diffuse materials in the bottom

row reveal more about the spatially varying light than the specular materials in the top row.

The real-time methods presented below focus on the efficient representation of diffuse materials. The reason for this is that diffuse materials are able to better visualise the spatial variance within the scene than specular materials do, as seen in Figure 3.7.

3.2.1

Spherical Harmonics

An extremely small and efficient representation of Lambertian diffuse materials can be obtained by using spherical harmonics. Describing spherical harmonics in depth is out of the scope of this thesis, so only a few details will be presented below. More detailed and well explained information on this subject can be found in [6] and [7].

Overview

A spherical function is one that is defined on the surface of a sphere. The aim of a spherical harmonic projection is to decompose a spherical function into a sum of weighted basis functions; the same principle as Fourier transforms applied to 1D and 2D functions. This allows a potentially complex spherical function to be represented, in a simple way, by a list of basis function coefficients. The inverse

process, summing all the basis functions weighted by their coefficients, allows the originally function to be reconstructed.

The following sections will briefly describe the process of projection and re-construction using spherical harmonics. The descriptions below will only involve real-valued functions, though complex-valued functions are also able to be han-dled by this method. This is because diffuse irradiance estimation and spherical harmonic lighting only require real-valued functions for their calculation.

Basis Functions

As stated in [7], the functions, pn(x), that satisfy an orthogonality relation over a specific domain [a, b] are known as basis functions. The orthogonality relation can be expressed as: b Z a w(x)pn(x)pm(x)dx = cnδnm= cmδnm (3.5) Where: δnm= ( 1 n = m 0 n 6= m

and w(x) is an arbitrary weighting function independent of n and m.

The importance of this property allows any real-valued function to be repre-sented as the sum of weighted basis functions, given an infinite amount of basis functions. For a limited set of basis functions, the resulting reconstruction is a band-limited approximation. This property is ideal when decomposing low-frequency spherical functions, such as diffuse lighting, as a finite series should produce adequate results with a concise representation.

The associated Legendre polynomials are a set of real-valued functions which can satisfy the orthogonality relation for the interval [−1, 1]. They are defined as:

P_lm(x) = (−1) m 2l_l! p (1 − x2₎m d l+m dxl+m(x 2_{− 1)}l _(3.6) where l ∈ N0 and 0 ≤ m ≤ l. The value of l determines the band of functions.

The requirement for making these sets of functions satisfy the orthogonality re-lation is to keep either the m or l value constant throughout the calcure-lation of Equation 3.5. That is, if pn(x) is the associated Legendre polynomial Plm(x) then

pm(x) must be either Plm0 (x) or Pm 0

l (x) in order for the orthogonality relation to succeed. Because of this requirement, the associated Legendre polynomials alone will not suffice as a complete spherical harmonic basis.

Another set of orthogonal functions are sine and cosine, and are used as part of Fourier analysis. These functions satisfy the orthogonality relation on the interval [−π, π] and can be used in conjunction with the associated Legendre polynomials to build a spherical harmonic basis. For the following basis functions definition,

3.2 Real-time Diffuse Rendering 23 ym_l (θ, φ) =      √ 2Nm l cos(mφ)Plm(cos θ) m > 0 N0 1P10(cos θ) m = 0 √ 2N_l|m|sin(|m|φ)P_l|m|(cos θ) m < 0 (3.7) Where: N_lm= s 2l + 1 4π (l − m)! (l + m)! (3.8) is the normalisation component derived from the complex-valued spherical har-monic series, as shown by Schönefeld [7].

Figure 3.8 shows a graphical representation of the first four bands of the spher-ical harmonic basis functions described in equation 3.7.

Figure 3.8. The first four bands of the spherical harmonic basis functions. The colour

green indicates positive values, while red indicates negative. This image was taken from [6].

Projection and Reconstruction

Now that the spherical harmonic basis functions have been defined, a real-valued spherical function can be decomposed. This is called spherical harmonic

projec-tion.

The process of projection is to calculate a scalar value kn that represents how much the original function, f , looks like each of the basis functions, pn. This procedure simply involves taking the integral of product of f and pn over the full domain of f :

Z

Figure 3.9. An example of function projection (decomposition into basis function

coef-ficients) and the subsequent reconstruction. This image was taken from [7].

This process can be seen applied to the first few Legendre polynomials in Figure 3.9(a). The process of reconstructing the function f is achieved by summing each of the basis functions weighted by its associated coefficient. For perfect reconstruction this is an infinite sum, however we will only be dealing with finite representations, so the resulting function , ˜f , is a band-limited approximation of

the original: ˜ f (x) = N X n=0 knpn(x) (3.10) This procedure can be seen in Figure 3.9(b), which uses the constants calculated from the projection in the neighbouring image.

Diffuse Projection and Reconstruction of Environment Maps

To find the basis function coefficients of an environment map we must be able to integrate over the sphere of radiance values that the environment map represents. For this thesis, two methods were tested:

Monte Carlo integration - N uniformly distributed random samples on the unit sphere are used to index the environment map. Since they are uniformly distributed over the sphere the normalisation of the integration becomes 4π_N. Riemann Sum - All the valid pixels in the environment map are iterated through, ensuring all radiance information is present in the calculation. In this the-sis, mirror-sphere environment maps were used, so N ≈ X₄2π where X is the number of pixels along one edge of the image (assumed square). Like

3.2 Real-time Diffuse Rendering 25

the Monte Carlo method, the normalisation of the integration is 4π_N due to the even spread of pixels within a mirror sphere environment map[1]. This method of integration is used in preference to Monte Carlo integration since it is more accurate and is quick enough to process on commodity hardware. Further discussion on implementation and performance is presented in Sec-tion 4.2.2.

Other than the choice of sample points, the algorithm for projection remains the same for each method. Each chosen sample point in the environment map, (u, v), corresponds to a direction which can written in terms of θ and φ, as shown in Appendix A. Given a θ and φ value and the radiance from that direction, L(θ, φ), each basis function coefficient, km

l , is calculated by the following equation:

km_l = 4π N N X i=1 y_lm(θi, φi) L(θi, φi) (3.11) Ramamoorthi and Hanrahan [15] show that a Lambertian diffuse represen-tation of an environment map can be sufficiently approximated by the first nine spherical harmonic basis functions, those in the first 3 bands (l ≤ 2). This is due to the low-frequency nature of diffuse lighting variation, which suits the low-frequency nature of the functions occupying these bands. The result of this projection are the nine associated floating point coefficients, which can be used to generate the diffuse lighting in real-time on commodity hardware. Since we’re dealing with real-valued spherical functions, each colour channel of the environment map must be considered individually. This means that, for an RGB image, three sets of 9 coefficients are generated.

All the necessary calculations for doing this are presented by Ramamoor-thi and Hanrahan [15, 16], and some more verbose descriptions by Schönefeld [7], Green [6]. In short, for each point on the surface being rendered, the irradi-ance is determined by all light incident from the hemisphere centered on its normal,

~

n = (x, y, z). This hemisphere can be determined by all (~n · ~ω) ≥ 0. Expressed in

terms of a convolution, the irradiance becomes:

E(~n) =

Z

Ω(~n)

L(~ω)(~n · ~ω)dω

= L ? max(~n · ~ω, 0)

Ramamoorthi and Hanrahan [15] present two forms for calculating this. The matrix form, with the normal expressed in homogeneous form, nt= (x, y, z, 1):

E(n) = ntM n (3.12) Where: M =     c1k22 c1k2−2 c1k21 c2k11 c1k−22 −c1k22 c1k−12 c2k−11 c1k21 c1k2−1 c3k20 c2k10 c2k11 c2k1−1 c2k10 c4k00− c5k20     (3.13)

c1 = 0.429043

c2 = 0.511664

c3 = 0.743125

c4 = 0.886227

c5 = 0.247708 (3.14)

And the polynomial form:

E(n) = c1k22(x 2_{− y}2_{) + c} 3k20z 2_{+ c} 4k00− c5k20 + 2c1(k−22 xy + k 1 2xz + k−12 yz) + 2c2(k11x + k −1 1 y + k 0 1z) (3.15)

Rendering with Light Probe Sequences

Each environment map in the sequence can be easily processed, per colour channel, into this succinct diffuse representation. Even with the 500 light probe sequence used for testing these tools, the processed data still remains extremely small, allowing it to reside in texture memory with a very small footprint. Due to this factor, all the coefficient data was stored in the matrix form describe by Equation 3.13 to make texture lookup and GPU calculations more straightforward. This is described in more detail in Chapter 4 and Appendix B.1.

Figure 3.10. Real-time rendering of a 1D light probe sequence using the spherical

harmonic method. The same parallax errors occur as the nearest neighbour method presented in Section 3.1.3.

Though this method for rendering diffuse lighting is very fast and efficient, even for large light probe sequences, it suffers from the exact same problem as the nearest neighbour method presented in Section 3.1.3. Since the irradiance in Equations 3.13 and 3.15 are a function of the normal only, they exhibit the same parallax errors as the rendered points diverge from the sample path, which is seen

3.2 Real-time Diffuse Rendering 27

in Figure 3.10. The Future Work section on page 41 discusses an alternate process-ing technique which allows spherical harmonics to be used within an extrapolated volume of samples. This technique was not explored in this thesis due to the time constraints imposed.

3.2.2

Down-Sampling

Due to the low-frequency nature of diffuse lighting, a down-sampling of the en-vironment map can provide a valid diffuse representation. By selecting a unique sample set of unit-length directions, an environment map can be split into Voronoi regions where each pixel (and its associated direction) can be uniquely associated with its closest sample. These pixels within each region of the environment map are averaged and the resulting value associated with the primary direction for that region. This allows a ray-projection algorithm to be employed where, for each point being illuminated, the set of directions can determine the correct set of down-sampled information to use.

Generating the set of unit-length directions can be done in many ways. Ideally, the directions should correspond to the main directions of light sources within the environment map. However, the directions chosen must be kept the same when processing all the environment maps in the sequence, due to the rendering algorithm remaining independent of any one map. Analysing the algorithm in the Section 4.2 and the GLSL code in B.2 makes this more clear. Because of the non-trivial nature of finding optimal sample directions for any sequence, two generic methods for generating them were tested:

Uniform Polar Samples - Generate exactly N number of samples in the 2D spherical coordinate plane. These samples are found using the following algorithm:

begin

pairList = Find integer pairs,(a, b), s.t. N − (a × b) ≤ 2

(a, b) = Search pairList for pair with smallest |a − b| remainder = N − (a × b)

if (remainder > 0)

Add sample at top pole

Generate max(a, b) azimuth samples

Generate min(a, b) polar samples between poles if (remainder == 2)

Add sample at bottom pole end

Figures 3.11(a) and 3.11(b) show examples of the regions formed by uni-formly sampling the spherical coordinate plane. Twelve and 42 sample points were chosen for comparison with following method.

Icosahedron and its subdivisions - The 12 vertices of an Icosahedron are uni-formly distributed over the surface of a sphere, producing uniform Voronoi

regions. Successive subdivisions of the Icosahedron produce near-uniform vertex distributions which are still good enough. The regions produced by an Icosahedron and its first subdivision can be seen in Figures 3.11(c) and 3.11(d).

(a) 12 Polar Sample Regions (b) 42 Polar Sample Regions

(c) 12 Icosahedron Sample Regions (d) 42 Icosahedron-based Sample Regions

Figure 3.11. Rendering of a sphere showing the difference in regions between the spherical sampling schemes. The positive directions for the x(red), y(green), and z(blue) axes are shown. The green dots represent the primary sample direction for that region. Figures 3.11(a) and 3.11(b) show the non-uniform regions produced by uniform sampling in polar coordinate space. Figures 3.11(c) and 3.11(d) show the more uniform regions produced by using the vertices of an Icosahedron and its first subdivision respectively.

For creating sample directions in a generic way, it is desirable for all directions to be uniform spaced over the surface of the sphere, which would lead to uniform regions associated with each direction. In the polar sampling approach, the regions at the top have much smaller area than those around the equator, which would lead to inaccuracies and bias when rendering with this information. For the Icosahedron

3.2 Real-time Diffuse Rendering 29

and subdivision approach, all regions are more evenly distributed allowing a better indication of the illumination around that sample direction. The only downfall is that it is not possible to generate an arbitrary number of sample directions. Uniformly sampling the surface of sphere with an arbitrary number of samples is a non-trivial process. A simulated annealing approach was attempted during experimentation, but the complexity was deemed out of the scope of this thesis and was duly dropped.

Once the samples have been decided upon, each environment map in the se-quence can be processed to find the average radiance per region. For N sample directions, the processing of each environment map produces N RGB radiance val-ues, representing the average radiance for the Voronoi region around each sample direction.

(a) 12 Polar Samples (b) 42 Polar Samples

(c) 12 Icosahedron Samples (d) 42 Icosahedron Samples

Figure 3.12. Real-time rendering of a 1D light probe sequence using the down-sampling

method. The same four samples sets shown in Figure 3.11 are used.

Just like the spherical harmonic method, this data can be used with modern graphics hardware and custom shaders. However, real-time rendering with this information is far more computationally expensive than the spherical harmonic method. The algorithm for processing this data involves a for-loop over all sam-ple directions, and for an arbitrary number of directions this becomes extremely inefficient on a GPU. Interactive framerates were still able to be maintained given a reasonable number of samples (details discussed below). More details regarding the implementation of this shader can be seen in Section 4.3 and Appendix B.2.

in Figure 3.11. Though problems with parallax still exist, the representation of the environment lighting is able to be improved upon. The four images display how vastly different the results can be when the sample directions that are chosen fit poorly with the environment lighting.

Figure 3.12(a) shows almost no difference compared to the spherical harmonic method. The reason for this can be seen more seen more clearly when looking the samples it is using, shown in 3.11(a). Each of the samples lie in either the x-y plane or the y-z plane, so all the ray projections will be non-oblique. Figure 3.12(c) improves on this by including samples in the oblique regions, which can be seen on the red cylinder. A definite under-sampling in the upper hemisphere as cause these samples misrepresent the illumination of the floor. Both twelve-sample tests obviously lack the angular resolution to handle this scene.

As you would expect, a larger number of samples results in better represen-tation of the scene lighting, as shown in figures 3.12(b) and 3.12(d). This comes at a greater cost in processing time, reducing the framerate dramatically from the previous tests. The response time for this still remains adequate for interaction, and since this provides a closer representation of the actually lighting it will be very much desired by the relevant directors and animators.

The limiting factor for performance of this technique is the number of fragments needing to be rendered, not the number of triangles in the scene. For tests run in an 800x800 viewport on a nVidia Quadro FX 2500M, the 12 sample datasets ran at 20 FPS for viewport-covered surfaces, and over 50 FPS when approximately half the viewport was covered. For the 42 sample datasets, a full viewport of fragments rendered at around 5-8 FPS, while an approximately half covered viewport ran at 15 FPS.

Generic sampling of the environment maps requires a greater number of sam-ples for adequate reproduction of illumination. An understanding of the prominent light sources within the scene would enable a better sampling of the environment maps through the manual specification of directions.

Chapter 4

Implementation and Usage

The following sections will present specific details regarding the implementation and use of the tools created during this thesis. The tools include:

• A light probe sequence processing utility for generating the data used for

real-time rendering,

• A Maya hardware shader for real-time interaction, and

• mental ray light and environment shaders for using light probe sequences.

4.1

Ray Projection Algorithm

Common to both real-time and off-line shaders is the ray projection algorithm. This is described in Section 3.1.4 and by Unger et al. [3], and is a special case for finding the closest point between two lines. Equation 3.3 labels this point as

zproj, which can be determined by solving (arg min∆(t)) and applying it to the ray equation r(t). To find the minimum ∆, where ∆ ∈ R+, is the same as finding the minimum of ∆2, allowing the task of finding the minimum to be considerably easier. This process can be seen in Appendix C and allows pproj to be calculated by the equation: pproj= p − pxdx+ pydy dx2+ dy2 · d

4.2

Processing

Pre-processing the light probe sequence allows us to use its diffuse information in a real-time setting. As mentioned in Section 3.2, two types of real-time ren-dering methods were tested: one using a spherical harmonic representation, and the other using a direction-based down-sampling of the environment map. The following sections will outline the exact algorithms used to process the data into their relevant forms.

4.2.1

Spherical Harmonic Representation

A small spherical harmonics projection library was created to allow for direct computation of HDR environment maps. This code is simply an implementation of Equation 3.7 that is able to calculate coefficients of the three colour channels simultaneously.

Listing 4.1 show how to use the Riemann sum method (described in Section 3.2.1) for projecting an environment map to obtain the spherical harmonic coef-ficients. This algorithm assumes an image iterator exists which can only point to appropriate pixels in the environment map. This iterator also calculates the relevant θ and φ values that pixel represents:

C o l o u r [ ] S H P r o j e c t i o n ( EnvMap , numBands ) { C o l o u r s h C o e f f i c i e n t s [ numBands ∗ numBands ] numSamples = 0 f o r p i x e l = a l l p i x e l s i n EnvMap f o r l = 0 t o numBands f o r m = − l t o l i n d e x = l ∗ ( l +1) + m s h C o e f f i c i e n t s [ i n d e x ] = Y( l , m, p i x e l . t h e t a , p i x e l . p h i ) ∗ p i x e l . c o l o u r numSamples++ // N o r m a l i s e t h e c o e f f i c i e n t s f o r i := 0 t o numbands ∗numbands s h C o e f f i c i e n t s [ i ] ∗= 4∗ PI / numSamples return s h C o e f f i c i e n t s }

Listing 4.1. Spherical Harmonic projection algorithm

Once all the coefficients for an environment map are calculated, they are put into the matrix form shown in Equation 3.13. Each colour channel’s matrix is then split into its column vectors and stored in a binary file. A group of three matrices, one for each colour channel, is referred to as a set. There are as many sets as there are images in the light probe sequence. Once the processing is complete, all sets’ matrices are stored in binary and input as a texture into graphics hardware. Figure 4.1 shows the representation from the perspective of a texture. Each pixel in this texture corresponds to a column vector for a matrix. These column vectors can be read in, combined into a matrix, and applied to a normal vector very quickly on the GPU. The GLSL code for this is shown in Section B.1.

4.2.2

Down-Sampled Representation

A small class for creating spherical samples in both polar and Icosahedron sub-division form provides the basis for the tests used in this thesis. This algorithm is not limited to these types of samples, as discussed in Section 3.2.2, and may benefit strongly from user-defined input.

In Listing 4.2 the algorithm for down-sampling one environment map is pre-sented. Just as in Listing 4.1, an image iterator is assumed to exist which returns

4.2 Processing 33 R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G R A B G set 1 set 2 set 3 column vectors

red matrix green matrix blue matrix

Figure 4.1. The binary format of the spherical harmonic data processed from the light

probe sequence, presented in terms of its use as a texture on the GPU.

only relevant pixels in the environment map. In this case, the pixel information also contains the direction that pixel corresponds to in the environment map.

s t r u c t S a m p l e I n f o { C o l o u r t o t a l R a d i a n c e ; C o l o u r a v e r a g e R a d i a n c e ; i n t n u m P i x e l s ; }

Sample [ ] DownSample ( EnvMap , V e c t o r d i r e c t i o n s [ ] ) { S a m p l e I n f o s I n f o [ d i r e c t i o n s . s i z e ] f o r p i x e l = a l l p i x e l s i n EnvMap f i n d d i r i n d e x i n d i r e c t i o n s s u c h t h a t d i r i s c l o s e s t t o p i x e l . d i r e c t i o n s I n f o [ d i r ] . t o t a l R a d i a n c e += p i x e l . c o l o u r s I n f o [ d i r ] . n u m P i x e l s++ f o r i = 0 t o s I n f o . s i z e s I n f o [ i ] . a v e r a g e R a d i a n c e = s I n f o [ i ] . t o t a l R a d i a n c e / s I n f o [ i ] . n u m P i x e l s return s I n f o }

Listing 4.2. Down-Sampling algorithm

Finding the closest sample direction for each pixel’s direction can be evaluated by taking the cross product between the directions and finding the one with the maximum value. For each pixel this has to be done N times, where N is the number of sample directions. If this algorithm is applied to an environment map sequence with images of the same dimensions, a lookup table can be easily be generated that stores the index of the closest sample direction for each pixel in the