A Measurement-Based Skin Reflectance Model for Face Rendering and Editing

abstract
we present a novel skin reflectance model for faces and its application to face appearance editing. we decompose the high-dimensional bidirectional scattering surface reflectance distribution function (BSSRDF) of skin into components that can be estimated from measured data. our model is intuitive, amendable to interactive rendering, and easy to eidt. high-quality renderings come close to reproducing real photographs. we have measured 3D face geometry, skin reflectance,and substance scattering for a large group of people using custom-built devices and fit the data to our model. the analysis of the reflectance data reveals variations according to subject age, race, gender and external factors (heat, cold, makeup, etc.) we derive a low-dimensional model using non-negative matrix factorization (NMF) that spans the space of skin reflectance in our database. a user can define meaningful parameters in this space-such as race, gender, and age -and change the overall appearance of a person (e.g., making a Caucasian face look more Asian) or change local features (e.g., adding models, freckles 雀斑 or hair follicles 毛囊).

1 introduction
one of the most difficult computer graphics challenges is creating realistic human faces. humans have evolved to be incredibly adept at interpreting facial appearance 识别面部表情。 for example, we can easily distinguish if a person is tired, hot, excited, or sick. although a lot of effort has been devoted to face modeling in computer graphics, no synthetic face model to data achieves this level of expressiveness and realism.

in this paper, we focus on modeling skin refletance of human faces, an important aspect of face appearance. it varies for different people (e.g., due to race or gender) and even varies for the same person throughout the course of a day (e.g, ht vs. cold skin). a realisitc skin reflectance model should be able to accomodate 适应 these variations. it should also allow a graphic artist to change the appearance of skin based on easy to interpret parameters (e.g., race, gender, or age). the model needs to easily connect to measurements of real faces for the creation of virtual doubles. images generated from the model——ideally in real-time——need to look photorealisic from arbitrary viewpoints. and the model should allow easy modification or transfer of skin appearance.

To achieve these goals we have developed a novel skin reflectance model whose components can be robustly estimated from measured data. Our model is accurate, compact, and intuitive to edit. It can be used in interactive and offline rendering systems and generates results that come close to reproducing real photographs. We use custom-built devices to measure in-vivo light reflection and subsurface scattering of a large and diverse group of people. 1 Our data ranges across age (13 to 74 years old), gender, race, and external factors (e.g., cosmetics, cold, and sweat). We fit our model to the measured data and compute a low-dimensional face reflectance space using non-negative matrix factorization (NMF) [Lee and Seung 1999]. User-defined parameters – such as gender, race, or tan – allow us guide the interpolation of reflectance data to change the appearance of a face overall or locally.

2 PreviousWork
Properties of human skin have been measured and studied extensively in the biomedical 生物医学, cosmetics 化妆品, and computer vision communities. In this section we provide an overview of the relevant work in the area of computer graphics and image synthesis.

Analytic Skin Reflectance Models:
Analytic reflectance models are attractive because of their computational efficiency. Hanrahan and Krueger [1993] modeled single-scattering of light in skin composed of multiple smoothly-bounded internal layers. Ng and Li [2001] extended this model by adding an oil layer to the skin surface. Stam [2001] developed an analytic approximation to multiple subsurface scattering in skin with a rough surface.
More recent work [Krishnaswamy and Baranoski 2004] proposes a biophysically-based 生物物理 multi-layer model for image synthesis with biologically meaningful parameters.
Several skin modeling approaches use analytic bidirectional surface reflectance functions (BRDFs) [Blanz and Vetter 1999; Debevec
et al. 2000; Haro et al. 2001; Paris et al. 2003; Tsumura et al. 2003; Fuchs et al. 2005]. The BRDF parameters can be estimated
from reflectance measurements using non-linear optimization. Although a BRDF describes local light transport at each surface point,
it ignores subsurface scattering, which is largely responsible for the appearance of skin.

Jensen et al. [2001; 2002] propose an analytic model for the bidirectional surface-scattering distribution function (BSSRDF). The
BSSRDF describes the full effect that incident light at a point has on the reflected light from a surface patch around that point. The
BSSRDF is eight-dimensional, assuming a two-dimensional parameterization of the surface. Because dense sampling of an eight dimensional function is challenging, we subdivide the BSSRDF into components that can be more easily measured (see Section 3).

Non-parametric Skin Reflectance Models: Instead of fitting an analytic BRDF model, Marschner et al. [1999] estimate a nonparametric
BRDF of a human face by combining reflectance samples from different points on the surface. They later extended this work by adding a detailed albedo texture [Marschner et al. 2000]. They observe that the BRDF of skin is quite unusual and exhibits strong forward scattering at grazing angles that is uncorrelated with the specular direction. We use the data-driven BRDF model of Matusik et al. [Matusik et al. 2003] to estimate a non-parametric surface BRDF at each surface point. We found that this introduces less error than imposing the behavior of a particular analytic BRDF model. More importantly, it does not require non-linear optimization and leads to a
more robust fitting procedure.

Image-based Face Modeling: Image-based methods have provided highly realistic representations for human faces. They easily
capture effects such as self-shadowing, inter-reflections, and subsurface scattering [Pighin et al. 1998]. Recent efforts allow variations
in lighting [Georghiades et al. 1999; Debevec et al. 2000], viewpoint, and expression [Hawkins et al. 2004]. Cula et al. [2005;
2004] collected a database containing more than 3500 skin texture images that were taken under various illumination and viewing conditions [Rutgers ]. However, the memory requirements for imagebased models are large. The measurement procedures are inefficient and assume non-local low-frequency lighting. Pure image-based representations are also inherently difficult to edit and modify. Borshukov and Lewis [2003] combine an image-based model, an analytic surface BRDF, and an approximation of subsurface scattering to create highly realistic face images for the movie Matrix Reloaded. Sander et al. [2004] developed a variant of this method
for real-time skin rendering on modern graphics hardware.
An interesting image-based method was presented by Tsumura et al. [2003], who use independent component analysis (ICA) to
decompose images of faces into layers (melanin and hemoglobin). Their method is capable of re-synthesizing new images while adding effects like tanning 晒黑 or aging.

3 skin reflectance model
overal skin reflectance can be described as the sum of specular reflection of the skin surface (air-oil interface) and diffuse reflection due to subsurface scattering (see figure 2). diffuse subsurface

figure 2: skin reflectance can be explained by a specular (BRDF) component at the air-oil interface, and a diffuse reflectance component due to subsurface scattering. most of the high-frequency spatial color variation in human skin is due to the epidermal layer, whereas strong light scattering is the dermal layer is a more slowly varying effect. we model the first (high-frequency) effect with an albedo map and the second (low-frequency) light transport with a translucency map.

scattering is due to absorption and light scattering in the epidermal and dermal skin layers. The epidermis scatters light strongly and contains melanin 黑色素 (along the interface to the dermis layer), which is highly absorbing. this absorption component is a local effec with high spatial variation across the face due to hair folliles 毛囊, sweat glands 汗腺, freckles 雀斑, dimples 酒窝, etc. the dermis/blood layers is highly scattering in the red channel and strongly absorbing in the greeen and blue channels (mainly due to haemoglobin血红蛋白). the dermal light scattering is a non-local, slowly varying effect.

we model the light that is immediately reflected from the oil-skin layer with a spatially-varying surface BRDF and divide diffuse subsurface reflectance into two components: A diffuse albedo map that captures high-frequency color variations due to epidermal absorption and scatterng, and a translucency map that captures low-frequency absorption and scattering in the dermal layer. fine-scale face geometry is represented by a normal map.
More formally, we denote the BSSRDF as S(xi,ωi, xo,ωo), where ωi is the direction of the incident illumination at point xi, and ωo is the observation direction of radiance emitted at point xo. Similarly, we use fs(xi,ωi,ωo) for the surface BRDF. The relative contributions of the surface BRDF and the diffuse reflectance due to subsurface scattering are modulated by Fresnel coefficients:

where Fr and Ft = (1 - Fr) are the Fresnel coefficient at the air-skin boundary for both the incoming and outgoing radiance, and η is the
relative index of refraction between skin and air (约等于 1.3). We model the subsurface scattering term S using the dipole diffusion approximation [Jensen et al. 2001], while the specular BRDF component fs is modeled using a data-driven approach. The parameters of both the BSSRDF and the BRDF are estimated from measurements as described in the following sections.

4 Measurement Procedure Overview 测量过程概述
A block diagram of our measurement pipeline is shown in Figure 3. We capture the 3D geometry of the face using a commercial 3D
face scanner. Digital photographs from different viewpoints and

Figure 3: A block diagram of our data processing pipeline. Blocks in grey are the parameters of our skin reflectance model.

with different illumination directions are taken in a calibrated facescan dome. The data is used to compute a normal map and to estimate the diffuse reflectance at each surface point. We subtract the diffuse reflectance from the measured data and fit a set of denselymeasured BRDFs to the remaining surface reflectance. We compress the BRDF basis using NMF to derive a small set of NMF basis BRDFs. We then measure the subsurface scattering of skin at few locations in the face using a special contact device and estimate skin translucency.
To map between 3D face space and texture space we use the area-preserving texture parameterization of Desbrun et al. [2002]. The data is densely interpolated using push-pull interpolation into texture maps of 2048×2048 resolution. The parameters of our reflectance model are the NMF basis BRDFs (typically four), textures with coefficients for the linear combination of basis BRDFs, one albedo map with diffuse reflectance values, and one translucency map. The following sections describe each of these processing steps in more detail.

5 measuring skin reflectance
figure 4 shows a photograph of our face-scanning dome 圆顶. the subject sits in a chair with a head rest to keep the head still during the capture process.

the chair is surrounded by 16 cameras and 150 LED light sources that mounted on a geodesic dome. the system sequentially turns each light on while simultaneously capturing images with all 16 cameras. We capture high-dynamic range (HDR) images [Debevec and Malik 1997] by immediately repeating the capture sequence with two different exposure settings. The complete sequence takes about 25 seconds 花费25秒 for the two passes through all 150 light sources (limited by the frame rate of the cameras). To minimize the risk of light-induced seizures we ask all subjects to close their eyes. We report more details about the system and its calibration procedure in [Anonymous 2005].(2)
(2) this technical report has been submitted as supplement material.

a commercial face-scanning system from 3QTech (www.3dmd.com) is placed behind openings of the dome. using two structured light projectors and four cameras, it captures the complete 3D face geometry in less than one second. the output mesh contains about 40,000 triangles and resolves features as small as 1 mm. we clean 清除 the output mesh by mannually cropping 手工清除 non-facial areas and fixing non-manifold issues and degenerate triangles. the cleaned mesh is refined using loop-subdivision [loop 1987] to obtain a high-resolution mesh with 500,000 to 1 million vertices. the subdivision implicitly removes noise. we store the high-resolution mesh as an unstructured list of point samples (surfels) 不知道什么意思 without connectivity. each surfel stores the necessary in formation for image reconstruction using EWA splatting [Zwicker et. al. 2002].

next we compute a lumitexel [Lensch et al.2001] at each surfel position from the image reflectance samples. each observed radiance value L(ωo) is normalized by the irradiance, Ei(ωi), of the corresponding light source l in order to obtain a BRDF sample value:

We calibrated the BRDF measurements using Fluorilon — a material with know properties [Anonymous 2005]. All processing is
performed on RGB data except where noted otherwise.