Xenoblade 3 Character Implementation
Presented by 舘野圭 (Tateno Kei) – (2022/11/12)
Written by じく (Jiku) – (2023/03/22)
Localised by BNK – (2024/08/03)
Original article: https://gamemakers.jp/article/2023_03_22_33475/
CEDEC+KYUSHU 2022, the first real conference for Japanese game developers in three years, was held on Saturday, November 12th 2022.
In the presentation titled “Character Implementation and Upsampling in Xenoblade Chronicles 3,” the topic of two-tone toon-shading and appearance developed to enhance graphical design was discussed, as well as the upsampling method implemented on the Nintendo Switch.
Kei Tateno, a programmer at Monolith Soft Inc., took to the stage. Mr. Tateno was in charge of the graphics team for “Xenoblade Chronicles X”, and was a contributing member of the graphics team for “Xenoblade Chronicles 2” (hereafter referred to as “XC2”), “Xenoblade Chronicles: Definitive Edition” (hereafter referred to as “XC:DE”), and “Xenoblade Chronicles: 3” (hereafter referred to as “XC3”). Mr. Tateno was in charge of character implementation for “XC2” and “XC3”.
(All titles were released by Nintendo Co. Ltd.. For the remainder of this article, “XC2” and “XC:DE” will be referred to in the past tense.)
The World Design of “Xenoblade Chronicles 3” in Toon & PBR
XC3, released exclusively for the Nintendo Switch on July 29th 2022, features the theme of “life stories connecting the future,” which is reflected in the detail and presentation of the graphics.
In-game, characters are rendered in with toon-shader (resembling an anime or a comic) and environment are rendered in with a PBR shader (Physical Based Rendering, which replicates the behavior of light in the real world). As can be seen in the screenshot below, the characters’ faces are rendered in toon-shading, while the environment, metal, and water surfaces are rendered in PBR shading.
Rendering uses Deferred Shading, with an implementation of Clustered Shading1. The result calculated from the G-Buffer2 on the left side of the image below is obtained and processed as the image on the right side of the image below.
Upsampling is applied to the 3D rendering results to achieve higher resolution.
Left image before upsampling, right image after upsampling
This presentation focused on “character implementation of toon-shading” and “upsampling,” along with the techniques and methods used.
Toon-Shader Implementation and Representing “Life and Death”
How the Toon-Shader Works
The basic toon-shader is made up of a base colour, a ramp (colour gradient), and a light (luminance and colour) multiplied together.
Each texture and environment has its own ramps.
Ramps are assigned on a per-material basis. For Mio, the character on the left side of the image below, the skin and hair are clearly two-tonal, while the shirt is coloured differently on the shadow side to simulate a sub-surface texture. Noah’s jacket, the character on the right side of the image, has a velvet-like luster due to the complexity of its ramp.
Ramp comparison of skin, hair, shirts, jackets, etc.
In addition, this also helps reduce saturation and increase contrast in dark areas, so a cold ramp is mainly applied in said areas rather than light areas. Linear interpolation between light and dark areas is performed per-pixel, and the coefficient is calculated based on the white exposure of the environment.
When the light ramp is applied at night, the skin colour looks unnatural with a more orange saturation,
so the dark ramp is applied to reduce the saturation and make it look natural.
In addition, ramp offset is performed by providing a dedicated mask for areas where control of the shadow colour is needed.
Texture for ramp offset is applied under the chin, the neck, the collarbone, etc.
Lighting to Avoid Overexposure
Since the lights are set and placed based on the appearance of the field using PBR, applying them to the toon as is will result in white blur.Therefore, clamping (limiting the amount of Bloom to a pixel) is applied with a white brightness of 1.0 at scene exposure.
Shader code on the left, comparison of character on the right shows overexposure compensation.
Normal Map-Based Hatching
Although not part of the toon-shader itself, normal map-based hatching3 is applied to add NPR4-like details. A texture of hatch-like irregularities is added to areas such with glossy properties and enhanced shading.
Development of the Two-Tone Toon Shader
Life and Death as Expressed in Light and Shadow
Since the theme of the game is “life”, or “the cycle of life and death”, the team wanted to use lighting to express the shadows of the characters, making use of higher contrast and more distinct shading than in previous games.
For example, in a scene where a character is illuminated by the fires of a battlefield, the shading needed to reflect the point light from the fires, while at the same time providing directional light from the moon.
In previous games, point lights and spotlights (hereafter referred to collectively as “point light (spotlights)” were reflected on characters but this required ignoring normals and applying only distance attenuation. With this method, it was impossible to tell from which direction the light was shining, and it was not possible to correctly represent shadows.
In previous games, only luminance and distance attenuation were applied
as taking normals into account would cause “unpleasant shadows” on the face.
Two-Tone Toon-Shading with Main and Directional Lights
A new “two-tone toon-shader” was developed for this game to allow for the colour and ramp direction of the directional light to be combined alongside the colour and ramp direction from the point lights (spotlight).
The implementation sets the dominant light on the character as the “main light” and reflects the directional light in the shadow areas where the main light is absent. This technique allows the two lights to coexist without destroying the shape of the shadows.
The implementation makes use of an adjusted ramp luminance relative to the main light as a mask to interpolate between the directional light and the main light.
The ramp results for the directional and main lights are interpolated
using the ramp luminance of the main light.
The “main light” is the dominant light applied to a character, but it is not the same as a pointlight (spotlight).
If both the pointlights (spotlights) and directional light are hitting the same area, said area should appear more luminous as a result. In addition, the appropriate lights should fade with distance, respectively.
In other words, if the pointlights (spotlights) are close then their combined colour and direction should be dominant; but if they’re far away then the colour and direction of the directional light should be dominant. For this reason, the “main light” is a composite of two light sources; the pointlights (spotlights) and the directional light.
The composition of pointlights (spotlights) and directional light is done per-pixel.
The composite formula for the main light.
Calculating luminance is simple addition, while calculating colour and vector are weighted average.
However, there are more than two lights in most scenes, where multiple point lights (spotlights) overlap each other. Therefore, a localised two-tone system is used, in which multiple pointlights (spotlights) are combined into a main light.
For example, if there is a green and a red point light as shown above,
then the main light will be yellow, and the directional light is reflected in the shadow area.
The composition formula for multiple point lights (spotlights) extends the composition formula laid out in a previous slide.
Advantages and Disadvantages of Two-Tone Shading
As the two-tone shading was adopted, the team ran into problems with indoor lighting. Indoors, characters are often lit from above, and in some cases, this causes the faces to look bad.
Most of the light is shining from above, and the shadow on Mio’s nose appears unnatural.
Therefore, with the toon-shader, the elevation angle of the light should be limited to ±40 degrees in world space.
Even when the light is directly above or below the character,
there’s a horizontal component such that the shading looks more natural.
One of the disadvantages of this two-tone shading is that continuous changes in light vector, such as those moving across an environment or with multiple light sources present, can appear noisy. This disadvantage was not been resolved in time for XC3’s release.
Another disadvantage is that, in some cases, the interpolation between the two light sources against the skin can give an “unpleasant” look. When the almost dominant main light and the directional light are close to each other, the intersection area will have unnatural shading.
The unnatural shading can be seen in the middle of the face in the image above, for example.
The team considered using single light shading for the skin only, but decided against it because two lights look better than one in many scenes.
Rim Lighting
In relation to the two-tone shading, a rim light was applied, which is toon-shader only and illuminates the edges and hair of characters.
The rim light can be adjusted in their intensity, colour, and thickness on both the light and dark side of the shading. These are set as part of the environmental information, with the light side reflecting the colour of the main light in a two-tone shader. This allows the rim light to saddle in the vicinity of the main light.
Adjusting the Shader to Make Character Faces More Appealing
With the two-tone shading, the team were able to correct the shadow areas on characters. Let’s take a look at what adjustments were made to make the faces appear more natural.
Detailed Facial Adjustments
Character faces are adjusted in almost every detail, including; shading, light angle, hatching, highlighting, and transparency.
Adjusted Shader
With the early model, dirty shadows appeared depending on the light angle, changing the curves on the face.
Especially in night scenes, where the contrast is high and the shadow boundaries are more noticeable, two-tone shading masks the “nasty shadows” and what was previously “a little weird looking” becomes “noticeably weird looking”.
The “before” image on the left compares the early model with the two-tone shader.
The “after” image on the right compares the final model with the two-tone shader;
the shading is more natural due to the adjusted shading described below.
Therefore, we adjusted the face normals to be more gradual and less angular. We created a dedicated mesh to prepare and apply the normals, which were done during asset creation rather than at runtime.
The character development workflow was changed from using vertex normals5 for the face normals to using normal maps6. A lot of character models had already been created at this stage, so it was difficult to make update the topology7 and make them consistent with new character models being created after this workflow change.
Fixed Light Elevation Angle
Even with the aforementioned adjustments, “unpleasant shadows” can sometimes still occur when the light hits a character’s face from diagonally above or below. This occurs even when the aforementioned light elevation angle is limited to ±40 degrees.
You can see these “unpleasant shadows” on the left side of Mio’s face.
The team tried to remedy this problem by locking the light elevation angle with respect to a character’s face. However, this requires an angle relative to the character’s head; this is so that light will hit from the intended direction even if the face is tilted in a cutscene. Therefore, the team decided to developer a head coordinate system as part of the shading process.
The XYZ coordinate system centered on the character head is passed via a constant buffer within the shader.
Since this games uses deferred shading, the data is passed via the G-Buffer. Therefore, the G-Buffer pass stores the index of each character on-screen, and the lighting pass retrieves it from another buffer containing an array of head coordinates.
The light elevation angle was fixed at +10 degrees when the light is coming from above. You can see this corrects for the “unpleasant shadows”.
The reason for setting the light elevation angle at +10 degrees is that a slight horizontal shift results in a more natural appearance in the middle of the face. With a light elevation angle of 0 degrees, the middle of the face will appear more straight, but with a light elevation angle of 10 degrees, the middle of the face has more curvature around the nose and mouth, resulting in a more natural appearance.
Interpolation Between Head and Neck
Fixing the light elevation angle will create a unique boundary on a character’s neck due to the different lighting techniques being applied separately on the face and body.
Therefore, the team created a special mask to interpolate between the two lighting systems. Note that for the eyes, the team wanted to create an exception to the fixed light elevation angle, so it was included as part of the mask.
Correcting “Unpleasant Shadows” With Hatching
Even after all these adjustments, intermediate “unpleasant shadows” still occurred in an angular range of 5 degrees between 12-17 degrees on the Y axis.
However, since it is difficult to get rid of these shadows, the team applied a hatching texture to turn the “unpleasant shadows” into an aesthetic choice.
Turning on hatching results in a faint dither effect on Nio’s face.
Triangular Highlights on Cheeks
The triangular highlights visible on the cheeks fade according to the light angle.
The implementation calculates the final light angle on the ZX plane within the head coordinate system, fades it, and applies a mask for highlighting.
Cheek highlights change according to the indicated light direction (white arrow).
Lip and Nose Highlights
Highlighting of the lips and nose is not done via shading, but by offsetting the highlight texture UVs according to view space normals, such that they are shifted according to the viewpoint.
When looking headon, the lips and nose are highlighted in the center,
but as the viewpoint shifts, the highlights shift as well.
Highlights in NPR do not always look good using specular calculations, so the team created special highlight textures to represent highlight transition by shifting UV coordinates.
Eyebrow and Eyelash Transparency
Eyebrows and eyelashes are processed through a stencil test.
Stencil testing is a rendering method which sets a reference for each pixel and decides whether or not to draw the pixel based on that value. Stenciling is used for techniques like object hollowing.
Techniques for More Natural Hair
Several techniques were used for character hair including; hair highlights, backlight scattering when backlit, hair tufts and parting, and shadow under the fringe.
Hair Highlights
Highlights in the hair, also known as halos, are mapped using a spherical environment map8 with the highlight texture.
In cutscenes, for example, highlight reflections are calculated using the aforementioned head coordinate system so that there is no visual discrepancy even when the face is headon or turned sideways.
Hair highlights are applied horizontally relative to Nio’s face, even if the face is rotated diagonally or sideways.
Hair Backlight Scattering
A character’s hair is subject to backlight scattering9, a pseudo-formula of scattering caused by backlighting.
When backlight scattering is turned on, Nio’s hair is more luminous at the neck and edges.
If the hair normals are used as is in the calculation of backscattering, the effect of backscattering will appear inside the hair as well. A corrective calculation uses the head coordinate system, and the vector from the center to the position of the hair is treated as a pseudo-spherically vector and blended into said calculation.
Shader code for backlight scattering. This calculation is based on the rim light calculation, so as to appear stronger when backlit.
SNN Filter
The hair is also subjected to an SNN filter10 which differentiates it from other textures using the toon-shader.
Nio’s hair mesh has a lot of tufts, but an SNN filter can improve hair uniformity.
The hair parting is a separate mesh, with an SNN filter applied to blend it with the rest of the main hair mesh to create a more natural look.
The hair parting mesh is made with an SNN filter in mind.
Fringe Shadows
The shadow of the fringe over a character’s forehead is implemented by placing a shadow-shaped mesh beneath the fringe just above the skin. This mesh uses a translucent material with lighting disabled.
The advantages of using a mesh for shadows include: avoiding accuracy amd resolution issues, low render time, allowing overlapping shadows, the ability to control the shadow’s shape, and the ability to weight the mesh differently from the hair itself.
On the flipside, the use of a mesh for shadows had some disadvantages, such as; increased man-hours to develop the meshes, and deviation from the toon-shader shader colouration.
To correct for deviation in future, the team plans to test out a ramp shift formula
that adds the shadow colour as part of the mesh’s colour.
Outlines and Skin Bloom
In addition to the face and hair adjustments, other changes were made such as outline and skin bloom.
Outlines
The outlines in this game are created using back-facing11. It is controlled via a dedicated vertex colour, with RGB representing the colour of the outline and A representing its thickness.
If outlines are on, Mio’s chin and other extremities are emphasised.
Thickness is proportional to screen occupancy, such that “close-ups are not too thick” and “faraways are not too thin”, with the outline being turned off when screen occupancy drops below a certain threshold.
Skin Bloom
In order to make skin exposed to strong light appear soft and luminous, the team implemented a skin bloom formula. This is a skin-only version of general diffusion12.
Skin bloom showing a soft luminosity.
When implementing skin bloom, it is integrated into existing bloom passes to reduce GPU calls.
The lighting result, skin bloom luminance, and pixel luminance are combined as shown above to obtain a high luminance result for the final bloom. Subsequent processing is simply blurring and superimposing as with normal bloom.
Tips for skin bloom implementation were also presented above, including “adjusting luminance according to camera distance and viewing angle”,
“avoiding skin tone influence on the luminance”, and “storing skin bloom luminance in an HDR format”.
- Unlike tiled shading, which creates groups of rectangular tiles in screen space, clustered shading compiles a series of light sources in clusters that are finer in quality than tiles. Only light sources that affect the given cluster are calculated, and up to 1 million light sources can be processed in real-time. This has a lower shading cost and allows for more complex shader techniques. ↩︎
- Geometry buffer, data that stores geometry and material information, such as textures, in the first pass of deferred shading. ↩︎
- A drawing style used in painting and concept art where multiple parallel and perpendicular lines are used for shading. ↩︎
- Non-Photorealistic Rendering, a rendering method that gives a style that is “not realistic” compared to photorealistic rendering. Examples include cartoons, illustrations, animation, watercolours, paintings, and 3DCG. ↩︎
- A normal that determines how shadows are cast and affects the appearance of a object’s surface; there are as many normals as there are surfaces adjacent to a single vertex, and they all converge in one direction when smooth shading is applied. ↩︎
- A technique that uses texture information give the illusion of depth upon a flat surface. The RGB channels are used to change the normal’s “direction” for a given pixel. ↩︎
- Mesh design and organisation in modelling ↩︎
- Special textures that including images of the surrounding scene and lighting effects around the object. ↩︎
- A phenomenon in which light appears to seep through thin layers of hair or skin when the object is between the viewpoint and the light source when exposed to relative backlight. ↩︎
- Symmetric Nearest Neighbour Filter, a process that calculated the average value from neighbouring pixels and then smooths the details to appear animated or hand-drawn. ↩︎
- A method of drawing outlines by extruding the mesh relative to its normals and applying a material that does not render the backside. ↩︎
- To diffuse reflected light to create a soft appearance. ↩︎