Note: The first thing on my list for the next Unity article was the decal system, but not only the code would need some serious revamp to be presentable, what would take some time, but unfortunately I lost the disk with the latest version in the move, I have an old backup but I still hope to find the disk somewhere, so it may take some time to publish something regarding the decal system, sorry if you were looking forward to it.

IMPORTANT: The material I’ve used for testing has a rough normal map that causes the “jagged” edges in the cel shading you see in the screenshots. I like that, but if you don’t, don’t worry, this is still relevant if you want smooth edges in the shading. TIP: the CalculateLight2 function produces extra-smooth shade lines.

This article shows how to setup custom lighting for the deferred render path in Unity, and while it may be useful if you’re interested in other types of custom lighting, it shows specifically how to have decent cel shading either calculated or optionally with ramp lookup from a bitmap.

Most if not all devs love deferred rendering… being able to have lots of dynamic lights in the scene without having to render each one of them independently is fantastic, there are however some drawbacks when using deferred rendering, most notably lack of anti-aliasing and transparency (which in Unity is rendered separately in forward path).

Another caveat in deferred rendering is that you can’t have custom lighting on a per-shader basis, since the lighting is processed all at once in a separated pass. Fortunately it is possible to customize that in Unity, however, the articles out there are either old, seriously incomplete, or just plain wrong. The official documentation also is lacking, (what’s the case for almost everything beyond the absolute basics).

Without further ado, let’s start customizing our deferred lighting:

The easiest way to start customizing the lighting is by modifying the default one, to do that download the built-in shaders from here: unity3d.com/get-unity/download/archive. Inside the zip archive there’s a file DefaultResourcesExtra/Internal-DeferredShading.shader, that’s the default deferred lighting that ships with Unity, copy that file to your Assets folder and optionally rename it to something else.

Now, open that file and also change the shader name to avoid confusion since the name Unity uses is the one in the file, not it’s filename:

Rename to something like:

So it shows properly in the Unity editor.

OK, now we have to enable deferred rendering in our project and also point Unity to that shader file so it knows what to use to calculate the lighting.

In Unity 5.5 or newer the rendering path isn’t in the player settings; now there are “Tiers” for configuration and it’s in the Graphics settings, just untick the default from all of them and select “Deferred” for the rendering path:

You may notice the lighting in the scene slightly change.

Immediately below those settings, you’ll find the Built-in shader settings, that’s where you have to configure Unity to use your custom shader for the deferred rendering, click the dropdown that says “Deferred” and select “Custom shader”, a field will appear where you can select the shader you just renamed:

Nothing is going to change when you do that because at the moment it does the same as the Built-in shader.

Everything is now setup to start playing with the deferred lighting, let me explain some basics of the default shader we have at the moment, we are only interested in the first pass, which looks like this:

The special function “frag” is the one that’s called by Unity, in this case it basically calls yet another function CalculateLight, which does the heavy lifting. The HDR_ON check should be self-explanatory.

A quick modification to check if everything is working is to make the CalculateLight return one of the gbuffers directly (instead of res), for example:

Which should result in an opaque image. These gbuffer variables are sampled from the camera (output) textures, basically gbuffer0 is the diffuse color, gbuffer1 the specular color, and gbuffer2 the normal direction.

On the code above you can see that the shader is calling UNITY_BRDF_PBS, which is the function that calculates the physically correct surface lighting distribution. This function is certainly not cheap, and for a toon shading you probably could go with a less accurate model, but using the output of that function can result in a very decent shader with very little effort and most importantly, a shader that’s “PBR-friendly”, in other words, would work somewhat consistently and you could take advantage of the settings of the new “Standard” shader for the materials.

The variable “res” includes the full image output, so we can easily extract its intensity just by summing up all the channels values, we can use something like this

And divide by some arbitrary value in order to extract “bands” for different intensities, effectively emulating a kind of “toon” effect. To have the colors we can simply multiply it by the sum of gbuffer0 and gbuffer1:

Here’s a one-line modification that will yield a quite decent result (in the CalculateLight return):

Result:

A slightly more elaborated version could look like this for instance:

Note how it’s approximately consistent in intensity with the default lighting. We’re not using the gbuffers directly this time but the information in the UnityStandardData object. I’m not sure in which cases they should differ, but I guess using the latter is the safest bet.

You may want to sum the diffuse and specular (even after nonlinear operations) to roughly maintain the energy conservation and keep it consistent with the Standard PBR shader settings. Of course in this case it’s not important to actually keep it physically correct, but at very least we want to prevent unpractical results (like absolute black or white materials). If you change the proportion of their influences you may break the energy conservation, what’s not a problem. Note that I’m using ceil so technically I should divide by 3 since it’s 3 channels so at max that sum would be 3 (although it could be 4 due float imprecision), but I’m using an arbitrary number that make it look a bit better. However, I’m not happy at all with the saturation in the image. The lighting also doesn’t look good. In an attempt to achieve a more cartoonish look, with more saturation but still a bit of gradients, I ended up with the following code:

Note how by simply separating the intensities for diffuse and specular (although the input is the same), you have extra control of the final result, and to add a bit of gradient I’m simply using the UNITY_BRDF_PBS output directly (last line). I’m satisfied with the result:

That’s exactly the kind of effect I was trying to achieve, not just something that looks a posterization filter, but something that actually improve it somehow. Normalizing the diffuse color is responsible for most of the extra saturation you see here. While that’s not a very orthodox approach, the final result has a completely different mood and makes some details more noticeable: especially the normal map details look way more pronounced. Aliasing is starting to get very annoying though, while we could tackle that on the shading lines, you can’t have proper AA on geometry, so a post processing filter is generally used for these cases (like FXAA).

Compatibility with the Standard shader is also a huge plus, look for example these different settings:

Perfect insulator:

Metallic (0.8, 0.8):

The next step is sampling the lighting ramp from a texture. First thing you need obviously is the texture ramp, here’s the one I’m using:

It’s not perfect but for example purposes it’s fine. Note that the visible bands on the edges are intentional, another trick is to apply a little bit of gradient in each shade, but in the inverse direction of the intensity, so the edges get more noticeable for an extra toon effect. Make sure the wrap mode is set to clamp, otherwise depending on the filtering the edges will interpolate in a wrap-around way causing artifacts in the final image.

Using that texture is a little bit tricky, and that information isn’t easy to find, in fact in multiple places people suggested that you could create a material property, select your shader file, and configure the texture in the inspector, but that simply doesn’t work. In fact, I don’t even know why the inspector shows controls when you select code in the project view since that doesn’t seem to have any use whatsoever.

To sample a texture, you have to set it using the Shader.SetGlobalTexture() function, the tricky part is that you can’t setup a material property for it in the shader, otherwise it won’t work. You just have to declare the texture somewhere like this:

And set it like this from a script:

Then you can proceed to sample it in the shader file, here’s my sampling function:

By using textures you have more freedom and flexibility to achieve specific custom lighting, here is an example of what I’ve managed to achieve (exactly same material settings, just deferred lighting changed):

The lighting above (SampleToon function) works pretty well with the Standard shader settings, here are some examples of what you can achieve just by changing the conductivity/glossiness:

INSULATOR

METALLIC

RANDOM PICK

I’m including below the project with lots of goodies, including NON-PR lighting approaches for smoother shade transition lines (although in my opinion the lines are pretty good using the PBR BDRF output as intensity), and also some variations, including some optional optimizations you could use, besides the “MyDeferredShading”, the other shader consists basically of this:

Note that the “frag” function has a lot of commented lines, each one represents a different lighting method so you can quickly switch between them in the code just by uncommenting the one you want. Read the comments on each line for information on Standard shader support and HDR.

Here is the link to download the full project used to make this article: Deferred Toon

I hope this article was useful, please consider following me on twitter for more stuff like this. Feel free to ask any questions either here or on twitter too. Thanks for reading.