Note: The first thing on my list for the next Unity article was the decal system, but not only the code would need some serious revamp to be presentable, what would take some time, but unfortunately I lost the disk with the latest version in the move, I have an old backup but I still hope to find the disk somewhere, so it may take some time to publish something regarding the decal system, sorry if you were looking forward to it.
IMPORTANT: The material I’ve used for testing has a rough normal map that causes the “jagged” edges in the cel shading you see in the screenshots. I like that, but if you don’t, don’t worry, this is still relevant if you want smooth edges in the shading. TIP: the CalculateLight2 function produces extra-smooth shade lines.
This article shows how to setup custom lighting for the deferred render path in Unity, and while it may be useful if you’re interested in other types of custom lighting, it shows specifically how to have decent cel shading either calculated or optionally with ramp lookup from a bitmap.
Most if not all devs love deferred rendering… being able to have lots of dynamic lights in the scene without having to render each one of them independently is fantastic, there are however some drawbacks when using deferred rendering, most notably lack of anti-aliasing and transparency (which in Unity is rendered separately in forward path).
Another caveat in deferred rendering is that you can’t have custom lighting on a per-shader basis, since the lighting is processed all at once in a separated pass. Fortunately it is possible to customize that in Unity, however, the articles out there are either old, seriously incomplete, or just plain wrong. The official documentation also is lacking, (what’s the case for almost everything beyond the absolute basics).
Without further ado, let’s start customizing our deferred lighting:
The easiest way to start customizing the lighting is by modifying the default one, to do that download the built-in shaders from here: unity3d.com/get-unity/download/archive. Inside the zip archive there’s a file DefaultResourcesExtra/Internal-DeferredShading.shader, that’s the default deferred lighting that ships with Unity, copy that file to your Assets folder and optionally rename it to something else.
Now, open that file and also change the shader name to avoid confusion since the name Unity uses is the one in the file, not it’s filename:
1 |
Shader "Hidden/Internal-DeferredShading" { |
Rename to something like:
1 |
Shader "MyDeferredShading" { |
So it shows properly in the Unity editor.
OK, now we have to enable deferred rendering in our project and also point Unity to that shader file so it knows what to use to calculate the lighting.
In Unity 5.5 or newer the rendering path isn’t in the player settings; now there are “Tiers” for configuration and it’s in the Graphics settings, just untick the default from all of them and select “Deferred” for the rendering path:
You may notice the lighting in the scene slightly change.
Immediately below those settings, you’ll find the Built-in shader settings, that’s where you have to configure Unity to use your custom shader for the deferred rendering, click the dropdown that says “Deferred” and select “Custom shader”, a field will appear where you can select the shader you just renamed:
Nothing is going to change when you do that because at the moment it does the same as the Built-in shader.
Everything is now setup to start playing with the deferred lighting, let me explain some basics of the default shader we have at the moment, we are only interested in the first pass, which looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
#pragma target 3.0 #pragma vertex vert_deferred #pragma fragment frag #pragma multi_compile_lightpass #pragma multi_compile ___ UNITY_HDR_ON #pragma exclude_renderers nomrt #include "UnityCG.cginc" #include "UnityDeferredLibrary.cginc" #include "UnityPBSLighting.cginc" #include "UnityStandardUtils.cginc" #include "UnityGBuffer.cginc" #include "UnityStandardBRDF.cginc" sampler2D _CameraGBufferTexture0; sampler2D _CameraGBufferTexture1; sampler2D _CameraGBufferTexture2; half4 CalculateLight (unity_v2f_deferred i) { float3 wpos; float2 uv; float atten, fadeDist; UnityLight light; UNITY_INITIALIZE_OUTPUT(UnityLight, light); UnityDeferredCalculateLightParams (i, wpos, uv, light.dir, atten, fadeDist); light.color = _LightColor.rgb * atten; // unpack Gbuffer half4 gbuffer0 = tex2D (_CameraGBufferTexture0, uv); half4 gbuffer1 = tex2D (_CameraGBufferTexture1, uv); half4 gbuffer2 = tex2D (_CameraGBufferTexture2, uv); UnityStandardData data = UnityStandardDataFromGbuffer(gbuffer0, gbuffer1, gbuffer2); float3 eyeVec = normalize(wpos-_WorldSpaceCameraPos); half oneMinusReflectivity = 1 - SpecularStrength(data.specularColor.rgb); UnityIndirect ind; UNITY_INITIALIZE_OUTPUT(UnityIndirect, ind); ind.diffuse = 0; ind.specular = 0; half4 res = UNITY_BRDF_PBS (data.diffuseColor, data.specularColor, oneMinusReflectivity, data.smoothness, data.normalWorld, -eyeVec, light, ind); return res; } #ifdef UNITY_HDR_ON half4 #else fixed4 #endif frag (unity_v2f_deferred i) : SV_Target { half4 c = CalculateLight(i); #ifdef UNITY_HDR_ON return c; #else return exp2(-c); #endif } |
The special function “frag” is the one that’s called by Unity, in this case it basically calls yet another function CalculateLight, which does the heavy lifting. The HDR_ON check should be self-explanatory.
A quick modification to check if everything is working is to make the CalculateLight return one of the gbuffers directly (instead of res), for example:
1 |
return gbuffer0; |
Which should result in an opaque image. These gbuffer variables are sampled from the camera (output) textures, basically gbuffer0 is the diffuse color, gbuffer1 the specular color, and gbuffer2 the normal direction.
On the code above you can see that the shader is calling UNITY_BRDF_PBS, which is the function that calculates the physically correct surface lighting distribution. This function is certainly not cheap, and for a toon shading you probably could go with a less accurate model, but using the output of that function can result in a very decent shader with very little effort and most importantly, a shader that’s “PBR-friendly”, in other words, would work somewhat consistently and you could take advantage of the settings of the new “Standard” shader for the materials.
The variable “res” includes the full image output, so we can easily extract its intensity just by summing up all the channels values, we can use something like this
1 |
res.r+res.g+res.b |
And divide by some arbitrary value in order to extract “bands” for different intensities, effectively emulating a kind of “toon” effect. To have the colors we can simply multiply it by the sum of gbuffer0 and gbuffer1:
Here’s a one-line modification that will yield a quite decent result (in the CalculateLight return):
1 |
return (gbuffer0+gbuffer1)*(floor((res.r+res.g+res.b)*2))/2; |
Result:
A slightly more elaborated version could look like this for instance:
1 2 3 4 |
half3 intensity = ceil(res.r+res.g+res.b)/5; half3 lighting = light.color * intensity; half3 litCol= (data.diffuseColor + data.specularColor) * lighting; return half4(litCol, res.a); |
Note how it’s approximately consistent in intensity with the default lighting. We’re not using the gbuffers directly this time but the information in the UnityStandardData object. I’m not sure in which cases they should differ, but I guess using the latter is the safest bet.
You may want to sum the diffuse and specular (even after nonlinear operations) to roughly maintain the energy conservation and keep it consistent with the Standard PBR shader settings. Of course in this case it’s not important to actually keep it physically correct, but at very least we want to prevent unpractical results (like absolute black or white materials). If you change the proportion of their influences you may break the energy conservation, what’s not a problem. Note that I’m using ceil so technically I should divide by 3 since it’s 3 channels so at max that sum would be 3 (although it could be 4 due float imprecision), but I’m using an arbitrary number that make it look a bit better. However, I’m not happy at all with the saturation in the image. The lighting also doesn’t look good. In an attempt to achieve a more cartoonish look, with more saturation but still a bit of gradients, I ended up with the following code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
half4 CalculateLight (unity_v2f_deferred i) { float3 wpos; float2 uv; float atten, fadeDist; UnityLight light; UNITY_INITIALIZE_OUTPUT(UnityLight, light); UnityDeferredCalculateLightParams (i, wpos, uv, light.dir, atten, fadeDist); light.color = _LightColor.rgb * atten; // unpack Gbuffer half4 gbuffer0 = tex2D (_CameraGBufferTexture0, uv); half4 gbuffer1 = tex2D (_CameraGBufferTexture1, uv); half4 gbuffer2 = tex2D (_CameraGBufferTexture2, uv); UnityStandardData data = UnityStandardDataFromGbuffer(gbuffer0, gbuffer1, gbuffer2); float3 eyeVec = normalize(wpos-_WorldSpaceCameraPos); half oneMinusReflectivity = 1 - SpecularStrength(data.specularColor.rgb); UnityIndirect ind; UNITY_INITIALIZE_OUTPUT(UnityIndirect, ind); ind.diffuse = 0; ind.specular = 0; half4 res = UNITY_BRDF_PBS (data.diffuseColor, data.specularColor, oneMinusReflectivity, data.smoothness, data.normalWorld, -eyeVec, light, ind); half intensity = clamp(floor((res.r+res.g+res.b)*4),0,1); half3 lighting = normalize(light.color) * intensity; half3 corrDiff = normalize(data.diffuseColor); half3 litDiff = corrDiff * lighting; half sintensity = floor((res.r+res.g+res.b)/3); half3 slighting = normalize(light.color) * 0.6 + light.color * 0.4 * sintensity; half3 corrSpec = data.specularColor; half3 litSpec = corrSpec * slighting; return half4(litDiff+litSpec*0.7 + res*0.3, res.a); } |
Note how by simply separating the intensities for diffuse and specular (although the input is the same), you have extra control of the final result, and to add a bit of gradient I’m simply using theĀ UNITY_BRDF_PBS output directly (last line). I’m satisfied with the result:
That’s exactly the kind of effect I was trying to achieve, not just something that looks a posterization filter, but something that actually improve it somehow. Normalizing the diffuse color is responsible for most of the extra saturation you see here. While that’s not a very orthodox approach, the final result has a completely different mood and makes some details more noticeable: especially the normal map details look way more pronounced. Aliasing is starting to get very annoying though, while we could tackle that on the shading lines, you can’t have proper AA on geometry, so a post processing filter is generally used for these cases (like FXAA).
Compatibility with the Standard shader is also a huge plus, look for example these different settings:
Perfect insulator:
Metallic (0.8, 0.8):
The next step is sampling the lighting ramp from a texture. First thing you need obviously is the texture ramp, here’s the one I’m using:
It’s not perfect but for example purposes it’s fine. Note that the visible bands on the edges are intentional, another trick is to apply a little bit of gradient in each shade, but in the inverse direction of the intensity, so the edges get more noticeable for an extra toon effect. Make sure the wrap mode is set to clamp, otherwise depending on the filtering the edges will interpolate in a wrap-around way causing artifacts in the final image.
Using that texture is a little bit tricky, and that information isn’t easy to find, in fact in multiple places people suggested that you could create a material property, select your shader file, and configure the texture in the inspector, but that simply doesn’t work. In fact, I don’t even know why the inspector shows controls when you select code in the project view since that doesn’t seem to have any use whatsoever.
To sample a texture, you have to set it using the Shader.SetGlobalTexture() function, the tricky part is that you can’t setup a material property for it in the shader, otherwise it won’t work. You just have to declare the texture somewhere like this:
1 |
sampler2D ToonRamp; |
And set it like this from a script:
1 |
Shader.SetGlobalTexture("ToonRamp", ToonRamp); |
Then you can proceed to sample it in the shader file, here’s my sampling function:
1 2 3 4 |
half3 GetToonRampColor(half intensity) { return tex2D(ToonRamp, half2(clamp(intensity,0,1), .5)); } |
By using textures you have more freedom and flexibility to achieve specific custom lighting, here is an example of what I’ve managed to achieve (exactly same material settings, just deferred lighting changed):
The lighting above (SampleToon function) works pretty well with the Standard shader settings, here are some examples of what you can achieve just by changing the conductivity/glossiness:
INSULATOR
METALLIC
RANDOM PICK
I’m including below the project with lots of goodies, including NON-PR lighting approaches for smoother shade transition lines (although in my opinion the lines are pretty good using the PBR BDRF output as intensity), and also some variations, including some optional optimizations you could use, besides the “MyDeferredShading”, the other shader consists basically of this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 |
#pragma target 3.0 #pragma vertex vert_deferred #pragma fragment frag #pragma multi_compile_lightpass #pragma multi_compile ___ UNITY_HDR_ON #pragma exclude_renderers nomrt #include "UnityCG.cginc" #include "UnityDeferredLibrary.cginc" #include "UnityPBSLighting.cginc" #include "UnityStandardUtils.cginc" #include "UnityStandardBRDF.cginc" sampler2D _CameraGBufferTexture0;sampler2D _CameraGBufferTexture1;sampler2D _CameraGBufferTexture2; half4 CalculateLight (unity_v2f_deferred i) {float3 wpos;float2 uv;float atten, fadeDist;UnityLight light;UNITY_INITIALIZE_OUTPUT(UnityLight, light);UnityDeferredCalculateLightParams (i, wpos, uv, light.dir, atten, fadeDist);light.color = _LightColor.rgb * atten;half4 gbuffer0 = tex2D (_CameraGBufferTexture0, uv);half4 gbuffer1 = tex2D (_CameraGBufferTexture1, uv);half4 gbuffer2 = tex2D (_CameraGBufferTexture2, uv);UnityStandardData data = UnityStandardDataFromGbuffer(gbuffer0, gbuffer1, gbuffer2);float3 eyeVec = normalize(wpos-_WorldSpaceCameraPos);half oneMinusReflectivity = 1 - SpecularStrength(data.specularColor.rgb);UnityIndirect ind;UNITY_INITIALIZE_OUTPUT(UnityIndirect, ind);ind.diffuse = 0;ind.specular = 0;half4 res = UNITY_BRDF_PBS (data.diffuseColor, data.specularColor, oneMinusReflectivity, data.smoothness, data.normalWorld, -eyeVec, light, ind);return res;} //################ // CUSTOM LIGHTING //################ struct cel { half3 base; half3 diff; half3 spec; }; cel CalculateLight2 (unity_v2f_deferred i) //NON PB { //DEFAULT COPYPASTA float3 wpos;float2 uv;float atten, fadeDist;UnityLight light;UNITY_INITIALIZE_OUTPUT(UnityLight, light);UnityDeferredCalculateLightParams (i, wpos, uv, light.dir, atten, fadeDist);half4 gbuffer0 = tex2D (_CameraGBufferTexture0, uv);half4 gbuffer1 = tex2D (_CameraGBufferTexture1, uv);half4 gbuffer2 = tex2D (_CameraGBufferTexture2, uv);light.color = _LightColor.rgb * atten; //END COPYPASTA half3 baseColor = gbuffer0.rgb; half3 specColor = gbuffer1.rgb; half3 normalWorld = gbuffer2.rgb * 2 - 1; float3 eyeVec = wpos-_WorldSpaceCameraPos; half3 halfDirection = normalize(light.dir-eyeVec); half spec = dot(normalWorld, halfDirection); half diff = dot(normalWorld, light.dir); cel bds; bds.base = gbuffer0.rgb; bds.diff = light.color * diff; bds.spec = light.color * spec * specColor; return bds; } struct pbsIntAndDiff { half4 intensity; half3 baseCol; }; pbsIntAndDiff CalculateLight3 (unity_v2f_deferred i) { //DEFAULT COPYPASTA float3 wpos;float2 uv;float atten, fadeDist;UnityLight light;UNITY_INITIALIZE_OUTPUT(UnityLight, light);UnityDeferredCalculateLightParams (i, wpos, uv, light.dir, atten, fadeDist);light.color = _LightColor.rgb * atten;half4 gbuffer0 = tex2D (_CameraGBufferTexture0, uv);half4 gbuffer1 = tex2D (_CameraGBufferTexture1, uv);half4 gbuffer2 = tex2D (_CameraGBufferTexture2, uv);UnityStandardData data = UnityStandardDataFromGbuffer(gbuffer0, gbuffer1, gbuffer2);float3 eyeVec = normalize(wpos-_WorldSpaceCameraPos);half oneMinusReflectivity = 1 - SpecularStrength(data.specularColor.rgb);UnityIndirect ind;UNITY_INITIALIZE_OUTPUT(UnityIndirect, ind);ind.diffuse = 0;ind.specular = 0; //END COPYPASTA half4 res = UNITY_BRDF_PBS (data.diffuseColor, data.specularColor, oneMinusReflectivity, data.smoothness, data.normalWorld, -eyeVec, light, ind); pbsIntAndDiff ret; ret.baseCol = (data.specularColor-0.25)*1.5+data.diffuseColor*0.75; // Adjustment for metallic workflow. // For specular workflow, just sum spec and diff. half3 ic = (res.r+res.g+res.b)/3; half3 lcm = 1.25 + (1 - (light.color.r+light.color.g+light.color.b)/3); ret.intensity = half4(light.color*ic*lcm*2,res.a); return ret; } //################# // CALCULATED TOON //################# half ToonizeCalcSingle(half val, uint steps) { return (floor(val*steps))/steps; } half3 ToonizeCalc(half3 vals, uint steps) { return half3(ToonizeCalcSingle(vals.r,steps), ToonizeCalcSingle(vals.g,steps), ToonizeCalcSingle(vals.b,steps)); } half4 CalcToon(unity_v2f_deferred i) { half4 res = CalculateLight(i); return half4(ToonizeCalc(res*1.5,4),res.a); } half4 CalcToon2(unity_v2f_deferred i) { cel res = CalculateLight2(i); return half4(res.base * ToonizeCalcSingle(clamp(res.diff*0.5,(res.spec-0.15)*2,1)*2.5,4), 1)*2; } half4 CalcToon3(unity_v2f_deferred i) { cel res = CalculateLight2(i); return half4(res.base * clamp(ToonizeCalc(res.diff,4),(ToonizeCalc(res.spec-0.15,3))*3,1)*1.5, 1)*2.5; } //##################### // TEXTURE SAMPLED TOON //##################### sampler2D ToonRamp; half3 GetToonRampColor(half intensity) { return tex2D(ToonRamp, half2(clamp(intensity,0,1), .5)); } half3 ToonizeSampleIndividual(half3 vals) { return half3(GetToonRampColor(vals.r).r, GetToonRampColor(vals.g).g, GetToonRampColor(vals.b).b); } half3 ToonizeSampleFast(half3 vals) { return GetToonRampColor((vals.r+vals.g+vals.b)/3)*saturate(normalize(vals))*2; } half3 ToonizeSampleSingle(half3 val) { return GetToonRampColor((val.r+val.g+val.b)/3)*3; } half4 SampleToon(unity_v2f_deferred i) { pbsIntAndDiff res = CalculateLight3(i); half4 celLight = half4(ToonizeSampleIndividual(res.intensity.rgb),1); // Alternatively use ToonizeSampleFast return half4(celLight.rgb+res.baseCol, res.intensity.a); } half4 SampleToon2(unity_v2f_deferred i) { half4 res = CalculateLight(i); return half4(res.rgb*ToonizeSampleSingle(res)*1.25,res.a); } half4 SampleToon3(unity_v2f_deferred i) { cel res = CalculateLight2(i); return half4(res.base*(ToonizeSampleIndividual(clamp(res.diff, (res.spec-0.25)*4, 1)+(res.spec-0.5))),1)*3; } //##################### // ACTUAL FRAG FUNCTION //##################### #ifdef UNITY_HDR_ON half4 #else fixed4 #endif frag (unity_v2f_deferred i) : SV_Target { // half4 c = CalculateLight(i); // this is the default lighting for comparison half4 c = SampleToon(i); // Standard-shader-ready high-quality toon using Unity's PB BRDF and ramp texture // half4 c = SampleToon2(i); // alternative toon with gradients using Unity's PB BRDF and ramp texture (bad HDR support) // half4 c = SampleToon3(i); // basic NON-PB toon shading with texture ramp // half4 c = CalcToon(i); // toon shading calculated in shader using Unity's PB BRDF (no texture ramp) // half4 c = CalcToon2(i); // basic NON-PB toon shading with calculated ramp (no texture ramp, no light color, no HDR support) // half4 c = CalcToon3(i); // similar to the one above but fancier and supports colored lights (kinda costly, no texture ramp, no HDR) #ifdef UNITY_HDR_ON return c; #else return exp2(-c); #endif } |
Note that the “frag” function has a lot of commented lines, each one represents a different lighting method so you can quickly switch between them in the code just by uncommenting the one you want. Read the comments on each line for information on Standard shader support and HDR.
Here is the link to download the full project used to make this article: Deferred Toon
I hope this article was useful, please consider following me on twitter for more stuff like this. Feel free to ask any questions either here or on twitter too. Thanks for reading.
Nothing happens in Unity 2017…
I’m sorry you couldn’t get it to work. This was only tested in Unity 5.4/5.5 and even though the project may be broken in version 2017, the tutorial and shader code should still be relevant for Unity 2017. I’m going to update this at some point.
Can I just say: this is awesome – thanks dude!
Has saved me tonnes of experimentation and head scratching and given me a working shader to use as a basis for my own stylistic noodling. I owe you a beverage of some sorts if we ever meet in real life.
Thank you for your kind words! I’m glad the shaders were useful! š