December 2nd, 2006

Alpha-Tested Shadows

One of the greatest drawbacks to stencil shadows is the inability to properly shadow alpha-tested objects; that is to say, an object with a transparent texture. Take, for example, a chain link fence. Many FPS game involve fences of some kind in one area or another, and in most cases these fences are nothing more than a simple quad with an alpha texture applied. It looks great, and it’s cheap, however it falls apart as soon as you attempt to shadow it. Stencils work by extruding the silloughete of a mesh, and since the transparency is a texture effect and not part of the mesh geometry, it isn’t possible to capture that in the stencil buffer.

However, this problem is resolved with shadowmapping. Because an object is draw into the depth buffer pixel by pixel, it is possible to do a lookup on the object’s diffuse texture, and determine whether or not the alpha component is solid or not. If it is, then continue to mark the depth. If not, kill the pixel shader (texkill is wonderful) and do not write out a depth (no shadow). It works great!

void VS(in float3 pos: POSITION, float2 texCoord: TEXCOORD0, out vsOut output) {

     output.pos = mul(float4(pos, 1.0), world);
     output.depth = length( -;
    output.pos = mul(output.pos, viewProj);

 output.texCoord = texCoord;


float4 PS(in float depth: TEXCOORD0, float2 texCoord: TEXCOORD1) : COLOR {
 clip(tex2D(DiffuseMapSampler, texCoord).a - 0.5f);

     return float4(depth, 0.0, 0.0, 1.0);

 As you can see here, it’s simple enough to implement. There is a rather major drawback though - texkill (clip in HLSL) interferes with a 3d card’s parallalism, and combined with an extra texture read, really chews up performance. However I’m not too worried about this - very few objects in the world will be required to use this special shader vs. the standard depth map shader.

December 1st, 2006

Not Forgotten