20120821 - Alpha-Test and MSAA


An easy and fast way to provide MSAA anti-aliasing with alpha-tested surfaces is to write the coverage mask with the results of in-shader super-sampling the software alpha-test.

Tools DX|GL
EvaluateAttributeAtSample() | interpolateAtSample() - Interpolate attribute at sample location.

SV_COVERAGE | gl_SampleMask - Provides pre-Z+stencil coverage mask on input, and writable sample coverage mask on output. Note this value is automatically ANDed to the post-Z+stencil coverage mask after the pixel|fragment shader exits, so no need to manually do this.

clip() | discard - Exit pixel|fragment shader without writing any output.

Forward Shading Without Alpha-Test Surfaces in Z-Pre-Pass
In the pixel|fragment shader fetch from the alpha-test texture at each sample position, and update the coverage mask. Only discard if all samples fail the software alpha-test (if generated coverage mask is zero). Otherwise shade pixel as normal.

Forward Shading With Alpha-Test Surfaces in Z-Pre-Pass
I'm mostly sure that z-pre-pass with alpha-tested surfaces is a bad idea in general, but just in case someone is doing this, in the z-pre-pass set the coverage mask, but later when shading use depth equals depth-test and skip the software alpha-test.

Deferred Lighting
In the z+normal+specPower-pre-pass set the coverage mask, and later when shading use depth equals depth-test and skip the software alpha-test.

Deferred Shading
In the g-buffer creation set the coverage mask.