20151116 - Mixing Temporal AA and Transparency

Jon Greenberg asks on twitter, "Okay, so here's the TemporalAA question of the day - transparency isn't TAA'd - how do you manage the jittered camera? Ignore it? Oy..."

The context of this question is often the following graphics pipeline,

(1.) Render low-poly proxy geometry for some of the major occluders in a depth pre-pass.
(2.) Render g-buffer without MSAA, each frame using a different jittered sub-pixel viewport offset.
(3.) Render transparency (without viewport jitter) in a separate render target.
(4.) Later apply temporal AA to opaque and somehow composite over the separate transparent layer.

Here are some ideas on paths which might solve the associated problems,

Soft Transparency Only
If the transparent layer has soft-particle depth intersection only (no triangle windows, etc), then things are a lot easier. Could attempt to apply temporal AA to the depth buffer, creating a "soft" depth buffer where edges are part eroded towards far background neighborhood of a pixel. Then do a reduction on this "soft" depth buffer, getting smaller resolution near and far depth value for the local neighborhood (with some overlap between neighborhoods). Then render particles into two smaller resolution color buffers (soft blending a particle to both near and far layers). Can use the far depth reduction as the Z buffer to test against. Later composite into the back-buffer over the temporal AA using the "soft" full-res depth buffer to choose a value between the colors in "near" and "far". Note there is an up-sample involved inline in this process, and various quality/performance trade-offs in how this combined up-sample/blend/composite operation happens. I say "back-buffer" because don't want to feedback the transparency into the next temporal AA pass.

Hard Transparency
Meaning what to do about windows, glasses, and other things which require full-resolution hard intersections with other opaque geometry. Any working solution here also needs an anti-aliased mask for post temporal AA composite. There are no great solutions to my knowledge with the traditional raster based rendering pipelines with viewport jitter. One option is to work around the problem in the art side, to make glass surfaces mostly opaque and render with matching viewport jitter over the lit g-buffer, also correcting so reprojection or motion vectors pick up the new glass surface instead of what is behind it. So glass goes down the temporal AA path.

Another option might be to use the "soft" depth buffer technique but at full resolution. Probably need to build a full resolution "far" erosion depth buffer (take the far depth of the local neighborhood), then depth test against that. Note depth buffer generated by a shader will have an associated perf cost when tested against. Then when rendering transparency can blend directly over the temporal AA output in the back-buffer. In the shader, fetch the pre-generated "near" and "far" reductions, and soft-blend the hard triangle with both. Then take those two results, lookup the "soft" depth from the full resolution, and use as a guide to lerp between the "near" and "far" result. This will enable a "soft" anti-aliased edge, in theory, assuming all the details that matter are correct...

Note on "Soft" Depth
The "soft" depth probably requires that temporal AA not be applied to linear depth, but perhaps some non-linear function of depth. I don't remember any more which transform works the best, but guessing if you take this transformed depth and output it to a color channel and see a clear depth version of the scene with anti-aliased edges, from nearest to far objects, that is a good sign.