Shader question regarding multiple passes, render targets and screenaligned quads

Jun 22, 2010 at 1:56 PM
Edited Jun 22, 2010 at 2:09 PM
I am working on an outline shader to use for my game. I've got all the techniques working (by working I mean they render what I want them to render when I use them one by one), but when it comes to merging them together to get the desired effect, it wasn't as easy as I thought.
The first technique (OutlineAnimated) renders the model using only white colour onto a black background, I then use that texture to create the outline texture in the second technique (EdgeAnimated), after that I want to render the model again using the model's texture and whatever lighting effects I would like to use along with the outline around the model (in the third technique, Projection), but that's where things go wrong. All I get drawn on the screen is the outline texture. I've used the same approach to creating this shader as is used in the book "Shaders for game programmers and artists", I've also tried my shader out in Rendermonkey and get it working just fine. I do use a screenaligned quad when rendering the edge texture in rendermonkey though, do I have to use a different approach when creating this effect in XEN?
This is the shader code:
float4x4 worldViewProj : WORLDVIEWPROJECTION;
float4x4 world : WORLD;
float4 viewPoint : VIEWPOINT;

texture2D DiffuseTexture;
sampler2D textureSampler = sampler_state
Texture = (DiffuseTexture);

texture2D OutlineTexture;
sampler2D outlineSampler = sampler_state
Texture = (OutlineTexture);

float4 blendMatrices[72*3];

void AnimatedOutlineVS(float4 position : POSITION,
out float4 positionOut : POSITION,
float2 texcoordIn : TEXCOORD0,
out float2 texcoordOut : TEXCOORD0,
float3 normal : NORMAL,
out float3 normalOut : TEXCOORD1,
out float3 viewDirOut : TEXCOORD3,
float4 weights : BLENDWEIGHT,
int4 indices : BLENDINDICES)
float4x3 blendMatrix
= transpose(float3x4(
blendMatrices[indices.x*3+0] * weights.x + blendMatrices[indices.y*3+0] * weights.y + blendMatrices[indices.z*3+0] * weights.z + blendMatrices[indices.w*3+0] * weights.w,
blendMatrices[indices.x*3+1] * weights.x + blendMatrices[indices.y*3+1] * weights.y + blendMatrices[indices.z*3+1] * weights.z + blendMatrices[indices.w*3+1] * weights.w,
blendMatrices[indices.x*3+2] * weights.x + blendMatrices[indices.y*3+2] * weights.y + blendMatrices[indices.z*3+2] * weights.z + blendMatrices[indices.w*3+2] * weights.w

float4 blendPosition = float4(mul(position, blendMatrix), 1);

positionOut = mul(blendPosition, worldViewProj);
float4 worldPos = mul(blendPosition, world);

texcoordOut = texcoordIn;

normal = mul(, (float3x3)blendMatrix);

normalOut = mul(, world);
viewDirOut = - viewPoint;

float4 OutlinePS( float2 texcoordIn : TEXCOORD0,
float3 normal : TEXCOORD1,
float3 viedDir : TEXCOORD3) : COLOR
return float4(1,1,1,1);
void EdgeVS(float4 Position : POSITION,
out float4 PositionOut : POSITION,
float2 Texcoord : TEXCOORD0,
out float2 TexcoordOut : TEXCOORD0
Position.xy = sign(Position.xy);
PositionOut = float4(Position.xy,0,1);
TexcoordOut.x = 0.5 * (1 + Position.x);
TexcoordOut.y = 0.5 * (1 - Position.y);
const float off = 1.0 / 1000.0;
float4 EdgePS( float2 texcoordIn : TEXCOORD0) : COLOR
float s00 = tex2D(outlineSampler, texcoordIn + float2(-off,-off));
float s01 = tex2D(outlineSampler, texcoordIn + float2(0,-off));
float s02 = tex2D(outlineSampler, texcoordIn + float2(off,-off));

float s10 = tex2D(outlineSampler, texcoordIn + float2(-off, 0));
float s12 = tex2D(outlineSampler, texcoordIn + float2(off, 0));
float s20 = tex2D(outlineSampler, texcoordIn + float2(-off, off));
float s21 = tex2D(outlineSampler, texcoordIn + float2(0, off));
float s22 = tex2D(outlineSampler, texcoordIn + float2(off, off));
float sobelX = s00 + 2 * s10 + s20 - s02 - 2 * s12 - s22;
float sobelY = s00 + 2 * s01 + s02 - s20 - 2* s21 - s22;
float edgeSqr = (sobelX * sobelX + sobelY * sobelY);
float val = 1-(edgeSqr > (0.07 * 0.07));
float4 color = float4(val,val,val,1);
return color;
void ProjectionVS(float4 Position : POSITION,
float2 TexcoordIn : TEXCOORD0,
out float4 PositionOut : POSITION,
out float2 TexcoordOut : TEXCOORD0)
PositionOut = mul(Position, worldViewProj);
TexcoordOut = TexcoordIn;
float4 ProjectionPS( float2 texcoordIn : TEXCOORD0,
float3 normal : TEXCOORD1,
//float3 tangent : TEXCOORD2,
float3 viedDir : TEXCOORD3) : COLOR
float4 color = tex2D(textureSampler, texcoordIn);
return color;
technique OutlineAnimation
VertexShader = compile vs_3_0 AnimatedOutlineVS();
PixelShader = compile ps_3_0 OutlinePS();

technique EdgeAnimation
VertexShader = compile vs_3_0 EdgeVS();
PixelShader = compile ps_3_0 EdgePS();
technique Projection
VertexShader = compile vs_3_0 AnimatedOutlineVS();
PixelShader = compile ps_3_0 ProjectionPS();
I'm fairly sure there's something I've missed regarding using multiple passes and how to use rendertargets. Can someone please clear this up for me? Code used to create similiar effects and such would be very appreciated.
Let me know if you want to see more of my code to get a better understanding of my problem. Thanks in advance for the help.
Jun 22, 2010 at 1:59 PM
Edited Jun 22, 2010 at 5:04 PM

Wow, great row breaking on my previous post. Let's see if I can fix that. ^^;<br>

Edit: Got it working now by using the DrawTargetTexture2D class and using that as a rendertarget and extracting the texture from there. Still not sure if this was the right approach to take when implementing shaders with multiple passes though. Any clarification would be appreciated :)

Jun 23, 2010 at 5:25 PM

It depends what you define a 'pass'.

If you are performing a fullscreen post processing pass, then you have no choice - you have to render to a texture and use that as a source in the shader.

The simple rules are:
Treat the screen as a special texture render target. There is no real difference to how it operates (other than you should only ever draw to it once within a frame)
You cannot render to a texture while also reading from it. You have to 'ping pong'. (However you can hack this a bit on the 360 due to EDRAM).
You can reuse a texture in the same frame. If you need to do 3 passes, this doesn't mean you need 3 render targets. You can render A->B->A. This is what the blur filter does, for instance.
 - in this case, if you use DrawTargetTexture2D's Clone() method, you can easily reuse the same internal texture resources.


Jun 23, 2010 at 6:15 PM
Edited Jun 23, 2010 at 6:16 PM
Just to make sure I understand what you said, for me to create this outline effect I need to render all the models to a single texture and then use that texture in my shader to create the outline effect, right?

By passes I mean that I need to draw multiple times to either one or more textures to create the desired effect, in this case I need to draw the models onto a texture first and then use that texture to calculate where the outlines should be drawn at. If you have used Rendermonkey I'm sure you understand what I mean.

Thanks a lot for the help.
Jun 23, 2010 at 8:03 PM
Edited Jun 23, 2010 at 8:04 PM

Yes, that is what I mean.
I was making sure, because in HLSL a pass has a very different meaning (and xen specifically doesn't support multipass shaders for a number of reasons).

You have to be quite explicit in Xen how you construct your render target textures and the order they are drawn. This is intentional ;-)
Don't be afraid to use many of them, if you need them, use them!

I didn't mention: The easiest way to draw a full-screen quad with a custom shader will be to use a ShaderElement (in Xen.Ex.Graphics2D).
It can be tricky to get the texture sampling to use correct pixel centres, the ShaderElement does this for you.


Jun 24, 2010 at 9:34 PM
Works like a charm now! Thanks again for the help StatusUnknown.