Custom Tool Issue

Dec 16, 2008 at 8:19 AM
Here I go again ;)

Been working with the shader system in Xen for a bit, but I keep coming across the same problem: shader.fx variables are somehow removed during compilation.

Here's an example of what I'm having:


float rotation;


//pixel shader
    //rotate our position
    float SinVal = sin(rotation);
    float CosVal = cos(rotation);
    float Nx = x * CosVal - y * SinVal;
    float Ny = y * CosVal + x * SinVal;

//those values are then passed back to position and transformed to screen positions


Pretty straight forward. And since it's a non-semantic float variable, you would expect the Custom Tool to produce an attribute Shader.Rotation. However, it does not. Any clue as to why? It feels to me like some kind of optomization bug, or I'm doing something very wrong here :P
Dec 16, 2008 at 9:09 AM
Found a strange occurrence here. Check the shader file I'm using: Shader File

If I change that float4 rotation to a normal float, the attribute disappears. With the float4, it compiles and appears in the list:


ParticleShader.init_gd = state.DeviceUniqueIndex;
ParticleShader.id_0 = state.GetNameUniqueID("rotation");
ParticleShader.id_1 = state.GetNameUniqueID("colour");
ParticleShader.ptid_0 = state.GetNameUniqueID("DisplayTexture");
ParticleShader.psid_0 = state.GetNameUniqueID("DisplayTextureSampler");


Regardless of wether or not taking sin/cos from a float4 makes any sense whatsoever, this seems pretty weird. Also, if I try doing rotation.x, or rotation[0], it will not compile the entire shader.fx and instead produce a completely empty shader.fx.cs
Dec 16, 2008 at 9:45 AM
Edited Dec 16, 2008 at 9:58 AM


This appears to be a bug on my part. (You seem to be good at finding bugs :-)

Let me explain.

The shader you are using is generating a PreShader for the vertex shader, which is a smaller shader run on the CPU. A preshader offloads all duplicate logic to the CPU (logic that would be identical for every vertex). This shader is generated by the DirectX shader compiler. It uses a similar syntax to ASM shaders.

For various reasons, I have to convert these preshaders to .net code.

The point at which it's failing, is the cos() function. This function is cos(rotation), so it gets put in the preshader, so the cos() isn't performed for every vertex. (it gets done once on the CPU).

Now, I'd tested HLSL cos(), and pretty much everything else I could think of - and HLSL cos() *always* produced a complex floating point approximation in the preshader. I have no idea why that in this case, it's actually asking for the real cos() function (which isn't implemented in my processor).

To make it even more confusing, the preshader language has some special commands that aren't documented - and aren't a part of any shader ASM language. And guess what, cos() is one of them. sincos() is documented, but cos() isn't. So I'm going to assume that sin(), tan() etc are all not documented either. Argh :-)

Also, any value that isn't used by the shader (or gets optimized out) will not appear as an attribute. Otherwise, having multiple shaders in the same file would bloat them all quite quickly.

Can you please download:

Extract the CustomTool.dll to ./xen/Xen.Graphics.ShaderSystem.CustomTool/, replacing the old one (you might want to back it up..). Make sure VS is closed when you copy it.

Let me know if it works correctly. I'll roll the changes into the next update.

And please, find more nasty obscure bugs! :-)


I'll also mention, when you had this problem, the shader.fx.cs file was blank. In this case, there should have been an error listed in the visual studio 'Error List' window, which should have said something along the lines of unrecognized preshader command 'cos''. If you encounter any other problems, they should also be reported in this way.

Dec 16, 2008 at 9:57 AM
Doesn't appear to change anything on my end. How can I be sure it's using the correct DLL?

I built Xen and copied the bin contents to a separate folder, and am using the following reference dll's:


And the file you posted is placed in:

Dec 16, 2008 at 10:02 AM
Edited Dec 16, 2008 at 10:05 AM

Sorry, I forgot to mention you will need to re-process the shader.

This can be done by either saving the file again, or right clicking it and selecting 'Run CustomTool'.

Unfortunately, VisualStudio isn't smart enough to re-run the custom tool if the tool changes.

*Also* you will need to place the .dll in the directory from where prebuild.bat was run (this registers the .dll's location), so if you have made a copy of the directory, and are working from the copy, you will need to overwrite the file in the original directory.

Dec 16, 2008 at 10:04 AM
Edited Dec 16, 2008 at 10:10 AM
EDIT: Trying with the original directory.
Dec 16, 2008 at 10:09 AM


I've probably missed something.

Close all copies of Visual Studio.

Go to the directory where you originally downloaded xen - or where you last ran prebuild.bat. This will be the directory where the custom tool was registered.

From within there (where prebuild.bat is), go to ./xen/bin/Xen.Graphics.ShaderSystem.CustomTool/  (I forgot the 'bin' last time)

In that directory, there should be 5 files and one directory.

One of the files is Xen.ShaderSystem.CustomTool.dll, this is the file that needs to be replaced.

I've tried this out on a local copy, and it seemed to work OK. So I'm thinking it's just a case of finding the original .dll.

The CustomTool.dll isn't used at runtime, it's used by VisualStudio, so it's actually totally unrelated to even the Solution / Project files (It's linked in through the registry, it's all rather complex...)

Dec 16, 2008 at 10:11 AM
You were right about the original directory. It was using a different .dll, and when I replaced the original it did work.

This problem appears fixed now, the vertices are rotated properly and everything too. Thanks!
Dec 16, 2008 at 10:15 AM


Well I hope that everything works OK from now on. Either that or you find some more subtle bugs that no one else has run into :P

Good timing too, as it's midnight and I'm off to bed :-)