This project is read-only.

Shader authoring related questions

May 22, 2011 at 7:22 AM


Having messed a tiny bit with shaders in XNA and Xen, I've come across some interesting issues/questions that I wanted to see if perhaps someone might know the answers to:

1) When I #define and #ifdef style "commenting out" of shader code, Xen doesn't like some of these and gives errors when generating the corresponding cs file. Specifically, for example if I comment out a sampler from being used (i.e. a tex2D() call and I'm guessing the hlsl compiler optimizes out the definition of the sampler) Xen tries to still generate code for it, and throws a shader compile error. Are there some known issues with using preprocessors in fx files with Xen?

2) I "ported" over a shader I used in pure XNA 4.0 over to use Xen, and I ran across some errors that were plain weird. While I wouldn't be totally surprised to see small differences say between the Direct3D compiler (fxc.exe) and XNA's Effect() C# class compiler, I wouldn't expect to see any differences between XNA and Xen's low level shader compilers as they seem to both use Effect(), but maybe that's just plain wrong and something else is going on. Specifically, I got many "asymmetric returns not supported" style errors. It's easy to fix them, but nevertheless threw me off a bit. Is Xen's shader compiler perhaps referencing an old version of the XNA libs which I remember reading that old XNA had some quirks supporting all SM3.0 features.

3) I believe Xen supports hot-swapping of particle definition files, such that the user saves the xml and the runtime immediately picks up the changes. I'm wondering if similar support exists or if it'd be easy to add similar support to Xen for shader code (.fx file updates). I'm guessing not because the code actually ends up being embedded in the code as a byte array along with other function hooks, but still figured I'd ask. :) Perhaps regenerate the byte[] offline and then push the data into memory... but that wouldn't update the external variables, only the logic.


May 23, 2011 at 7:30 AM

To follow up, regarding my questions #1 and #2, it looks like these might be related to the fact that the Native.ShaderDecompiler in the ShaderSystem actually relies on the d3d9 and d3d9x lib files that look like they are statically linked into the Xen.Graphics.ShaderSystem.Native.dll.

I'm guessing at this point that these d3d libs are perhaps old versions of the d3d9 libs that don't properly deal with commented out or #defined our pieces of shader code. Unfortunately I cannot dive down and fix the issue because the Xen.Graphics.ShaderSystem.Native sln seems to be looking for a ShaderInvoke project that doesn't exist in the downloaded version of the Xen codebase (or am I missing something?). I'd be awesome if I could get my hands on it so I can look at potentially fixing this.

I also saw that the reason why the samplers detection is acting up is potentially related to the fact that XNA 4.0 has removed the sampler state querying and now the code relies on the native d3d compiler.

Hoping that I don't have some local issues, I should also add that I'm seeing some really iffy errors where the compiler complains in some particular cases  where a bool variable used in an if statement is invalid because apparently "if clause needs to evaluate to a scalar" or something along those lines. I have no idea one bool would generate such an error while another bool goes thru just fine.


May 23, 2011 at 8:25 AM

Apologies up front for my hammering on this forum.

I dug a little deeper into the ShaderSystem code base along with the fxc compiler, and found out that the errors I'm seeing seem to be a by product of the "modified" shader compilation when the TechniqueExtractor class marks unused variables are tries to recompile the effect.

I'll leave it at that, but at least it looks like I might be able to fix it as the source I need to peek into is the C# code rather than the managed C++ code.

May 27, 2011 at 9:42 AM

Unless I run into more issues down the line, I wanted to provide a final follow up of what I've done.

I've been able to dig into the ShaderSystem code and fix all of the issues that I've been seeing (noted in my posts above). There seemed to be several issues and I'll quickly list out what I did to fix them:

1) "Asymmetric returns" related issue has been fixed by taking out the D3DXSHADER_USE_LEGACY_D3DX9_31_DLL option when creating the compiler in the "native" library.

2) "if clause needs to evaluate to a scalar" issue has been fixed by making sure "unused bools" are not treated the same way as if it's an unused float.

3) "preprocessors" and "unused sampler" related issues turned out to be a relatively simple issue. In my fx files I usually use "sampler" instead of "sampler2D" or "sampler3D", and apparently the native compiler code did not check for "sampler". The fix was to simply check for "sampler" as well. I was going down another road to find all of the compiler stripped samplers, only to realize that that wasn't really necessary.

My question about getting hot swapping working with shader binaries is still open. Any ideas very much appreciated.