This project is read-only.

Sending more than just world matrices for instances

Mar 16, 2012 at 11:00 PM

Hi,

I am trying to use instancing in a way where I'd like embed more than just a world matrix as per-instance data (e.g. a few unique integers, float3 color tint etc.). Looking at the current code in Xen it seems like it assumes that the only thing we are allowed to pass per-instance is a unique world matrix. I really hope that I am actually wrong about this and perhaps I'm not seeing a way to work around that. So I was either hoping that I'm wrong and it's doable, or I'm wondering how simple it might be to extend Xen to support more data per-instance, or perhaps hear from someone else who might've already implemented such a thing.

Also along those lines, see I see that the special instance data is sent into the shader via the POSITION12,13,14,15 semantics, but I didn't see where in the code that was actually being set, or if it's perhaps something XNA handles automatically (which I kinda of doubt).

Thanks a bunch.

Mar 17, 2012 at 4:29 AM

So it looks like doing what I wanted wasn't too much of a pain. I went into Xen and added generics to the InstanceBuffer as well as the StreamBuffer classes so that they handle more than just the Matrix type such that I can now provide my own struct that can also contain a Matrix inside. The results work as expected and it makes me happy. :)

StatusUnknown, thanks for making Xen. The work you've put into it is very much appreciated. Soon I hope to share the work we've been doing so far.

Cheers.

Aug 8, 2012 at 12:19 AM

Mind sharing your solution Abuzer? I'm currently implented this over my old system as well.

I've got most of the generics down, but a having some trouble on a few points.

What did you do with the 'nullInstanceBuffer'?

In InternalDrawBatch() how did you change this.matrices.World.SetMatrix(ref instances.instances[i]); to fit the new generic method? I can't very well put in a layer of Interface that forces the generic type to have a Matrix property, or that wouldn't allow me to use Matrix or any other built in type as generic type.

How did you change AddInstance(DrawState state) in verticesgroup, as it is required by for example BatchModel.

 

Would be most helpful if you wouldn't mind sharing your sourcefiles.. most likely VerticesGroup.cs, Vertices.cs, DrawStateGeometry.cs and maybe BatchModel.cs

 

Thanks! and yeah, zen is really great! Just a bit tougher using it than it would be if there were more people who knew the odds and ends of it.

Aug 8, 2012 at 12:23 AM

Well this will be a length reply I suppose. :) Let me gather the details and I'll reply ASAP.

Just out of curiosity jsmars, while i'm getting that info, what kind of a game are you working? Anything you can share?

Cheers

Aug 8, 2012 at 12:32 AM
Edited Aug 8, 2012 at 12:36 AM

How I love fast replies, I usually don't even bother with most forums as I can come up with the solution faster myself than the standard several days it takes for replies. So thanks for that! Looking forward to getting this working.

I've been working on a voxel-ish game for about a year now, I rather not place anything on the web at this point, but I'd love to share some stuff with you directly since it seams we are the only two active Xen'ers at the moment. Mind adding me on Skype: jsmaars?

Aug 8, 2012 at 1:32 AM

So first things first. I kept the old method (Matrix only method) as a "sub-set" or inheriting implementation of the new way I'm doing it.

So anywhere that was:

public void DrawBatch(DrawState state, Matrix[] instances, int instanceCount)

I changed it to be:

public void DrawBatch<T>(DrawState state, T[] instances, int instanceCount) where T : struct

The old StreamBuffer is now along with the stream buffer interface is declared as:

 

        internal interface IStreamBuffer
        {
            int FreeIndices { get; }
            int WrittenIndices { get; }
            bool InUse { get; }
            void Clear();
            Type GetDataType();
        }

		//when rendering a batch, store indices in a stream buffer
		//stream buffers are cleared every frame
		internal sealed class StreamBuffer<T> : IStreamBuffer where T : struct
		{
            public Type GetDataType() { return typeof(T); }
            //private readonly Matrix[] instanceMatricesData;
            private readonly T[] instanceData;
            //private readonly Graphics.Vertices<T> instanceMatrices;
            private readonly Graphics.Vertices<T> instanceVertData;
			private int index;
			private bool bufferActive;
            private static int InstanceSize = System.Runtime.InteropServices.Marshal.SizeOf(typeof(T));
			internal Graphics.CustomInstanceBuffer<T> FrequencyInstanceBuffer;

			public StreamBuffer(int count)
			{
				int createSize = 256;
				while (count > createSize)
					createSize *= 2;

				this.instanceData = new T[createSize];
				this.instanceVertData = new Graphics.Vertices<T>(this.instanceData);
				this.instanceVertData.ResourceUsage = Xen.Graphics.ResourceUsage.DynamicSequential;
				this.index = 0;
			}

			public int FreeIndices
			{
				get { return this.instanceData.Length - index; }
			}
			public int WrittenIndices
			{
				get { return index; }
			}

			public T[] Prepare(out int startIndex)
			{
				if (bufferActive)
					throw new InvalidOperationException("A call to BeginDrawBatch has been made while already within a BeginDrawBatch/EndDrawBatch operation./nIf using DrawState.GetDynamicInstanceBuffer, make sure to use the buffer during the frame it was got");
				bufferActive = true;
				startIndex = index;
				return instanceData;
			}

			public void Fill(DrawState state, Graphics.VerticesGroup streamGroup, Graphics.IVertices vertices, T[] matrices, int count)
			{
				if (bufferActive)
				{
					bufferActive = false;

					if (matrices == this.instanceData)
						this.instanceVertData.SetDirtyRange(index, count);
					else
                        this.instanceVertData.Buffer.WriteBuffer(
                            state.Application,
                            0,
                            count * InstanceSize,
                            (this.instanceVertData as Xen.Graphics.IDeviceVertexBuffer).GetVertexBuffer(state.Application, state.graphics),
                            matrices);

					this.index += count;
				}

				streamGroup.Count = vertices.Count;
				streamGroup.SetChild(0, vertices);

				streamGroup.SetChild(1, instanceVertData);
				streamGroup.SetIndexOffset(1, index - count);
			}

			public void Clear()
			{
				if (bufferActive)
					throw new InvalidOperationException("A call to GetDynamicInstanceBuffer has been made without a matching draw call.");
				
				index = 0;
			}

			public bool InUse { get { return this.bufferActive; } }
		}

 

 

For InternalDrawBatch, since i'm not planning to support systems that don't have hardware instancing available, I simply throw an assert where it was previously trying to do something in the case where hardware instancing isn't supported.

In state.cs, I updated the DrawState to hold the list of streambuffers using the new interface i declared:

this.streamBuffers = new List<IStreamBuffer>();

Now the other stuff.

I created a class called CustomInstanceBuffer which is a generic accepting class with a declaration:

 

public class CustomInstanceBuffer<InstanceDataFormat> where InstanceDataFormat : struct

 

The guts simply replace anywhere where the old InstanceBuffer used to rely on the Matrix type to be of generic type T.

Then the old InstanceBuffer class can be declared as a class that simply inherits from CustomInstanceBuffer as in:

        public sealed class InstanceBuffer : CustomInstanceBuffer<Matrix>
	{
		internal InstanceBuffer()
		{
		}

		/// <summary>
		/// Construct a new instance buffer, with the specified maximum number of indices
		/// </summary>
		/// <param name="maxCount"></param>
		public InstanceBuffer(int maxCount)
            : base(maxCount)
		{
		}

		/// <summary>
		/// Construct a new instance buffer, with the specified list of static instances. The instances will be readonly
		/// </summary>
		public InstanceBuffer(Matrix[] instances)
            : base(instances)
		{
		}
        
		/// <summary>
		/// Add an instance to the buffer, using the DrawState to get the current World Matrix
		/// </summary>
		/// <param name="state"></param>
		public void AddInstance(DrawState state)
		{
			if (lastWriteIndex == -1)
				throw new InvalidOperationException("InstanceBuffer is Readonly");
#if DEBUG
			if (index == instanceLength)
				throw new IndexOutOfRangeException();
#endif

			instances[instanceIndex] = state.matrices.World.value;

			instanceIndex++;
			index++;
		}

		/// <summary>
		/// Add an instance to the buffer
		/// </summary>
		/// <param name="position"></param>
		public void AddInstance(ref Vector3 position)
		{
			if (lastWriteIndex == -1)
				throw new InvalidOperationException("InstanceBuffer is Readonly");
#if DEBUG
			if (index == instanceLength)
				throw new IndexOutOfRangeException();
#endif
			Matrix mat = new Matrix();

#if XBOX360
			mat = new Matrix();
#endif

			mat.M11 = 1;
			mat.M22 = 1;
			mat.M33 = 1;
			mat.M41 = position.X;
			mat.M42 = position.Y;
			mat.M43 = position.Z;
			mat.M44 = 1;

			instances[instanceIndex] = mat;

			instanceIndex++;
			index++;
		}
	}

Doing it this way allows Xen's existing samples to continue to function the same way with minimal modifications to the InstanceBuffer interface.

Also made this modification in Vertices.cs, so that I can add a matrix into a struct and Xen will use it as the WorldMatrix for the instance data:

                    if (!f.ReflectedType.IsValueType)
			throw new ArgumentException("Field " + type.Name + "." + f.Name + " is a not a ValueType (struct)");

			int size = Marshal.SizeOf(f.FieldType);

                        //special case for instancing:
                        if (f.FieldType == typeof(Matrix))
                        {
                            elements.Add(new VertexElement(offset,    VertexElementFormat.Vector4, VertexElementUsage.Position, 12));
                            elements.Add(new VertexElement(offset+16, VertexElementFormat.Vector4, VertexElementUsage.Position, 13));
                            elements.Add(new VertexElement(offset+32, VertexElementFormat.Vector4, VertexElementUsage.Position, 14));
                            elements.Add(new VertexElement(offset+48, VertexElementFormat.Vector4, VertexElementUsage.Position, 15));
                            offset += size;
                            continue;
                        }

                        bool attribSet = false;

                        foreach (object o in f.GetCustomAttributes(true))
                        {
                            if (o is VertexElementAttribute)
                            {
                                VertexElementAttribute att = (VertexElementAttribute)o;
                                VertexElementFormat? format = att.VertexElementFormat;
                                if (format == null)
                                    format = DetermineFormat(f);
                                else
                                {
                                    int formatSize;
                                    if (!formatMappingSize.TryGetValue(format.Value, out formatSize))
                                        throw new ArgumentException(string.Format("Invlaid VertexElementFormat ({0}) specified in VertexElementAttribute for {1}.{2}", format, type.FullName, f.Name));
                                    if (formatSize != Marshal.SizeOf(f.FieldType))
                                        throw new ArgumentException(string.Format("VertexElementFormat size mismatch in {4}.{5}, {0} requires a size of {1}, specified type {2} has size {3}", format, formatSize, f.FieldType.FullName, Marshal.SizeOf(f.FieldType), type.FullName, f.Name));
                                }

                                elements.Add(new VertexElement(offset, format.Value, att.VertexElementUsage, (int)att.UsageIndex));
                                attribSet = true;
                                break;

Once all of that is in, then it's a good idea to also update the existing xen models to use DrawBatch functions as such (taken from Sphere.cs but should be done to Cone, SphericalCone, Cone and Cube as well):

        /// <summary></summary>
        public void DrawBatch<T>(DrawState state, Xen.Graphics.CustomInstanceBuffer<T> instances) where T : struct
        {
            verts.DrawInstances(state, inds, PrimitiveType.TriangleList, instances);
        }
        /// <summary></summary>
        public void DrawBatch<T>(DrawState state, T[] instances, int instanceCount) where T : struct
        {
            verts.DrawInstances(state, inds, PrimitiveType.TriangleList, instances, instanceCount);
        }

If I was implementing this from scratch, I would've gotten rid of the name CustomInstanceBuffer and just pushed the functionality into InstanceBuffer, but I didn't want to break the legacy Xen stuff. So I kept it split.

So now you can push a structure like the following into an instance buffer and it works fine:

        public struct ElementInstanceData
        {
            public Matrix WorldMatrix;
            [VertexElement(VertexElementUsage.TextureCoordinate, VertexElementFormat.Vector4, 11)]
            public Vector4 ColorTint;
        }

Hope that helps.

Cheers