Creating your own file format to import .FBX, .OBJ and .X in your Windows 8 modern UI game (or 3D engine)

There is a lot of different file format when it comes to 3D objects. One of the most used is the FBX from Autodesk. This file format can be exported by all major DCC but the key point is that it can be complex for a game or 3D developer to open such file format.

I would like to propose here a solution that can allows you to easily offline files importation. The idea is to simulate a MSBuild execution to reuse the importation process of the XNA pipeline.

Indeed, XNA is able to load file formats such as .X, .OBJ and .FBX. So with the following code, you will be able to parse 3D files and generate a complete in memory object model based on the content of the files.

Why do I need to offline file parsing?

It is a great idea to parse your assets offline because in this case you can create your own file format and efficiently load it at runtime.

You no longer need to have all the different parsers in your game engine and in some case you can optimize things using complex and costly algorithms.

Using MSBuild alongside XNA

The main trick here is to use the power of MSBuild with the XNA Framework.

To do so you just have to use the following code (extracted from Babylon, the 3D engine I wrote for WorldMonger):

public void GenerateBabylonFile(string file, string outputFile, bool skinned)
{
    using (FileStream fileStream = new FileStream(outputFile, FileMode.Create, FileAccess.Write))
    {
        using (BinaryWriter writer = new BinaryWriter(fileStream))
        {
            writer.Write(Version);


            var services = new BabylonImport.Importers.FBX.ServiceContainer();

            // Create a graphics device
            var form = new Form();
            services.AddService<IGraphicsDeviceService>(GraphicsDeviceService.AddRef(form.Handle, 1, 1));

            var contentBuilder = new ContentBuilder();
            var contentManager = new ContentManager(services, contentBuilder.OutputDirectory);

            // Tell the ContentBuilder what to build.
            contentBuilder.Clear();
            contentBuilder.Add(file, "Model", null, skinned ? "SkinnedModelProcessor" : "ModelProcessor");

            // Build this new model data.
            string buildError = contentBuilder.Build();

            if (string.IsNullOrEmpty(buildError))
            {
                var model = contentManager.Load<Model>("Model");
                ParseModel(model, writer);
            }
            else
            {
                throw new Exception(buildError);
            }
        }
    }
}

Please note the usage of the  skinned boolean: It allows me to use the standard XNA ModelProcessor which does not take in account skinned meshes or my own SkinnedModelProcessor to add support for skinned models (I will not detail these files here, you can have a look to the complete solution if you want more information)

To use this code, you will need the ServiceContainer class (a simple implementation of the IServiceProvider interface):

using System;
using System.Collections.Generic;

namespace BabylonImport.Importers.FBX
{
    public class ServiceContainer : IServiceProvider
    {
        Dictionary<Type, object> services = new Dictionary<Type, object>();

        /// <summary>
        /// Adds a new service to the collection.
        /// </summary>
        public void AddService<T>(T service)
        {
            services.Add(typeof(T), service);
        }

        /// <summary>
        /// Looks up the specified service.
        /// </summary>
        public object GetService(Type serviceType)
        {
            object service;

            services.TryGetValue(serviceType, out service);

            return service;
        }
    }
}

You will also need the GraphicsDeviceService class which contains all the required resources to create a XNA GraphicsDevice class:

using System;
using System.Threading;
using Microsoft.Xna.Framework.Graphics;

#pragma warning disable 67

namespace BabylonImport.Importers.FBX
{
    class GraphicsDeviceService : IGraphicsDeviceService
    {
        static GraphicsDeviceService singletonInstance;
        static int referenceCount;

        GraphicsDeviceService(IntPtr windowHandle, int width, int height)
        {
            parameters = new PresentationParameters();

            parameters.BackBufferWidth = Math.Max(width, 1);
            parameters.BackBufferHeight = Math.Max(height, 1);
            parameters.BackBufferFormat = SurfaceFormat.Color;
            parameters.DepthStencilFormat = DepthFormat.Depth24;
            parameters.DeviceWindowHandle = windowHandle;
            parameters.PresentationInterval = PresentInterval.Immediate;
            parameters.IsFullScreen = false;

            graphicsDevice = new GraphicsDevice(GraphicsAdapter.DefaultAdapter,
                                                GraphicsProfile.Reach,
                                                parameters);
        }

        public static GraphicsDeviceService AddRef(IntPtr windowHandle, int width, int height)
        {
            if (Interlocked.Increment(ref referenceCount) == 1)
            {
                singletonInstance = new GraphicsDeviceService(windowHandle, width, height);
            }

            return singletonInstance;
        }

        public void Release(bool disposing)
        {
            if (Interlocked.Decrement(ref referenceCount) == 0)
            {
                if (disposing)
                {
                    if (DeviceDisposing != null)
                        DeviceDisposing(this, EventArgs.Empty);

                    graphicsDevice.Dispose();
                }

                graphicsDevice = null;
            }
        }

        public void ResetDevice(int width, int height)
        {
            if (DeviceResetting != null)
                DeviceResetting(this, EventArgs.Empty);

            parameters.BackBufferWidth = Math.Max(parameters.BackBufferWidth, width);
            parameters.BackBufferHeight = Math.Max(parameters.BackBufferHeight, height);

            graphicsDevice.Reset(parameters);

            if (DeviceReset != null)
                DeviceReset(this, EventArgs.Empty);
        }


        public GraphicsDevice GraphicsDevice
        {
            get { return graphicsDevice; }
        }

        GraphicsDevice graphicsDevice;

        PresentationParameters parameters;

        public event EventHandler<EventArgs> DeviceCreated;
        public event EventHandler<EventArgs> DeviceDisposing;
        public event EventHandler<EventArgs> DeviceReset;
        public event EventHandler<EventArgs> DeviceResetting;
    }
}

Finally you will need the ContentBuilder class to handle the MSBuild process (A big thank to Shawn Hargreaves for this one !):

using System;
using System.IO;
using System.Diagnostics;
using System.Collections.Generic;
using Microsoft.Build.Construction;
using Microsoft.Build.Evaluation;
using Microsoft.Build.Execution;
using Microsoft.Build.Framework;
using System.Windows.Forms;

namespace BabylonImport.Importers.FBX
{
    class ContentBuilder : IDisposable
    {
        const string xnaVersion = ", Version=4.0.0.0, PublicKeyToken=842cf8be1de50553";

        static readonly string[] pipelineAssemblies =
        {
            "Microsoft.Xna.Framework.Content.Pipeline.FBXImporter" + xnaVersion,
            "Microsoft.Xna.Framework.Content.Pipeline.XImporter" + xnaVersion,
            "Microsoft.Xna.Framework.Content.Pipeline.TextureImporter" + xnaVersion,
            "Microsoft.Xna.Framework.Content.Pipeline.EffectImporter" + xnaVersion,
            "SkinnedModelPipeline"
        };

        Project buildProject;
        ProjectRootElement projectRootElement;
        BuildParameters buildParameters;
        readonly List<ProjectItem> projectItems = new List<ProjectItem>();
        ErrorLogger errorLogger;

        string buildDirectory;
        string processDirectory;
        string baseDirectory;

        static int directorySalt;

        public string OutputDirectory
        {
            get { return Path.Combine(buildDirectory, "bin"); }
        }

        public ContentBuilder()
        {
            CreateTempDirectory();
            CreateBuildProject();
        }

        public void Dispose()
        {
            DeleteTempDirectory();
        }

        void CreateBuildProject()
        {
            string projectPath = Path.Combine(buildDirectory, "content.contentproj");
            string outputPath = Path.Combine(buildDirectory, "bin");

            // Create the build project.
            projectRootElement = ProjectRootElement.Create(projectPath);

            // Include the standard targets file that defines how to build XNA Framework content.
            projectRootElement.AddImport(Application.StartupPath + 
                                         "\XNA\Microsoft.Xna.GameStudio.ContentPipeline.targets");

            buildProject = new Project(projectRootElement);

            buildProject.SetProperty("XnaPlatform", "Windows");
            buildProject.SetProperty("XnaProfile", "Reach");
            buildProject.SetProperty("XnaFrameworkVersion", "v4.0");
            buildProject.SetProperty("Configuration", "Release");
            buildProject.SetProperty("OutputPath", outputPath);
            buildProject.SetProperty("ContentRootDirectory", ".");
            buildProject.SetProperty("ReferencePath", Application.StartupPath);

            // Register any custom importers or processors.
            foreach (string pipelineAssembly in pipelineAssemblies)
            {
                buildProject.AddItem("Reference", pipelineAssembly);
            }

            // Hook up our custom error logger.
            errorLogger = new ErrorLogger();

            buildParameters = new BuildParameters(ProjectCollection.GlobalProjectCollection)
                                  {Loggers = new ILogger[] {errorLogger}};
        }

        public void Add(string filename, string name, string importer, string processor)
        {
            ProjectItem item = buildProject.AddItem("Compile", filename)[0];

            item.SetMetadataValue("Link", Path.GetFileName(filename));
            item.SetMetadataValue("Name", name);

            if (!string.IsNullOrEmpty(importer))
                item.SetMetadataValue("Importer", importer);

            if (!string.IsNullOrEmpty(processor))
                item.SetMetadataValue("Processor", processor);

            projectItems.Add(item);
        }

        public void Clear()
        {
            buildProject.RemoveItems(projectItems);

            projectItems.Clear();
        }

        public string Build()
        {
            // Clear any previous errors.
            errorLogger.Errors.Clear();

            // Create and submit a new asynchronous build request.
            BuildManager.DefaultBuildManager.BeginBuild(buildParameters);

            var request = new BuildRequestData(buildProject.CreateProjectInstance(), new string[0]);
            BuildSubmission submission = BuildManager.DefaultBuildManager.PendBuildRequest(request);

            submission.ExecuteAsync(null, null);

            // Wait for the build to finish.
            submission.WaitHandle.WaitOne();

            BuildManager.DefaultBuildManager.EndBuild();

            // If the build failed, return an error string.
            if (submission.BuildResult.OverallResult == BuildResultCode.Failure)
            {
                return string.Join("n", errorLogger.Errors.ToArray());
            }

            return null;
        }

        void CreateTempDirectory()
        {
            baseDirectory = Path.Combine(Path.GetTempPath(), GetType().FullName);
            int processId = Process.GetCurrentProcess().Id;

            processDirectory = Path.Combine(baseDirectory, processId.ToString());
            directorySalt++;

            buildDirectory = Path.Combine(processDirectory, directorySalt.ToString());

            Directory.CreateDirectory(buildDirectory);

            PurgeStaleTempDirectories();
        }

        void DeleteTempDirectory()
        {
            Directory.Delete(buildDirectory, true);

            if (Directory.GetDirectories(processDirectory).Length == 0)
            {
                Directory.Delete(processDirectory);

                if (Directory.GetDirectories(baseDirectory).Length == 0)
                {
                    Directory.Delete(baseDirectory);
                }
            }
        }


        void PurgeStaleTempDirectories()
        {
            // Check all subdirectories of our base location.
            foreach (string directory in Directory.GetDirectories(baseDirectory))
            {
                // The subdirectory name is the ID of the process which created it.
                int processId;

                if (int.TryParse(Path.GetFileName(directory), out processId))
                {
                    try
                    {
                        // Is the creator process still running?
                        Process.GetProcessById(processId);
                    }
                    catch (ArgumentException)
                    {
                        // If the process is gone, we can delete its temp directory.
                        Directory.Delete(directory, true);
                    }
                }
            }
        }
    }
}

Please note that the XNA assemblies used by the MSBuild process are located in the application folder:

The MSBuild targets for XNA are located in this folder:

Application.StartupPath + “\XNA\Microsoft.Xna.GameStudio.ContentPipeline.targets”

Parsing object models

Once all the build process is setup, you just have to browse the objects generated by XNA through the XNA content pipeline:

var model = contentManager.Load<Model>("Model");
ParseModel(model, writer);

The ParseModel method create the final file according to your needs:

void ParseModel(Model model, BinaryWriter writer)
{
    var effects = model.Meshes.SelectMany(m => m.Effects).ToList();
    var meshes = model.Meshes.ToList();
    var total = effects.Count + meshes.Count;
    var progress = 0;
    SkinningData skinningData = null;
    if (model.Tag != null)
    {
        skinningData = model.Tag as SkinningData;
        total += skinningData.BindPose.Count;
    }

    if (skinningData != null)
    {
        // Bones
        for (int boneIndex = 0; boneIndex < skinningData.BindPose.Count; boneIndex++)
        {
            ParseBone(boneIndex, skinningData, writer);
            if (OnImportProgressChanged != null)
                OnImportProgressChanged(((progress++) * 100) / total);
        }

        // Animations
        foreach (var clipKey in skinningData.AnimationClips.Keys)
        {
            ParseAnimationClip(clipKey, skinningData.AnimationClips[clipKey], writer);
        }
    }

    foreach (Effect effect in effects)
    {
        ParseEffect(effect, writer);
        if (OnImportProgressChanged != null)
            OnImportProgressChanged(((progress++) * 100) / total);
    }

    foreach (var mesh in meshes)
    {
        ParseMesh(mesh, writer);
        if (OnImportProgressChanged != null)
            OnImportProgressChanged(((progress++) * 100) / total);
    }
}

In my case, I go through all meshes and effects and I write on the final output file what I need for my game:

void ParseMesh(ModelMesh modelMesh, BinaryWriter writer)
{
    var proxyID = ProxyMesh.Dump(modelMesh.Name, writer);
    int indexName = 0;

    foreach (var part in modelMesh.MeshParts)
    {
        var material = exportedMaterials.First(m => m.Name == part.Effect.GetHashCode().ToString());

        var indices = new ushort[part.PrimitiveCount * 3];
        part.IndexBuffer.GetData(part.StartIndex * 2, indices, 0, indices.Length);

        for (int index = 0; index < indices.Length; index += 3)
        {
            var temp = indices[index];
            indices[index] = indices[index + 2];
            indices[index + 2] = temp;
        }

        if (part.VertexBuffer.VertexDeclaration.VertexStride > PositionNormalTextured.Stride)
        {
            var mesh = new Mesh<PositionNormalTexturedWeights>(material);
            var vertices = new PositionNormalTexturedWeights[part.NumVertices];
            part.VertexBuffer.GetData(part.VertexOffset * part.VertexBuffer.VertexDeclaration.VertexStride,
 vertices, 0, vertices.Length, part.VertexBuffer.VertexDeclaration.VertexStride);

            mesh.AddPart(indexName.ToString(), vertices.ToList(), indices.Select(i=>(int)i).ToList());
            mesh.Dump(writer, proxyID);
        }
        else
        {
            var mesh = new Mesh<PositionNormalTextured>(material);
            var vertices = new PositionNormalTextured[part.NumVertices];
            part.VertexBuffer.GetData(part.VertexOffset * PositionNormalTextured.Stride, vertices, 0, 
vertices.Length, PositionNormalTextured.Stride);

            mesh.AddPart(indexName.ToString(), vertices.ToList(), indices.Select(i => (int)i).ToList());
            mesh.Dump(writer, proxyID);
        }

        indexName++;
    }
}

Using strictly the same you can import .OBJ or even .X files!

Feel free to use it for your own game:

https://www.catuhe.com/msdn/babylonimport.zip