“Scenario Testing” a game engine by misusing an unit test framework.

I don’t post regularly on this blog, but I really should post more… ^^”

If you have ever read me here before, you probably know that one of my pet project is a game engine called Annwvyn.

Where did I get from

Annwvyn was just “a few classes to act as glue code around a few free software library”. I really thought that in 2 months I had some piece of software worthy of bearing the name game engine. Obviously, I was just a foolish little nerd playing around with an Oculus DK1 in his room, but still, I did actually manage to have something render in real time on the rift with some physics and sound inside! That was cool!

Everything started as just a test project, then, I decided to remove the int main(void)  function I had and stash everything else inside a DLL file. That was quickly done (after banging my head against the MSDN website and Visual Studio’s 2010 project settings, and writing a macro to insert __declspec(dllexport) or __declspec(dllimport) everywhere.)

The need for testability and the difficulties of retrofitting tests

So let’s be clear: I know about good development practice, about automated testing, about TDD, about software architecture, about UML Class Diagrams and all that jazz. Heck, I’m a student in those things. But, the little hobby project wasn’t intended to grow as a 17000 lines of C++ with a lot of modules and bindings to a scripting language, and an event dispatch system, and a lot of interconnected components that abstract writing data to the file system (well, it’s for video game save files) or rendering to multiple different kind of VR hardware, to go expand the Resource Manager of Ogre. Hell, I did not know that Ogre had such a complex resource management system. I thought that Ogre was a C++ thing that drew polygon on the screen without me having to learn OpenGL. (I still had to actually learn quite a lot about OpenGL because I needed to hack into it’s guts, but I blogged about that already.).

Lets just say that things are really getting out of hands, and that I seriously needed to start thinking about making the code saner, and to be able to detect when I break stuff.

So, I searched about a way of testing the engine. The code base is not really intended to be unit tested. Many of the components of the engine cannot exit by themselves in a test case, and it makes little sens to test most of the methods individually.

Using automated testing with the “Catch” library!

With that in mind, I still needed a way to automate testing of the code. I settled on testing the engine “features by features” instead of “code units by code units”. The basic outline is to startup a “game” ask some of the components to do something, check either the state of some of them, or check if they set the state of some flag variables to indicate success.

Then, I needed a way to easily automate the testing. For that, I decided to misuse an Unit Test framework. I choose Catch with recommendation from the CppLang slack, and because the library is available as a single header. Also, it integrates really well with Resharper Ultimate.

Then it was just a matter of adding a few lines into the main CMakeLists file to create an additional project that generate an executable, and including the single header of the library.

add_executable(AnnwvynUnitTest ${TestCode} ${TestPCH})

target_link_libraries(AnnwvynUnitTest
    Annwvyn
    ${OGRE_LIBRARIES}
    ${OGRE_HlmsPbs_LIBRARIES}
    ${OGRE_HlmsUnlit_LIBRARIES}
    ${OGRE_Overlay_LIBRARIES}
    )</pre>

Catch can implement it’s own “main” function, and it will use the command line arguments to get it’s configuration, but if you are like me and need to do some setup for things to work well, you’ll want to write you own “main” function. This is really easy, what it takes is to specify the right #defines before including catch, and launch the test session yourself via Catch::Session::run

#define CATCH_CONFIG_RUNNER

#include <catch/catch.hpp>

int main(int argc, char* argv[])
{
    Annwvyn::AnnEngine::setNoConsoleColor();

    auto result = Catch::Session().run(argc, argv);

    return (result < 0xff ? result : 0xff);
}

At this point, tests are fairly easy to write, you just need to call a few macros:

namespace Annwvyn //For commodity, I'll put all tests inside the Annwvyn namespace
{
	TEST_CASE("Basic engine start") //Declare test
	{
		auto GameEngine = std::make_unique<AnnEngine>("BasicInit", RENDERER);
		REQUIRE(GameEngine != nullptr); //Here's an assert. The expression should be true
	}
}

The RENDERER variable is defined in a configuration file, I may need more static configuration in the future, so I created a header to specify them. Right now it only contains the following:

#pragma once

#define RENDERER "OgreNoVRRender"</pre>

Testing engine features instead of code units

Annwvyn is organized as a section of “modules” that are called “sub systems”. Each subsystem is responsible for one part of the game engine functionality. Everything in Annwvyn that is not the 3D renderer itself is organized around this concept.

The core subsytsem of Annwvyn are spawned into existence by creating an instance of the AnnEngine class. AnnEngine needs to choose a “renderer” object from the ones available (currently they are all compiled into the library, but it’s on plan to load them as DLL at runtime in the near future).
Annwvyn is focused to making VR experiences. But there’s one renderer that will only render to a window on your desktop called the “NoVR” renderer. The static configuration will tell Annwvyn to start with this renderer only.

This aside, many test would probably want to have a few objects and a source of light existing in the scene. Instead of repeating this code multiple time, I added a little “test engine factory” to the test project

#pragma once
#include <string>
#include "configs.hpp"
#include <Annwvyn.h>

#include <catch/catch.hpp>

namespace Annwvyn
{
	//All tests will need at least this :
	inline std::unique_ptr<AnnEngine> bootstrapTestEngine(const std::string& name)
	{
		//Start engine
		auto GameEngine = std::make_unique<AnnEngine>(name.c_str(), RENDERER);
		REQUIRE(GameEngine);

		//Construct environment
		auto sun = AnnGetGameObjectManager()->createLightObject("_internal_test_sun"); //physics based shading crash shaders if no light
		sun->setType(AnnLightObject::ANN_LIGHT_DIRECTIONAL);
		sun->setPower(97);
		sun->setDirection(AnnVect3{ 0, -1, -2 }.normalisedCopy());
		REQUIRE(sun);

		//Fixed object in space : the floor
		auto floor = AnnGetGameObjectManager()->createGameObject("floorplane.mesh", "_internal_test_floor");
		floor->setUpPhysics();
		REQUIRE(floor);
		REQUIRE(floor->getBody());

		return move(GameEngine);
	}

	inline std::unique_ptr<AnnEngine> bootstrapEmptyEngine(const std::string& name)
	{
		auto GameEngine = std::make_unique<AnnEngine>(name.c_str(), RENDERER);
		REQUIRE(GameEngine);

		return move(GameEngine);
	}
}

Then, I’m currently in the process of writing test submodules by submodules. Currently, I’m not attempting to test things that regards processing user inputs, but I may plan to do this in the future. (One of my small project is a library for simulating user inputs on the keyboard).

Currently, there’s only an handful of test, but they all helped to find some pretty major issues, like the fact that the file system manager wasn’t handling the case when files weren’t created successfully, or that it wasn’t actually able to create directories on Linux.

The test code lives here, https://github.com/Ybalrid/Annwvyn/tree/master/tests, and I’ll try to add to the tested functionalities in the near future.

Currently, I’m focusing on getting things actually ready for releasing Annwvyn 0.4, and I’m working on a new website to replace the one currently hosted at Annwvyn.org with a wordpress one that will be easier to manage than PluXML