Make it as easy as possible for game developers to unit test their games

[digulla]

Describe the project you are working on

Converting a pen-and-paper 2D round-based space combat simulator into a computer game.

Describe the problem or limitation you are having in your project

The game editor doesn't support unit tests for my game.

The documentation, as far as I can tell, contains lot of information about all the different libraries and tools that I can use the game but there is nothing about how to add tests to my project.

In my experience, this leads to the following situation: People develop using the poor-man's-coding loop.

That means 1) write a few lines of code, 2) run them, 3) eyeball the results, 4) repeat, 5) quickly turn everything into an ugly mess, 6) give up or waste a lot of time cleaning up a minefield of bad code, forgotten assumptions and badly understood features.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Write a piece of code. Run it.

Press a button "Create Test".

The editor should remember what you did (which controller/mouse buttons you clicked/keys you pressed, which events were created and what the results of the event handlers were) and just turn that into a little piece of code which repeats exactly the same thing, maybe every time you save the code.

Ideally, the editor should just put the stuff into the test that I'm currently working on. So maybe just save the events/parameters of the function/handlers that were recently edited and just call those; not start the whole game and run everything. If I'm working on a key event handler, just remember the key event and which parts of the game state the handler read and which it modified.

The goal here is to have a very cheap way to preserve what I feel is "good" so I can run all of them tomorrow and see what has changed. Tests should be cheap (ideally: button press) to create, so people don't feel too bad about throwing them away when they change/delete/modify code.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

When developing the game, the editor should remember what code I look at and what I change.

When I press the "Create test" button, it should remember the inputs and outputs of that code. Ideally, it should not start the whole game engine for a "press button" or "enemy collision" handler. It should instead just take the last parameter set, call just the handler, examine the changed game state.

Something like:

def collision(player, enemy):
    if player.intersects(enemy):
        showHugeExplosion()

In this case, it should add the parts of the player and enemy necessary to call "intersects" and remember whether the code calls showHugeExplosion() or not (and not actually call showHugeExplosion()).

I think for simple cases, the editor should be able to figure out most of the stuff by itself. For more complex ones, show the developer a dialog where they can deselect things that don't matter to them.

If the function was called many times (as in the collision example), remember only the last one or last 10 and let the developer pick.

Another approach would be to allow to set a breakpoint and then click "Create test" when the code stops. This would make it easier to pick which local variables or which parts of object structures should be preserved for the test to be meaningful.

If this enhancement will not be used often, can it be worked around with a few lines of script?

I was thinking about a function which allows to save local state but that would make the code messy since you would need it in a lot of places.

An interesting approach would be a function accepting a boolean which would create a test (using the local scope again) when the condition is true. That would make it much easier to select a local state.

Is there a reason why this should be core and not an add-on in the asset library?

I don't know how deep an asset or plugin can dig into the core. Would it be possible to write a debugger for Godot as an add-on?

If so, that might be great because it would allow to have two or three such projects and then pick best of breed.

[digulla]

Background: A lot of games, even really big titles who really should know better, fail when it comes to testing. For games, "testing" is often "let someone play and get their feedback".

Automated unit testing, like it's being done by the whole software industry, seems like a very novel and alien idea to game developers. This is especially bad for game engines which offer "quick results" because that always means "shallow understanding". A human brain simply isn't able to understand something complicated quickly.

My feeling is that this would be a unique and useful feature for Godot. It would help with tutorials: instructors would put correct results into unit tests with little effort and ask the pupils to "make them all green".

It would also help with bug reports: The same framework can be used to serialize the game state, so developers can see what happened.

And it would get game developers used to having tests from the very start.

[Calinou]

See also #432 and #1760.

I feel this adds a lot of complexity for something that hasn't been "proven" to work well in other game engines, so I'm skeptical about this idea. Not to mention unit testing is still not very popular for actual game logic right now. I don't think most game developers' opinion on unit testing is going to change anytime soon. The idea of TDD in particular will often cause game developers to flee 🙂

[mrjustaguy]

I think this would just be unused by most devs, and personally, I don't see this as being a useful feature to me.

and as a side note: Quick results != Shallow understanding

People can get Quick results in Godot, because it's made that way, but that doesn't prevent them from having a deeper understanding of what's going on in the background, not to mention that really, the main issue in terms of bugs is often poor logic on the dev's side, and not even the scripting going on, as that part is simple and straight forward most of the time.. I don't see unit tests helping with that too much.

[Xrayez]

The idea of TDD in particular will often cause game developers to flee 🙂

I think following pure TDD approach in game development is a waste of time, because code is rewritten often and effort needed to maintain those tests is a real burden, especially when unit tests only cover surface API functionality that works correct most of the time once a function is there and won't have to be touched again.

However, I find writing test cases extremely useful for:

• When algorithm itself is complex and you "build up" correctness of it by writing test cases with parameterized input. At some point, you realize that it's silly to run 5-10 scenes on every code change to verify the result manually, so this becomes an automatization problem as well, which is natural to solve regardless of whether you like or dislike writing tests, and you have to do it anyway to ensure correctness, otherwise you'll end up with a broken game in production for weeks or months until you're able to fix all bugs reported by players.
• Simply being able to prevent regressions is really valuable, especially for code in production. So, if we're not too picky about terminology here, I'd say any kind of testing will do. Writing regression/integration test suites is bare minimum for developers to strive for.

Therefore, if we were to implement such a testing framework directly in Godot core (code must be tested in export templates as well), it probably has to be simple enough to increase adoption and ease the maintenance burden. Something which is going to aid existing development workflow, but won't have all the features you see in testing frameworks. Of course, being able to run all tests at once is a "must have" feature.

If you look at plugins such as GUT or WAT, you'll notice that there are tons of features to be implemented. While I share the desire behind this proposal, I don't think Godot core developers would be interested in maintaining such a codebase.

Maybe it's something to be implemented in Goost?

[Xrayez]

Some ideas/essential features:

• Make it possible for assert() not to break into the debugger. Perhaps have to introduce @test annotation/keyword for the script in order for script editor debugger to report passing/failing assertions directly in the editor, while the game/project is running. Tests can be run in tool scripts (within editor), so could also write both test and tool keyword/annotations in the same script (annotations are part of Godot 4.x).
• Editor must have a way to run all test scenes or scripts. This is where @test annotation could be useful for test collector. At the same time, I'd personally prefer scene-based approach over scripts, because it's also possible to write tests in C# and other languages, and I think people create scenes first rather than scripts in Godot (scripts are attached later).
• Being able to run tests in continuous integration manner. For that, one must return appropriate exit code when tests pass/fail.

[MikeSchulze]

Hi @digulla try out GdUnit3 it supports already "Create Test" functionallity and test-suite template. It is a full integrated unit testing API in th Godot Editor.
https://github.com/MikeSchulze/gdUnit3

[BenMcLean]

The idea of TDD in particular will often cause game developers to flee

No one should have to defend the idea of unit testing. Just because you want the ability to run some unit tests for some code doesn't mean you have to advocate for everyone everywhere to adopt pure TDD with 100% coverage for all projects. We just want to be able to run some automated tests and have some kind of standard so we can read and understand each other's tests.

[ryehollister]

What would it take to implement a basic MVP unit test for Godot? Is it a project that someone could reasonably take on? I am thinking of contributing to Godot and want to improve the testing experience.

[MikeSchulze]

What would it take to implement a basic MVP unit test for Godot? Is it a project that someone could reasonably take on? I am thinking of contributing to Godot and want to improve the testing experience.

GdUnit3 is already MVP :)
It presents the test results in an inspector where you can navigate and view the bug reports

[ryehollister]

What would it take to implement a basic MVP unit test for Godot? Is it a project that someone could reasonably take on? I am thinking of contributing to Godot and want to improve the testing experience.

GdUnit3 is already MVP :) It presents the test results in an inspector where you can navigate and view the bug reports

Cool. Thanks for the link! :)

[a2937]

As a curiosity, I am wonderful if GDUnit3 is compatible with multiple programming languages. And I was about to suggest this issue myself actually. I love the idea of being able to test code that I write for game engines as it decreases the chances of a bug being reintroduced or worse never found and reported.