20 Testing Stendhal
Guidance for Testing Stendhal.
20.1 Introduction
In the team study sessions from week 2 onwards, you’ll be working on your team coursework. The focus initially is on writing tests to make the issues visible: that is, you will write tests that fail when the bug reported by the issue is present in the code base, and pass when it is not present. The process is:
- First, understand the bug by replicating it manually in your local copy of the game.
- Check to see whether any existing tests fail because of the presence of the bug.
- If not, you need to write a test that has this property. Use the existing test suite as a source of inspiration and ideas for this.
- Check that the test fails on the original version of the code base (i.e., it reveals the presence of the bug).
- Figure out what is causing the bug and fix it in the production code.
- Check that all the tests are now passing, including the one that reveals the presence of the bug.
In this approach, we start off with a green JUnit bar (because although the bug is present, no test reveals it), before moving to a red JUnit bar (because we have added a test that reveals the bug) and then back to a green JUnit bar (because we have a test that reveals the presence of the bug, but the bug is now fixed).
This document contains some suggestions for carrying out the first few of these steps more efficiently. You’ll look at techniques for carrying out the full process in the workshops in week 3.
20.2 Manually Replicating The Issues
You will find the code base much easier to navigate, and the code relating to your issue much easier to find and understand, if you first replicate the issue manually. This means running the game on your local server, to see the bug happening in the context of game play. This will give you additional keywords for searching (to add to the keywords you can extract from the issue description itself) and will help you locate the relevant tests and the source of the bug.
This, however, raises a problem. Some of the issues would require many hours of play to get a player character to the required level to be able to acquire the objects necessary and get to the NPCs involved in the issues. You don’t have that time to spare if you are going to get your fix tested and sorted in the timescales given for the coursework.
Following common practice, the Stendhal team have provided facilities for supporting targetted replication of issues through manual testing. These facilities allow you to create a player character with admin status. This gives the player special powers — including the ability to teleport right to any NPC, to summon any item or creature, and to interrogate the internals of any item from within the game.
Information on these useful features can be found on the Stendhal wiki at stendhalgame.org/wiki/Stendhal:Administration
Think carefully within your team about how you will use these admin features, and especially what test accounts you will use. The admin accounts are specified through the contents of a file in the source code base, and therefore something that is potentially under version control. You don’t want to mess up your Git history by continually committing and changing everyone’s favourite test accounts in this file. A little coordination at the beginning can keep your Git history clean, while allowing everyone to be able to access at least one admin level player account.
20.3 Getting Inspiration for Writing your Own Test Cases
Most people find writing test cases for bugs in large unfamiliar code bases very challenging. This is normal and to be expected. Writing test code is quite a different style of coding than most of you will be accustomed to, and you are working blind, since you don’t have much existing experience of working with this particular code. The idea of the first team coursework exercise is for you to get experience in writing fairly simple tests. It will be difficult at first, but remember that we are on hand to give help whenever you need it. Don’t sit stewing in silence because you don’t know what you are doing. Just ask!
A major source of inspiration for your test cases is the existing test suite. A good starting point for writing your own test case is to look through the code base for test cases that are similar to the one you want to write.
In some cases, you may find that test cases already exist for the piece of functionality you are working on, and you just need to add some additional cases to cover the specific functionality affected by the bug you are solving. That is the easiest case. If there is no test at all for you to work from, you can look at similar tests for ideas. For example, if you need to write a test that checks the properties of an object, you can look for other tests on the Item class and get ideas from those. Or, if you are writing a test for a quest, then you can look at tests for other similar quests, to get an idea of how these tests are structured, and what testing utility code the Stendhal team have provided to help you get started quickly.
Sometimes you might need to put ideas together from two different places to write the test you need. For example, if you are dealing with an issue that describes an error in how pets are affected by stings from poisonous creatures, you might look for tests that deal with poisoning of player characters, and tests that deal with the health of pets. Putting the ideas from these two tests together helps you write the new test you need. You can also look at the production code for ideas when working with functionality that is not well covered by tests.
Existing test cases will also give you lots of useful tips on where to locate your new test code, whether in an existing test class or whether you need to create a brand new test class for your issue.
You should not feel embarrassed about copying and pasting existing test code and modifying it to fit your own issue. This is a normal survival technique for software engineers in the wild. Do make sure though that you understand the code you have copied, at least at a high level. Don’t leave bits of code in there when you are not sure what they are for. That will just lead to brittle and slow tests.
Of course, since this is a coursework exercise, you should feel embarrassed about copying and pasting solutions written by the members of other teams for your issues. That would be plagiarism. Just use your own version of the Stendhal code base, and the work of your team members, to build your own tests on.