QA Makes the Game Too

The official role of QA (Quality Assurance) in software projects is to assure quality — that is, the software works as designed, and the design is reasonable. They find bugs.

But in a game, they do more. QA plays the game more than anyone, and has the best sense of how it works. Is it fun? Is it too hard or too easy? Does the UI work? What’s missing?

In King of Dragon Pass, the “heroic combat” concept came about because Rob Heinsoo felt something was lacking. (He ended up writing most of these scenes, too.)

I just finished implementing a suggestion from Liana Kerr:

I feel like there’s not a lot of connection between your opening questionnaire and your clan management. I have no emotional connection to the fact that we know the secrets of [redacted], because it’s never referenced again.

Well, it now is. And while a few questions don’t get an explicit mention later, I just made sure that every answer from two questions shows up in at least one scene. (The others are at least mentioned implicitly, like your ancestral enemy, or give bonuses in scenes.)

Earlier, she suggested

Raid adviceAdvice about raiding-related promises currently shows up in the War screen, but if you go to the Raid screen, it doesn’t. As a player I’d be more likely to expect to see it in the Raid screen and would entirely miss it in the War screen.

and

One problem I’ve always had with KoDP is that someone dies and I immediately forget who they were — that is, I just see the name and I don’t necessarily connect it with the face that I’ve been looking at for several years. It may be a little different with Six Ages, since the UI is a little different with the ring members’ names underneath their pictures … Two suggestions: …

and so on. More good ideas that got implemented.

For that matter, it’s not just QA that can influence the game. Much of the current combat feedback is based on a suggestion by Jan Pospíšil.

Not every team suggestion ends up in the game. Some are still on the backlog of possible tasks. But more input makes for a higher quality game.

One Step Build

I just now made build 365, which seemed as good a number as any to mention.

One way I’ve improved the development process is to have an easy way to post a copy of the game for the rest of the team to test. With King of Dragon Pass, this was basically a manual step. Now, there’s a script that increments the build number and does a clean build, then copies it to a web site. It also uses rsync to copy any script files to a Dropbox folder, so QA has easy access to the corresponding source.

This isn’t build automation (it’s not automatic, like Travis or buddybuild), but it’s painless. The basic build takes about 40 seconds (uploading is more variable). And it does mean we pass one of the steps in the Joel Test. Our builds aren’t daily (since we’ve been at this more than a year), but they are at least frequent.

Where The Money Goes

A comment on our last status update mentioned our budget. Our plan is to make a new game for a lot less than it cost to do King of Dragon Pass. Hopefully our experience (with KoDP and games in general) will let us do things more efficiently this time. We will likely do a lot less reworking of certain elements than we did the first time, since we have a better idea ahead of time what should work. And automated testing will let QA concentrate on bugs only a human can find.

budgetplanThe pie chart shows where we expected to spend our budget shortly after starting. The bulk of the expense (44%) was projected to be art.

Note that this doesn’t include any sort of overhead (like development machines, internet access, etc.). And I’m not directly expensing my own time. In other words, this is just what we expect to pay our freelancers.

This was just a plan, and once we were far enough along to see how things were working, we realized had to do more writing and art. So the final percentages will end up a bit different, though art will predominate.

(And yes, the title of this post is an excuse to watch OK Go.)

Tutorial Planning

We’re screen-shot-2016-09-29-at-19-53-09 in the process of working on the game’s tutorial.

We’re starting a bit earlier than we did with King of Dragon Pass (since the game is not done), and we’re also taking a somewhat different approach.

King of Dragon Pass had a scripted tutorial, which tried to take you on a tour of the important elements of the game. But it was too easy to get off the rails and either miss things, or confuse the tutorial. (And as a side note, it was actually not the best first play experience since you raided during planting season — we wanted the tutorial to mention this, but if you then continued the game, you were in a small hole.)

Six Ages will take a more reactive approach. Instead of telling you to switch to the Map screen, and then describing exploration, it will wait until you switch on your own, and then describe exploration. It will also (for example) mention the size of a reasonable sacrifice the first time you’re asked to sacrifice. So you’ll get the information when it actually matters.

One thing I’d like to do is ask King of Dragon Pass players what confuses you about the game. For example, we once got a bug report that clan magic went negative. That’s actually by design, but if it confused someone, we can pop up a tutorial card when it happens. I expect we’ll have a manual too, but if something seems wrong, you’re not going to go look it up.

So please let us know what confused you or you think needs better explanation, here in the comments (or to @SixAges on Twitter).

Treasures

Like King of Dragon Pass, Six Ages will include magical treasures. They’re Gloranthan (they were in the original White Bear & Red Moon game, and lots more in the subsequent Nomad Gods), and they are a useful reward and fun to collect.

And they’re essentially all unique, which is great as a player but less wonderful as a developer.

As a developer, I want treasures to be easier to work with this time. KoDP had some annoying limits on how many there were (adding a treasure meant breaking the file format of saved games), which was an obvious thing to improve. But KoDP had the implementation of treasures scattered everywhere. Basically, each treasure needed specific C++ code, plus it might have to be listed in an IsBattleTreasure or TreasureThe (it’s “Vinga’s Comb,” not “the Vinga’s Comb”) function.

First, we designed as many as possible with script tags in mind. In other words, Robin Laws would describe a treasure’s effects as “+1 to Bargaining and Diplomacy in scenes tagged @outlanders.” Then, I’d specify treasures as Lua data like

modifiers = { Bargaining = 1, Diplomacy = 1 },
requires = { "@outlanders" }

so there’s no treasure-specific code. Compare to KoDP having to insert lines like

case code_Leadership:
    if (ourClan->HaveTreasure(kRingMadeFromVingkotsCrown))
        skill++;

Part of the advantage is that the details of the treasure effect are all in one place, next to metadata (such as name, description, value, use of “the,” and whether a treasure needs to be chosen in battle).

The data-driven approach also handles things like “mood+=13 on any Food win”

onWin = {"Food", "mood", 13},

Combat has some additional complexity, so “+1 Combat when raided by elves” is the Lua table

battle = { type = type_Raid, defender = true, opponentStereotype = "aldryami", bonus = 1 },

To summarize: we tried to design to a number of patterns, wrote general-purpose code to handle the patterns, and then specify how a treasure slots into the pattern with Lua data.

Of course, magic doesn’t always fit a pattern, so we can run an OSL script

everyTurn = "fragment_MostaliMill"

or have a script check for a treasure

code: fragment_SacredTime
[HaveTreasure('bowlOfUnity)] {
    # .magic +=1 each Sacred Time if Expansiveness>0

Not quite as good as pure data but better than requiring code.

Quasi-Treasures

A treasure is now a way to encapsulate a conditional modifier. I started to notice another pattern, where player choice could result in a modifier. Rather than come up with a new scheme, I just made new treasures. (I called them quasi-treasures because they act like treasures but don’t look like treasures. Notably, they don’t appear in lists of treasures in the user interface.) In fact, one of the examples above is actually a quasi-treasure.

Treasures are typically gained permanently and quasi-treasures often go away after a certain number of years (for example, the visiting priest’s blessing lasts five years). This can be handled by the normal GainTreasure/LoseTreasure functionality (i.e. just queue a script to run in five years that executes the LoseTreasure function).

malletOfPlentyPictures?

We discussed showing treasures, and Jan mocked up a treasure illustration. However, I decided against a visual representation. It’s a lot of work just designing what dozens of treasures look like, let alone getting all the pictures. More importantly, it would increase the difficulty of adding new treasures in an update (which we did twice in KoDP, despite the hassle), since they would need pictures.

So this is the only place you’ll see the mallet…

Testing In Bulk

Like King of Dragon Pass, Six Ages is a large, complex game. That means that during development, there will be a lot of bugs.

Graph of scenes written, coded, testedThe best way to find them is to start testing early. I like to start QA as early as possible, once there’s a runnable game. For example, we can start the process of making sure every interactive scene works.

We have a really good QA tester, but like KoDP, there are a lot of scenes, with a lot of branches. One of the process improvements over KoDP is that we have a way to brute force test every response of every scene. This is only a supplement to human testing. It can make sure nothing crashes, but can’t find typos or situations where the wrong person is mentioned. But it does mean we don’t waste QA’s time with testing something that isn’t fully implemented.

Some bugs have patterns. If one scene has a problem after the player has made a major story choice, it’s likely that others will too. I usually track these in our bug database as a “sweep,” meaning once all the scenes are coded, we will have to search the code for possible occurrences. We still report fix bugs as we go, and when this situation came up again in a scene, I decided to try out an idea: change the brute-force testing to set up the story situation, and then run the brute-force test. Bingo! Ten failures. Or, ten places that a person doesn’t have to either read code and see what’s going on (in a code sweep), or write up a bug report (manual testing).

Last night I got a bug report about a scene failing when there was no chief. So I decided to do another brute-force test. This found a lot of problems in scenes that had not yet been tested.

Unfortunately, the brute-force testing of every branch of every scene takes about two minutes (and getting worse, as we code more scenes). So while I probably should run each brute-force variant, for now I’m relegating them to occasional situations. But I’m definitely going to look for more places to do this.