Leveraging QA

giving a potRight now, the game is undergoing internal testing while being finished. Most of the bugs are easy to deal with: fix a typo, clarify something that’s unclear, fix a logic error, or make sure advice and recommendation match.

One that came in today had to deal with being unable to proceed after having decided to give gifts, when your clan had no goods to give. The specific script was

[ChooseYesNo("Do you take a gift?")]
yes: {
    w = ChooseGoods(“What do you give to them?”)
    TransferGoods(otherClan, w)

Normally players who have no wealth wouldn’t be choosing to gift, and it wouldn’t be a problem. But of course part of QA’s job is to test the boundaries.

It would be easy enough to fix this scene, but this has the potential to be a wider problem. ChooseGoods is used fairly often. Any of those situations had the potential of failing, too. One answer to that is to do a code sweep, searching for all occurrences, and making sure they are conditioned. But that’s a manual step, which means it can be prone to error (especially if there are lots of places).

Another is to have the computer do this. We already have a unit test that exhaustively runs scenes. So I set goods to 0 in this test, and found six other problem scenes.

So human QA gives the best results, but automation can give decent testing over the entire game, and it’s easier to set up certain conditions (usually a clan doesn’t stay at 0 goods long, since crafters and traders are continually making more).

Logo Development

When I talked to a marketing consultant, she advised that I needed a logo as soon as possible. For a variety of reasons, I didn’t follow up on that for over a year.

Last month I met with Pixel Parlor, a local design studio. I showed some of the art from the game and discussed my overall goals (e.g. it’s unlikely that we’ll have a physical box, which drove the King of Dragon Pass logo). In the meeting they asked me two questions I didn’t have an answer for: what were some logos I liked? And what were competitive games?

It turned out that was an interesting search. There are an awful lot of clichéd logos for fantasy computer games! So I broadened my search to include paper & dice RPGs and board games. For digital games, I thought the Monkey Island and Legend of Zelda series worked well. I liked the current Dungeons & Dragons logo, with the draconic ampersand (though it doesn’t work as well as D&D, since the three glyphs smudge together without enough contrast). The Fate logo also stood out. And the Scythe boardgame had a good logo.

As for competitive games, since Six Ages will be similar to King of Dragon Pass, there’s really nothing else like it. I suspect it will appeal to people who also like Sunless Sea, 80 Days, Sorcery, Banner Saga, Out There, and Reigns. But I also wouldn’t say they’re competition.

Pixel Parlor presented six design approaches. Here’s a representative subset:

I ran the designs by the Six Ages team and Chaosium (licensor of the setting). There was no clear consensus, so I asked Pixel Parlor to iterate on the two favorites. Here’s a couple examples from round 2:

Again I asked what the team thought. Again there wasn’t a definite favorite. I zoomed out to see all the logos at the same time. When they were small, it seemed like the carved style font was the most recognizable at small sizes. Which would be important in say a Steam listing.

I didn’t really like the E that looked like a greek Σ, however. So I asked for another set of variations. Again there wasn’t a clear winner, but I finally picked the one that wasn’t likely to be confused for a game about ancient Greece.

Hopefully this will prove distinctive and help suggest what the game is about (one of the artists noted that the lettering reminded him of the Gloranthan runes). At any rate, it’s nice seeing in the game.

QA Makes the Game Too

The official role of QA (Quality Assurance) in software projects is to assure quality — that is, the software works as designed, and the design is reasonable. They find bugs.

But in a game, they do more. QA plays the game more than anyone, and has the best sense of how it works. Is it fun? Is it too hard or too easy? Does the UI work? What’s missing?

In King of Dragon Pass, the “heroic combat” concept came about because Rob Heinsoo felt something was lacking. (He ended up writing most of these scenes, too.)

I just finished implementing a suggestion from Liana Kerr:

I feel like there’s not a lot of connection between your opening questionnaire and your clan management. I have no emotional connection to the fact that we know the secrets of [redacted], because it’s never referenced again.

Well, it now is. And while a few questions don’t get an explicit mention later, I just made sure that every answer from two questions shows up in at least one scene. (The others are at least mentioned implicitly, like your ancestral enemy, or give bonuses in scenes.)

Earlier, she suggested

Raid adviceAdvice about raiding-related promises currently shows up in the War screen, but if you go to the Raid screen, it doesn’t. As a player I’d be more likely to expect to see it in the Raid screen and would entirely miss it in the War screen.


One problem I’ve always had with KoDP is that someone dies and I immediately forget who they were — that is, I just see the name and I don’t necessarily connect it with the face that I’ve been looking at for several years. It may be a little different with Six Ages, since the UI is a little different with the ring members’ names underneath their pictures … Two suggestions: …

and so on. More good ideas that got implemented.

For that matter, it’s not just QA that can influence the game. Much of the current combat feedback is based on a suggestion by Jan Pospíšil.

Not every team suggestion ends up in the game. Some are still on the backlog of possible tasks. But more input makes for a higher quality game.

No Kicking While Down

Back in May, I wrote

By the way, the @help/@hurt idea is not implemented at the moment. I’m not sure if it will be, but the flexible tagging system makes it easy to add at any time.

This was indeed easy to add, and it was in fact handled mostly with script tags.

furrows inundatedThe basic idea behind these tags is that we want to avoid death spirals. If you lose a lot of cattle due to disease, the game won’t feel as fun if you then have a large herd mysteriously disappear. You’ll end up in a hole you won’t be able to dig out of. The occasional flood is fine, but if it happens when you are short of food, it’s just kicking you when you’re down.

Of course, mysterious disappearances and floods need to occur, so they can happen more often when you are able to recover. (This may still set back your plans for cattle trading or gifting, of course.)

One issue with implementing this adaptive difficulty is figuring out what constitutes a bad or good situation. The game takes a number of factors into account (herds, wealth, military, and even issues like curses or magical bounties). And scenes have to be tagged. Some were designed to be helpful or harmful, but many are more of a grey area.

It’s worth noting that we don’t actually force or prevent anything, just alter the odds for picking scenes randomly. You can think of this process loosely as “pick a card, any card” each season, where cards are returned to the deck after several years pass. If times are bad, we essentially put multiple copies of the favorable scenes into the deck, and replace and reshuffle if a disaster is pulled.

As in King of Dragon Pass, we also do something similar for the yearly omens. Maybe an ancient economy experiences crop failure about once very seven harvests, but two in a row seems like a bit much to hit the player with. The goal is to give them challenges, not an accurate simulation of widespread famine.

I added this feature only recently. It required going through all the scenes and determining which would be reasonable to use to help the player, or to make less common in bad times, and adding the right tag. The determination of good and bad times was already used to give advice, and needed only a little tweaking for the new use.

Since this is only just in the game, it hasn’t been tuned. So I’m also tracking the values in the metrics we capture.

Hopefully all this will keep the game challenging but not too frustrating.

March Update

the Manual and Help buttonsThe game is still missing a bunch of assets (both art and music), but the user interface is basically complete. There’s a first draft manual, and the game has been winnable for some time. So we’re starting to look at tuning and polish.

QA is still rigorously testing every scene. 89% are completely tested, and most of the rest have at least some coverage.

We’re still finding plenty of bugs, though usually you can keep playing (that is, they’re not severe enough to crash or prevent progress).

Part of the tuning process has been figuring out what the game still needs. I just added a new scene this week, which is one of the reasons it’s still hard to talk about a release date.

One Step Build

I just now made build 365, which seemed as good a number as any to mention.

One way I’ve improved the development process is to have an easy way to post a copy of the game for the rest of the team to test. With King of Dragon Pass, this was basically a manual step. Now, there’s a script that increments the build number and does a clean build, then copies it to a web site. It also uses rsync to copy any script files to a Dropbox folder, so QA has easy access to the corresponding source.

This isn’t build automation (it’s not automatic, like Travis or buddybuild), but it’s painless. The basic build takes about 40 seconds (uploading is more variable). And it does mean we pass one of the steps in the Joel Test. Our builds aren’t daily (since we’ve been at this more than a year), but they are at least frequent.


Like King of Dragon Pass before it, Six Ages will have a lot of text. Which means a lot of opportunity for typos or other misspellings.

Most of the text is in OSL scripts, such as these excerpts:

saga: <expeditionLeader> was attacked, but returned home with <his/her> escort.
sagaText: The worshipers suffered the same fate.
text: We eventually pieced this together from stories told by wandering traders.

The scene compiler outputs all strings into a single text file. It looks something like this:

<expeditionLeader> was attacked, but returned home with <his/her> escort.
<expeditionLeader> was attacked and wounded, but returned home with <his/her> escort.
Unfortunately, <he/she> lost the livestock they were driving home.
<expeditionLeader> and <his/her> escort <disappeared mysteriously/were ambushed by trolls and completely devoured>.
The worshipers suffered the same fate.
We eventually pieced this together from stories told by wandering traders.

And that file can be spellchecked. I just use TextEdit. The biggest issue with proofreading is that the game uses a lot of proper names and jargon specific to Glorantha. Luckily it’s easy enough to add “Orlanth” to the dictionary (or the ignore list). More problematic is that variable names (like expeditionLeader) also show up here, though ignoring them usually works too.

Another issue is that it’s a big file. It may be generated by our tool but it still takes a human a while to review, so that doesn’t happen often (in fact the first complete review was today).

This is just a brute force pass. Many typos end up with words that are spelled correctly. And once in a while game-specific names get misspelled. So QA still needs to keep an eye open for problems.

Embarrassment of Riches

I’ve written before about art thumbnails, the initial sketch for a scene that we use to make sure the layout and storytelling work.

three alternate thumbnail sketchesThis set seems particularly hard to choose between. They work in a purely functional sense (i.e. they can be covered with text and still read well). But each has some nice elements. For example, alternative 1 has a closeup, which is nice. But alternative 2 is good for showing the extended family. And I particularly like the children in all three.

Emailing Debug Info

Six Ages tries to make it easy for the player to send us debug info.

Previously, we saw that the game keeps a pair of debug logs on the player’s device. Back in the late 1990s when were did this for King of Dragon Pass, we asked players to navigate to those files (with Finder or Windows Explorer), and send them.

When I adapted King of Dragon Pass for iOS, I had to use a different approach (since the files aren’t accessible). Luckily, iOS had an easy way to send an email message, and we were using Fogbugz to do bug tracking. So the game let the player mail in a report (to the special Fogbugz address), automatically attaching the log, as well as the most recent saved game.

In Six Ages, I’ve improved that slightly, compressing the files into a ZIP archive, and including a few other debug files that help with tuning.

I’m also detecting certain kinds of crashes, and offering to email the report. (In this case, I take a screen shot and include that in the ZIP file.)

Since some crashes prevent this, I also try to detect a crash on the next launch, and offer to send the report then.

This is all fairly trivial code (I’m not using one of the third party libraries that no doubt do a much better job with edge cases), but it makes sending the debug info much simpler for the player. Which makes fixing the bug much easier for me.

Debugging: The Log

Like King of Dragon Pass, Six Ages logs all scripts, which helps show exactly what happened and the state of the game.

Right now a lot of my time is spent fixing bugs in the code. Given that the game is large and complex, it can be hard to know just what happened that led up to a bug. So we keep track of everything important.

This is actually something that we did in King of Dragon Pass. Shawn Steele was implementing our scripting language, OSL, and wanted a way to check that things like conditionals and calculations worked. So he came up with a way to log this to a file. You can see his focus from some of the options:

kAllBranches = 0x01,
 kListSizeing = 0x02,
 kTraceOSL = 0x04,
 kTraceMath = 0x08, // Extra COSL::StartMath debugging information
 kSetVariables = 0x10,
 kTraceFixed = 0x20, // Extra CFixed debugging information
 kLoadVariables = 0x40,
 kGetStrings = 0x80,
 kTraceTribes = 0x100,
 kTraceOSLVariables = 0x200, // CFixed::PrintFileDebug should print OSL Variable Names the hard way
 kTraceMathResult = 0x400,
 kTraceList = 0x800, // Attempt to show list content

The output shows addresses and opcodes, showing its focus on debugging the language:

OSL 0xd8b694 Running from 0, fRunDepth 0:
0000 : 0202 019E Picture "scene018" 
0002 : 0203 0001 Position 0001 
0005 : 4005 E
 Loading Variable 4005 (E) 0.000 (gValue)
0006 : 0800 = 
0007 : 068D RandomElement 
0008 : 0480 ( 
0009 : 070E ClanMembers 
000A : 0841 - (subtract) 
000B : 0718 RingMembers 7ffa0000 -- P00000000:0021a700 (gValue)
 Result: 7ffa0000 -- P00000000:005e18fe (gValue)
000C : 0481 ) -- gValue = 7ffa0000 -- P00000000:005e18fe (gValue)7ff20001.0000 : Brenna (gValue)
 Setting Variable 4005 (E)
 Result: 7ff20001.0002 : Brenna (gValue)
000E : 4021 otherClan
 Loading Variable 4021 (otherClan)7ff30017.0000 : Blue Jay (gValue)
000F : 0800 = 
0010 : 068D RandomElement 
0011 : 0480 ( 
0012 : 068A NeighboringClans 
 4 COSL::Neighbors:
 9 Boskovi
 15 Blackrock
 18 Greydog
 23 Blue Jay

0013 : 0481 ) -- gValue = 7ffb0000 -- C00000000:00848200 (gValue)7ff30012.0000 : Greydog (gValue)
 Setting Variable 4021 (otherClan)
 Result: 7ff30012.0000 : Greydog (gValue)
0015 : 4012 R
 Loading Variable 4012 (R) 0.000 (gValue)
0016 : 0800 = 
0017 : 0600 FALSE 0.000 (gValue)
 Setting Variable 4012 (R)
 Result: 0.000 (gValue)
0018 : 0212 01A4 Saga "<4005> told us we should take in Orlkensor Bronzebones, a warrior outlawed by the <21>.plural." 
 Loading Variable 4005 (E)7ff20001.0002 : Brenna (gValue)
ReplacePlaceHolders...7ff20001.0002 : Brenna (gValue)
 Loading Variable c321 (otherClan)00000000.0000 : Greydogs (gValue)
ReplacePlaceHolders...00000000.0000 : Greydogs (gValue)
001A : 0201 01CF Music "IsItAdventure" 
001C : 0A00 NewChoices

Once OSL was reliable, it turned out that the log was useful to help debug the OSL scripts themselves. If something weird happened, you could see what code branch was taken, and what some of the variables were. For example, the output above shows all the neighboring clans

And we output other information to the log, such as the scene queue, some of the clan decisions, and more.

The debug log can grow quite large (a quick search shows one at 7.7 MB), so the game makes a new one every time you launch. But if you had to relaunch because of a crash, that would mean that any evidence would be deleted. So actually, we rename the log, and actually delete the previous one.

While I reworked OSL for Six Ages, I based it on Shawn’s work, and wasn’t so concerned about debugging the language itself. Instead, I wanted to focus on the scripts, since that seemed like where most of the bugs would be.

<OSL: 0x170194ec0> ® 2 Affiance her to …
: Saga "We affianced her to <.an> <ourClan> groom." {a} {C_1 Arrowstone}
: (ourClan).commoners {277}
: += 
: # 1 {277} {278}
 ourClan.commoners ← 278
: ChooseYesNo
: ( 
: String "Do you accept the 10 cows?" 
: ) ↖︎ = Do you accept the 10 cows?
sendToCurrentScene: kNewChoice
… exploded
Restarting OSL (self.result=kNewChoice)

This is more compact (thus easier to read): more values are shown on the same line, and it doesn’t bother to show addresses or opcodes.

The next post will talk more about how to make use of this.