Effective Asserts

Asserts are a common tool in the development of software. They are used to validate "stuff" in our programs, and they can be quite useful to trap bugs when you've got a team of people working on a single piece of software. The humble assert macro evaluates an expression and if that expression evaluates to false (i.e. the assertion doesn't hold), it takes some action to indicate the error condition and give clues as to what caused it to occur. Having an enormous number of asserts in your codebase serves not only as documentation for the functionality by indicating the expected conditions at that point of execution, but they also serve as living tests that your code is actually maintaining those conditions as it executes.

Charles Nicholson over at Power of 2 Games wrote a post some time ago on their blog, describing (and making available) a very cunning implementation of this macro, and he spends a good number of words on the way to construct such a thing so that it can behave exactly as you want it to. If the dear reader has not ingested all that Charles has to say on the topic, I wholly suggest that if this topic interests you, you do.

Now, since the implementation of the assert mechanism is now a solved problem (thanks, Charles!), I feel that it is time to discuss ways to most effectively use them.

So, what are effective asserts? Asserts that are effective are asserts that tell you:

• when they are failing
• why they are failing
• where they are failing

Ok, obvious enough, eh?! So what's the point of this post? Well, we're reaching back to the definition of assert itself, with the goal of better defining when it is appropriate to use the macro in the first place. WordNet, via Dictionary.com, says that assert means 1) to state categorically, and 2) to declare or affirm solemnly and formally as true. What this means is that the assertion (i.e. the expression inside the assert macro's parentheses) is not saying, "this should be the case." No, quite a step away from that, the assertion says, "this is the state of the environment at this time." Yes, indeed, anything to the contrary indicates a logical failure in the application. The program cannot be allowed to continue in the face of a failing assertion for this reason alone. Taking a no-nonsense stance on this, it means that a program with one or more assertion failures cannot be executed further until the cause of those failures is addressed.

In practice, what does this mean? In the simplest form, assertions validate programming logic. Typical usage is to validate preconditions for functions, although validating postconditions is certainly useful as well. However, it is important to keep one's eye on the fact that the use of assertions should be restricted to logic errors. It doesn't make (logical) sense to assert that the computer has a mouse connected, or a display, or even that a given file exists on the hard drive and has appropriate contents. Even if the program requires all of these things, their non-existence or invalid contents are not a logical failure of the program. Looking at the default assert implementation on many systems, provides a view into the only valid thing to do in the face of assertion failures, and that's to call exit(). After all, if you know that the program is in a logically invalid state, what else can you expect to accomplish?

Calling exit() in the face of a missing data file, however, is less than user-friendly. Indeed, any errors that occur outside of the program's control should be handled by and with a user-friendly system to display a message along with information on how to remedy the error condition. This is not the job of the humble assert macro.

Looking at this hard-line stance of "assert must kill the program in the face of failures" from the other direction, if the program can continue in the face of any given error condition, then the assertion itself isn't valid; the truth of the asserted expression not a requirement for proper execution of the program, despite its appearance in an assert macro implying as much. In a sense, it's like an out-of-date comment, the "requirement" put forth by the assertion is not a requirement in any way.

Assert is very handy to use to capture error conditions in your program. It is perhaps a little too handy, though, as it often falls into use as a general-purpose "tell the user something" mechanism. At best, this leads to frustration on the development team ("I don't care if a mouse is attached for the work I'm doing, so why am I getting all of these pop-ups?"), and at worst it leads to confusion about how the system works ("this assertion says the pointer is non-null, but the code explicitly and correctly handles the null case?"). Clearly neither end of this range is desirable, so it's best to keep asserts for the things that the term itself implies, statements of truth.

Excuses

Sorry dear readers, I've been away for far too long. Working in game development will do that to a person. I'm preparing the second platform's version of the second game to be completed in two months (both at the end of an almost year-long cycle), and it's been hairy to say the least. Time spent not working has been spent working around the house and working on my car, so I haven't really given much thought to the sorts of content that would be appropriate here.

I have, however, done a talk proposal for a certain conference, and I'm going to flesh the whole thing out and do a series here, for the enjoyment of whoever finds joy in technical discussions. I'll use this forum as somewhat of a dry run for the talk, which will eventually be given somewhere. I just don't yet know where. :)

Hang tight. Games are almost done for another year.

Pragmatism and passing away

This post is a considerable diversion from the recent topics of discussion, and if you, fair reader, want to avoid feeling depressed, it'd probably be best to skip this one.

Yesterday I spent a good chunk of the day at a funeral. It was the first time I have buried a friend. That's not what this post is ultimately about, but it is what evoked these thoughts and questions.

In this modern and connected world, what happens when someone passes away? What happens to the web presence? The domains, email addresses, myspace account? What happens to one's media collection, widescreen TV, and car? What happens to the gigabytes of, umm, "fringe-interest media" that one accumulates? When I've thought about my own possible passing, near- or far-term (hoping for the latter!), it seems that Google will perpetuate this blog for an eternity, but email to my domain will collect for a year before I get cut off for non-payment, but my gmail account will likely collect email for the rest of eternity also. I find this very strange, and somewhat unsettling.

While the presence is now virtual in many senses of the word, it's also somewhat indelible. Hunting for a new place to lay a new blog, I tried all sorts of clever variations of words related to the topic and everything was taken. Sadly, none of them were actually utilized; one had a one post that's four years old, another had one post that was random typing on a keyboard ("does this work"-style), and three or four others had no posts at all (but creation dates more than a year ago). Now here we are in this new frontier, quickly staking our claims and then in many cases letting them completely rot. In some senses I'm glad I'll have at least a little more content to be my legacy, but in other senses I don't really want to consume a place that someone else might want to inhabit after I'm no longer using it. I suppose one could say that's part of the reason why I made up a word to name this thing; it's slightly less likely that someone else will choose to take it up. ...by that time, my hold on the domain through Network Solutions will likely be expired anyway.

Good times.

Lessons from Lean Manufacturing

I talk about Agile development with anyone who'll listen. It's especially interesting to me to talk with non-game developers, because other development projects have concerns that I've often never considered. Recently I was talking with a couple of guys who I play in a band with about their day jobs, which are in manufacturing.

The topic of Lean Manufacturing came up. Wow, were there some opinions there. Terms were thrown around like Black Belt and Six Sigma and other things that I googled at the time, but generally the whole affair was pretty well derided by these compadres of mine.

As their story went, to comply with the Lean Manufacturing rules, one needed to have some number of people to fill various titular roles. Ostensibly, these folks are there to provide support for the needs of the rest of the staff, support that was never there before but which was always needed. Theoretically you could convert or otherwise transition some of your existing staff into these roles, but "full compliance" demands certification for these roles and, as it was explained to me, their employer desired an "out of the box" solution so chose the route of hiring a number of consultants to fill these roles.

In my head I compared this to the agile "methodology" with which I am most familiar, Scrum. Scrum "creates" a number of roles in the scrum master and product owner, among others, although in my opinion and experience, it is highly desirable to take these people from the team. Insofar as compliance goes, by-the-book Scrum doesn't require anything of the participants outside of an understanding of what it is they're doing. Over the past few years there's even been a growing drive on the certification front, with Scrum certification courses happening around the world.

My question is, is this really necessary? This came up because of the Lean Manufacturing talk, actually. These friends of mine asserted that these new people in their manufactured roles added nothing but bureaucracy to their jobs. That tickled an itch of mine, because the whole point of Agile Development is to make it easier for the team to make the software. For any given participant, the amount of new bureaucracy should be easily offset by a reduction in other non-productive tasks that get done, overall yielding more productive time per hour worked.

As the story goes, in this implementation of Lean Manufacturing, there was only overhead added and no new efficiencies. A checkbox for the corporate certification but no actual tangible benefit.

I fear that this is where a blindered approach to agile practices could lead, as well. Already I know of some in my industry who get all of their people "certified" for various of the practices, but to what actual tangible benefit? (Disregarding that it's nice for a company to send you to a conference, because I could think of a number of other conferences more relevant to what we do.) Especially when the "certification" is of the "give us money and we'll certify you" variety; how can someone claim "master"-level command of a subject if no sort of examination was given to test that skillset and knowledge base?

This brings me back around (in my mind, anyway) to one of my earliest posts here, and that is the single most important takeaway for me from all of the agile learning I've done over the past five years. That is, simply, that it's not the process that matters. Devotion to a book or a method or a procedure is only suitable as long as it's applicable to the group that's using it, and it's only applicable to the group if it actually helps that group to accomplish what they're working toward.

Clearly I'm not a fan of certification. In general, I'm not a fan of the "institutionalization" of anything that we do as game developers, because there is no silver bullet. There is no one-size-fits-all. We've got a million problems before us (ok, only 750kloc in my current project but lets assume for the sake of argument that there is more than one bug per line), and no game is the same as any other. This is great! There is huge variety in the teams out there, different teams doing different games with different development strategies, and it would be foolish to think that any one method of doing things would apply across the board. Sure, certain aspects of Scrum, for example, could be useful across a number of teams. Certain bits of XP will work out of the box for a broad range of developers. We can, and probably should, figure this out through experimentation with our own teams, though. We shouldn't, and will probably fail if we try to, go to a conference, hear about ProcessX and say, "this is what our team will use."

...and that is what was bothering my friends in the Lean Manufacturing shop. All show, no go. The "Black Belts" were (or at least, appeared to be) there to ensure compliance, rather than being the thoughtful senseis that the people behind Lean Manufacturing likely intended. But then that's why I thought of this comparison, because I know plenty of folks who are in "agile" shops who swear up and down that it ain't making their lives any easier, it's just bullet points on the marketing brochures.

Agile helps development as much as it's allowed to help. It is empowered by the flexibility that it is given. As much as the team is a living organism, so the processes that the team uses needs to breathe. Inhaling and exhaling, in with the reflections on the process, out with the bits that aren't ineffective. An open mind is all that is required. I would imagine that Lean Manufacturing is intended to make better products faster, fairly obviously since otherwise it wouldn't even be considered for implementation. It's too bad, and I feel sorry for my friends there that the people running the joint don't "get" it. A manager has got to truly grok agility to be able to pull it off, but it is sadly clear that many do not.

Less Yields More?

One of the promises that agile makes is that its adoption leads to less crunch. The implication that is readily available here is that the developers spend less time at their places of employ, more time with their families, and through the magic of agile, the same, if not more, work gets done. How is this possible? Doesn't this assertion violate some fundamental law of conservation? How can it be that any change in process can lead to more productivity?

In Code Complete (in section 28.5), Steve McConnell cites a number of studies that suggest there is over a factor of ten difference in the productivity between programmers. Further studies have indicated that there are also significant, though less pronounced, differences in productivity between otherwise-comparable teams working on identical projects. Is this relevant to the discussion on Agile in any way? In my opinion, most definitely. I would posit that the differences in productivity are related to work habits as much as, or quite likely more than, anything else. Many times I have heard people say, "Agile isn't telling us anything new," and "Agile is merely repackaging things we should be doing anyway." Maybe some of these things, which we all know we should be doing anyway, are some of the things that productive individuals and teams actually do in their own work.

I do fully embrace the idea that the Agile practices can lead to increased productivity. Many of the practices can be applied by the interested individuals, "under the radar" if you will, and those developers will find their own productivity improve. At the end of the day, though, the most effective way to increase one's own productivity is to in have the desire in the first place to become a better programmer. It is those people who want to become better programmers that are the ones seeking out the means by which they can become better; those reading the various forums and learning about any number of techniques to make themselves better at what they do are clearly more likely to make changes in their own work habits to make themselves more effective.

Agile can present enabling technologies to a team desiring to become more efficient. After all, most of the practices are things we've been told we should be doing since we started programming. By themselves, though, they are no magic bullet. Truly increasing performance requires that those involved actually desire to so improve. Doing so otherwise is like washing a cat, it yields only pain and suffering to the washer, while thoroughly aggravating the cat. Avoid the scratches and make sure that the others involved want to take part.

Agile Ain't Easy

Agile's a big deal these days. Everyone is talking about it, it's getting a lot of press, and by the sounds of it, you'd be leaving your company's neck out for the competition to come snack on if you do anything but put your company on the Agile track it Right Now.

Why does it seem then that so few organizations are actually going ahead and installing one of the multitude of Agile Systems? Why are many of the people who are using these practices complaining that their organizations feel anything but agile?

It really comes down to the fact that the literature talks about Agile as if it's easy to install and execute. The problem is, sadly, that it ain't. It is in fact really quite hard to implement completely.

So what's up, then? Why are there so many people out there generally tossing words around as if it's dead simple to just turn Agile on in your own organization? The fact is that the individual practices are pretty simple to do, so they make for good articles and lectures. The infrastructure to support the various practices, whatever your flavor of agile, shouldn't take more than a couple of days to put into place. What tends to be the hard part, though, should in many ways be the easiest because it doesn't require any purchases or any sign-off procedure or any of the other corporate trappings that often interfere with getting things done. This part, which is perhaps the most important step in adopting Agile in any form, is the change in mentality, mindset, and focus of the management team that is required to participate in anything that truly is to be considered Agile.

So, what is this change in mentality, and what's hard about it? In short, the change inverts much of the management structure that exists in the "traditional" development methods. Part of the difficulty is that the manager's role is now almost exactly reversed from what it was before in practice, but even more insidious are the parts which require a substantial shift in the underlying principles that the manager needs to hold dear. No longer is the manager really "the boss." The manager certainly stays on the hook for the development goals, but this person's role changes from one of authority to one of service to the team. This is a tough change for many managers to make.

Indeed, it has been said that this is one of the hardest parts of switching to Agile, and is one that many simply fail to make at all. The other difficulties are similar, in that truly being agile requires comprehensive changes in the management structures that surround the developers, to enable those developers to be truly productive.

One picture of how this can come together is to examine the feedback mechanism by which the "scheduling" of agile projects occurs. In Scrum, for example, there's a backlog of tasks to be done "at some point in the future," and every iteration cycle the team takes some number of tasks from that backlog and sets their "iteration goals" in such a way as to make visible forward progress with the product while committing to deliver an amount of functionality that they believe they can deliver. Compared to a more typical development cycle where the management defines the work to be done over the current cycle, the obvious change hereis that the people doing the work are selecting and committing to only those tasks which they think they can deliver by the end of the iteration cycle. The more insidious effects, however, are that the management team may still cling to a hope that the team could (or even worse, that they should) deliver more functionality. The management, then, need to embrace the idea that the team knows best how things get done, and how long it takes to do these things. Otherwise you're left with friction between what the team is intent on doing and what the management wishes the team would do, and friction is really what Agile strives to eliminate in the first place.

So is long-term scheduling out the window? Not at all; long-term scheduling actually becomes more viable in this model. Since the parties are no longer in an adversarial relationship, they are able to discuss product goals and reach a mutual understanding of what needs to get done and what it'll take to get it done. Agile practices make it easier to see how much is getting done as the project progresses and future development can take this historical information into account while planning. Again, though, management needs to embrace the idea that they are not in control of the project's destiny. The fact is that in a traditional development environment the management is no more in control, but the power structure is comfortable for some and the power game is well-played by many. For agility to take root and truly be effective, the management needs to recognize and accept that they were never in control to begin with, and embrace the idea that the team is the best source of information upon which to base the future plan.

Additionally, as much as the game industry is a hit-driven one, it is also an opportunistic one (and I would guess that much of the software industry is similar). Slipping by even six months can mean that the competition can barrel on by and all of those cool features that were in the plan mean nothing because "now everyone's got them." Shorter development cycles also require the team to focus on what really matters, the core areas and features that will sell the product. Longer and longer development cycles are often the result of "kitchen sink design," where a team tries to add every feature they can think of into the game. Why not monetize often and early, and defer secondary features until the next revision of the product? Since the industry is so hit-driven, shouldn't it be obvious that more releases yield more opportunities for a hit? Agile practices specifically enable this frequent release model, yet many game studios have trouble adopting the underlying mentality to enable themselves to produce games quickly enough for it to matter.

This is only a glimpse into the sorts of challenges that development teams can run into when transitioning themselves to Agile. Sadly, few of the Agile sales force talk about these pitfalls, which in my opinion are much deeper and harder to solve in the long-term adoption of Agile in any given organization. Implementing the things that we all hear about really is easy, but the ease with which agile planning or agile development is implemented belies the complexity of the thought-change that needs to occur to be successful over the long haul with Agile. In these pages I will tend to talk about this side of Agile, as it is in my opinion, very much the important side. Indeed, a change to agile-minded management without implementing the easily-observable aspects of agile can lead to a very effective team. On the flip side, doing pair-programming and unit testing does not automatically make a team agile.

My History

My experience with so-called agile practices started many years ago. Hungry for information and learning, I spent a lot of time reading comp.lang.c++. All of the sudden this new guy (who many know as simply Phlip) appears and starts answering posts with, "oh check out your answer here" and gives a link to Ward's Wiki. So began my introduction to Extreme Programming, and I pushed an early version of CppUnit into my workflow soon thereafter.

From there I took a stab at starting my own game company, during which time one of the other partners stumbled across Scrum and we all agreed to implement it. In many ways, things went spectacularly. Not so spectacular was the fact that money ran out without securing more, and thus it came to a close.

This experience, however, landed me a job at Southern California game developer, where discussion in the interview turned to Agile quickly, and the team appeared really interested to learn more about the whole affair but hadn't done much along those lines to date. This second project ended up bearing significant risk from day one, so I introduced Scrum to help deal with this risk. I also installed a unit testing framework that we could all use as we wished, from my perspective I wanted it mostly to "protect" functionality that I was depending on. Sadly, and despite hitting our agreed-on milestones several months in a row, the studio's primary project was floundering and the financiers really wanted to see more progress. Thus, the second project was shut down, the staff relocated to the primary project, and a Scrum-style system implemented and embraced for the entire team of 70 developers. This studio is now known as somewhat of a poster-child for Agile in games, and I'm glad for having been a part of that.

After that experience, I've been around the block quite a bit. A little Scrum implementation here, a little unit testing there, and some pair programming introductions at this other place. The pair programming was one of the most interesting experiments, because the programmers fought it tooth and nail despite the fact that they ended up getting far more functionality done in any given time period than they had when they were flying solo. At this time, I'm at Yet Another SoCal Developer and basically the guy behind what'll be The New Agile in the company and with any luck, in the industry at-large.