blogger.com sucks. so long!

I'm moving my site to another provider. Blogger.com totally sucks. Newline handling is daft, and I changed the setting to fix it but wouldn't you know it it removed ALL linebreaks from my old posts. Great! Source code handling is also abysmal, so...

I'm going to move here in the next few days. See you soon!

Do you understand your code?

All but the most trivial of programs accomplish tasks that require the coordination and combination of multiple concepts. How those disparate concepts are arranged can have a huge impact on the comprehensibility of the program. Complexity in the software not only impedes others' understanding of that software, but also negatively impacts the modifiability of that code. On the flip side, time taken to simplify a body of code leads not only to a more clear understanding of the problem and the solution being solved, but often presents opportunities for optimization in a number of different ways.

The complexity of a given body of code can often be determined by casual observation. In the case of a function, how long is it? How many branches does it have? How much state does it track? Of that state, how much is external to the function; how broad is the scope of that state (is it truly global data, or is it more restricted as members of a class instance)? The more that these attributes are represented in any item of code, be it a function, a file, a class, a library, the more complex that item is. The more complex a body of code is, the more resistant to change it is. The more resistant to change a thing is, the harder it is to add or change functionality and fix bugs.

I don't think it's hard to argue that complex code is difficult to work with.  Software developers have come up with a number of catchy phrases to help guide a reduction in these sorts of complexity. "Do Simple Things" encourages us to, well, make the code simple to start with; while it doesn't directly indicate a method to reducing complexity the "mandate," if you will, can expand to the task of reducing complexity by taking small, aka "simple," steps. The "One Responsibility Rule" tells us that any unit of code (a function, a file, a class, a library, and possibly even an entire executable program) should focus on doing one thing and doing it well. The Law of Demeter encourages us in a similar way; it tells us very specifically about what sorts of data interactions any unit of code is allowed to have. "Tell Don't Ask" is a corrolary to the Law of Demeter that suggests we author our code to give the necessary state to the things it calls, rather than requiring a given piece of code to query near and far for that state. Finally, a personal favorite of mine, "Feature Envy" is a body of code that seems more interested in the state of things outside of itself than it does on its local state. Perhaps I may also take the opportunity to add a humorous statement that I read somewhere once upon a time that suggested that every branch in your program is a bug waiting to manifest.

The challenge then becomes simplifying the complex thing. Something can only  practically be complex if it is made up of a number of less complex things. So the first step is to separate those less complex things so that they become individual items each of lower complexity. As a trivial example, consider an application like 'sort' which reads a file into memory, sorts the individual lines, then writes those lines back out. A straightforward implementation of this could readily be done in one function, open the file, read the file, close the file, split the file into individual lines, sort those lines, and then open another file, write the lines, close the file, done. At this point it's not terribly complex, but it can be made simpler if the mainline consists of three calls, one that reads the file into the internal data structure, one that sorts the data structure, and a third that writes the data structure. At a high level, this program has become less complex to the observer exactly because there's less to comprehend, there's less state to manage. An additional benefit that comes to light when the sort routine itself needs to start being more complex, if you want to reverse the sort or sort the lines based on something other than the string of characters starting in the first column. When we start talking about this, we realize that we need to pass switches on the command line, so we add a fourth function to our "main", one to parse the arguments and set up some state based on what was passed. The actual sorting function then examines the state that represents those parameters, and anyone looking at the program code doesn't need to deal with the mechanism of parsing the command line arguments unless specifically looking at the function that does that handling.

I made a big deal about state above, so I'll come back to it. The broader the scope of any element of non-constant data, the more likely it is that it will not be configured in the way you want it to be when it comes time to use it. This makes it more difficult to depend on, and worse, even though everything may work fine at the moment you make a dependency on state with a broad scope, if someone else comes in and changes that state without completely knowing the effects of that change then the previously-working code will not be working anymore. Many would chalk such a change up to a bad or otherwise lazy programmer on the team, but if we're honest with ourselves we realize that we're under all sorts of pressures to ship our software, and our software is so big that any one person can't truly grok the entire body of code. In this case, the onus is on each of us to make our software as robust as possible in the face of external changes. Finally, if you have a piece of code that only operates on state local to that code, that state is pretty much immune to changes to the outside world, assuming that the parameters it is handed remain valid. This is a thing of beauty and we should endeavor to keep it that way.

As developers on a software team, we should all keep in mind that other people will be reading our code, and we should write our code accordingly. If we author code to be the simplest code that it can possibly be, then anyone, even the most junior team member, should feel comfortable reading, understanding, and editing that code. The easier it is to edit a bit of code, the more readily it will grow to meet the user's needs. Finally, who knows, with simpler code, it may just be that new insights into the problem being solved will be gained!

Effective Asserts

Asserts are a common tool in the development of software. They are used to validate "stuff" in our programs, and they can be quite useful to trap bugs when you've got a team of people working on a single piece of software. The humble assert macro evaluates an expression and if that expression evaluates to false (i.e. the assertion doesn't hold), it takes some action to indicate the error condition and give clues as to what caused it to occur. Having an enormous number of asserts in your codebase serves not only as documentation for the functionality by indicating the expected conditions at that point of execution, but they also serve as living tests that your code is actually maintaining those conditions as it executes.

Charles Nicholson over at Power of 2 Games wrote a post some time ago on their blog, describing (and making available) a very cunning implementation of this macro, and he spends a good number of words on the way to construct such a thing so that it can behave exactly as you want it to. If the dear reader has not ingested all that Charles has to say on the topic, I wholly suggest that if this topic interests you, you do.

Now, since the implementation of the assert mechanism is now a solved problem (thanks, Charles!), I feel that it is time to discuss ways to most effectively use them.

So, what are effective asserts? Asserts that are effective are asserts that tell you:

• when they are failing
• why they are failing
• where they are failing

Ok, obvious enough, eh?! So what's the point of this post? Well, we're reaching back to the definition of assert itself, with the goal of better defining when it is appropriate to use the macro in the first place. WordNet, via Dictionary.com, says that assert means 1) to state categorically, and 2) to declare or affirm solemnly and formally as true. What this means is that the assertion (i.e. the expression inside the assert macro's parentheses) is not saying, "this should be the case." No, quite a step away from that, the assertion says, "this is the state of the environment at this time." Yes, indeed, anything to the contrary indicates a logical failure in the application. The program cannot be allowed to continue in the face of a failing assertion for this reason alone. Taking a no-nonsense stance on this, it means that a program with one or more assertion failures cannot be executed further until the cause of those failures is addressed.

In practice, what does this mean? In the simplest form, assertions validate programming logic. Typical usage is to validate preconditions for functions, although validating postconditions is certainly useful as well. However, it is important to keep one's eye on the fact that the use of assertions should be restricted to logic errors. It doesn't make (logical) sense to assert that the computer has a mouse connected, or a display, or even that a given file exists on the hard drive and has appropriate contents. Even if the program requires all of these things, their non-existence or invalid contents are not a logical failure of the program. Looking at the default assert implementation on many systems, provides a view into the only valid thing to do in the face of assertion failures, and that's to call exit(). After all, if you know that the program is in a logically invalid state, what else can you expect to accomplish?

Calling exit() in the face of a missing data file, however, is less than user-friendly. Indeed, any errors that occur outside of the program's control should be handled by and with a user-friendly system to display a message along with information on how to remedy the error condition. This is not the job of the humble assert macro.

Looking at this hard-line stance of "assert must kill the program in the face of failures" from the other direction, if the program can continue in the face of any given error condition, then the assertion itself isn't valid; the truth of the asserted expression not a requirement for proper execution of the program, despite its appearance in an assert macro implying as much. In a sense, it's like an out-of-date comment, the "requirement" put forth by the assertion is not a requirement in any way.

Assert is very handy to use to capture error conditions in your program. It is perhaps a little too handy, though, as it often falls into use as a general-purpose "tell the user something" mechanism. At best, this leads to frustration on the development team ("I don't care if a mouse is attached for the work I'm doing, so why am I getting all of these pop-ups?"), and at worst it leads to confusion about how the system works ("this assertion says the pointer is non-null, but the code explicitly and correctly handles the null case?"). Clearly neither end of this range is desirable, so it's best to keep asserts for the things that the term itself implies, statements of truth.

Excuses

Sorry dear readers, I've been away for far too long. Working in game development will do that to a person. I'm preparing the second platform's version of the second game to be completed in two months (both at the end of an almost year-long cycle), and it's been hairy to say the least. Time spent not working has been spent working around the house and working on my car, so I haven't really given much thought to the sorts of content that would be appropriate here.

I have, however, done a talk proposal for a certain conference, and I'm going to flesh the whole thing out and do a series here, for the enjoyment of whoever finds joy in technical discussions. I'll use this forum as somewhat of a dry run for the talk, which will eventually be given somewhere. I just don't yet know where. :)

Hang tight. Games are almost done for another year.

Pragmatism and passing away

This post is a considerable diversion from the recent topics of discussion, and if you, fair reader, want to avoid feeling depressed, it'd probably be best to skip this one.

Yesterday I spent a good chunk of the day at a funeral. It was the first time I have buried a friend. That's not what this post is ultimately about, but it is what evoked these thoughts and questions.

In this modern and connected world, what happens when someone passes away? What happens to the web presence? The domains, email addresses, myspace account? What happens to one's media collection, widescreen TV, and car? What happens to the gigabytes of, umm, "fringe-interest media" that one accumulates? When I've thought about my own possible passing, near- or far-term (hoping for the latter!), it seems that Google will perpetuate this blog for an eternity, but email to my domain will collect for a year before I get cut off for non-payment, but my gmail account will likely collect email for the rest of eternity also. I find this very strange, and somewhat unsettling.

While the presence is now virtual in many senses of the word, it's also somewhat indelible. Hunting for a new place to lay a new blog, I tried all sorts of clever variations of words related to the topic and everything was taken. Sadly, none of them were actually utilized; one had a one post that's four years old, another had one post that was random typing on a keyboard ("does this work"-style), and three or four others had no posts at all (but creation dates more than a year ago). Now here we are in this new frontier, quickly staking our claims and then in many cases letting them completely rot. In some senses I'm glad I'll have at least a little more content to be my legacy, but in other senses I don't really want to consume a place that someone else might want to inhabit after I'm no longer using it. I suppose one could say that's part of the reason why I made up a word to name this thing; it's slightly less likely that someone else will choose to take it up. ...by that time, my hold on the domain through Network Solutions will likely be expired anyway.

Good times.

Lessons from Lean Manufacturing

I talk about Agile development with anyone who'll listen. It's especially interesting to me to talk with non-game developers, because other development projects have concerns that I've often never considered. Recently I was talking with a couple of guys who I play in a band with about their day jobs, which are in manufacturing.

The topic of Lean Manufacturing came up. Wow, were there some opinions there. Terms were thrown around like Black Belt and Six Sigma and other things that I googled at the time, but generally the whole affair was pretty well derided by these compadres of mine.

As their story went, to comply with the Lean Manufacturing rules, one needed to have some number of people to fill various titular roles. Ostensibly, these folks are there to provide support for the needs of the rest of the staff, support that was never there before but which was always needed. Theoretically you could convert or otherwise transition some of your existing staff into these roles, but "full compliance" demands certification for these roles and, as it was explained to me, their employer desired an "out of the box" solution so chose the route of hiring a number of consultants to fill these roles.

In my head I compared this to the agile "methodology" with which I am most familiar, Scrum. Scrum "creates" a number of roles in the scrum master and product owner, among others, although in my opinion and experience, it is highly desirable to take these people from the team. Insofar as compliance goes, by-the-book Scrum doesn't require anything of the participants outside of an understanding of what it is they're doing. Over the past few years there's even been a growing drive on the certification front, with Scrum certification courses happening around the world.

My question is, is this really necessary? This came up because of the Lean Manufacturing talk, actually. These friends of mine asserted that these new people in their manufactured roles added nothing but bureaucracy to their jobs. That tickled an itch of mine, because the whole point of Agile Development is to make it easier for the team to make the software. For any given participant, the amount of new bureaucracy should be easily offset by a reduction in other non-productive tasks that get done, overall yielding more productive time per hour worked.

As the story goes, in this implementation of Lean Manufacturing, there was only overhead added and no new efficiencies. A checkbox for the corporate certification but no actual tangible benefit.

I fear that this is where a blindered approach to agile practices could lead, as well. Already I know of some in my industry who get all of their people "certified" for various of the practices, but to what actual tangible benefit? (Disregarding that it's nice for a company to send you to a conference, because I could think of a number of other conferences more relevant to what we do.) Especially when the "certification" is of the "give us money and we'll certify you" variety; how can someone claim "master"-level command of a subject if no sort of examination was given to test that skillset and knowledge base?

This brings me back around (in my mind, anyway) to one of my earliest posts here, and that is the single most important takeaway for me from all of the agile learning I've done over the past five years. That is, simply, that it's not the process that matters. Devotion to a book or a method or a procedure is only suitable as long as it's applicable to the group that's using it, and it's only applicable to the group if it actually helps that group to accomplish what they're working toward.

Clearly I'm not a fan of certification. In general, I'm not a fan of the "institutionalization" of anything that we do as game developers, because there is no silver bullet. There is no one-size-fits-all. We've got a million problems before us (ok, only 750kloc in my current project but lets assume for the sake of argument that there is more than one bug per line), and no game is the same as any other. This is great! There is huge variety in the teams out there, different teams doing different games with different development strategies, and it would be foolish to think that any one method of doing things would apply across the board. Sure, certain aspects of Scrum, for example, could be useful across a number of teams. Certain bits of XP will work out of the box for a broad range of developers. We can, and probably should, figure this out through experimentation with our own teams, though. We shouldn't, and will probably fail if we try to, go to a conference, hear about ProcessX and say, "this is what our team will use."

...and that is what was bothering my friends in the Lean Manufacturing shop. All show, no go. The "Black Belts" were (or at least, appeared to be) there to ensure compliance, rather than being the thoughtful senseis that the people behind Lean Manufacturing likely intended. But then that's why I thought of this comparison, because I know plenty of folks who are in "agile" shops who swear up and down that it ain't making their lives any easier, it's just bullet points on the marketing brochures.

Agile helps development as much as it's allowed to help. It is empowered by the flexibility that it is given. As much as the team is a living organism, so the processes that the team uses needs to breathe. Inhaling and exhaling, in with the reflections on the process, out with the bits that aren't ineffective. An open mind is all that is required. I would imagine that Lean Manufacturing is intended to make better products faster, fairly obviously since otherwise it wouldn't even be considered for implementation. It's too bad, and I feel sorry for my friends there that the people running the joint don't "get" it. A manager has got to truly grok agility to be able to pull it off, but it is sadly clear that many do not.

Less Yields More?

One of the promises that agile makes is that its adoption leads to less crunch. The implication that is readily available here is that the developers spend less time at their places of employ, more time with their families, and through the magic of agile, the same, if not more, work gets done. How is this possible? Doesn't this assertion violate some fundamental law of conservation? How can it be that any change in process can lead to more productivity?

In Code Complete (in section 28.5), Steve McConnell cites a number of studies that suggest there is over a factor of ten difference in the productivity between programmers. Further studies have indicated that there are also significant, though less pronounced, differences in productivity between otherwise-comparable teams working on identical projects. Is this relevant to the discussion on Agile in any way? In my opinion, most definitely. I would posit that the differences in productivity are related to work habits as much as, or quite likely more than, anything else. Many times I have heard people say, "Agile isn't telling us anything new," and "Agile is merely repackaging things we should be doing anyway." Maybe some of these things, which we all know we should be doing anyway, are some of the things that productive individuals and teams actually do in their own work.

I do fully embrace the idea that the Agile practices can lead to increased productivity. Many of the practices can be applied by the interested individuals, "under the radar" if you will, and those developers will find their own productivity improve. At the end of the day, though, the most effective way to increase one's own productivity is to in have the desire in the first place to become a better programmer. It is those people who want to become better programmers that are the ones seeking out the means by which they can become better; those reading the various forums and learning about any number of techniques to make themselves better at what they do are clearly more likely to make changes in their own work habits to make themselves more effective.

Agile can present enabling technologies to a team desiring to become more efficient. After all, most of the practices are things we've been told we should be doing since we started programming. By themselves, though, they are no magic bullet. Truly increasing performance requires that those involved actually desire to so improve. Doing so otherwise is like washing a cat, it yields only pain and suffering to the washer, while thoroughly aggravating the cat. Avoid the scratches and make sure that the others involved want to take part.