[[This article was originally e-published on ArenaNet.]]

I copied it as well as linked to it because there's no guarantee how long the article will survive at it's current location and it's seems particularly relevant to discussions in recent days (today is Feb 25 2002).

"The Challenges of Making a Multiplayer Game" Patrick Wyatt

Introduction

As you've probably noticed among your computer gamer friends, on-line gaming has grown by leaps and bounds over the last few years. Many more homes have Internet connections, cable modem and DSL are expanding into more neighborhoods, and the number and quality of games that involve people in the on-line environment have also grown rapidly. While many quote statistics from Jupiter Media and other research firms, we've seen firsthand how fast on-line computer games are growing by being directly involved in creating and running a gaming network.

As game developers, and as gamers, we're excited about this growth, and we want to satisfy the expanding market for on-line games. But as developers, we are acutely aware of the challenges involved in reaching our goals. In this article we discuss some of the issues involved with creating a good on-line game, and outline some of the challenges in making everything work.

What Makes a Multi-Player Game Great?

Depth: A successful game will be designed so that there is not a single "best" way to play. If there is one very effective "killer strat," or if one character or racial type is wildly unbalanced or has noticeable advantages over the others, then everyone will be tempted to play precisely the same way, which creates a dull play environment.

The ideal game will offer many different ways of winning, and will encourage exploration of different characters and strategies. A good developer can incorporate elements of the "paper, scissors, rock" challenge, where for every in-game choice - be it strategy, character type, item use, area of spell research, point placement - the exponential rise of variable interactions is dealt with. Balancing two spell types against one another isn't remarkably difficult. But balancing those same two spells against one another, when used by a variety of character types, and when used in combination with a multitude of other spells and items, becomes much more complex. This is an effect of combinatorics, where adding more elements to the game design causes the number of possible combinations to rise on a massive scale.

Ideally, no individual item, character, or strategy should be inherently better than any other. As an example, in StarCraft, if you run ten of the heaviest Siege Tanks into a base, you will nearly always lose in your attempted assault, despite that fact that the unit itself is very effective. Its effectiveness lies in the combinations you choose. If you layer your choice of Siege Tanks with other units - Vultures, Marines, Firebats - you'll have a far greater chance of success. So you have chosen one possibly successful strategy, and of course that strategy can be countered with a dozen defensive strategies. When we developed the game - and when we develop our future games - we have to consider all the combinations, singly and in multiples, so that any one choice isn't always victorious, but rather that a variety of choices are given a balanced opportunity of victory against different defenses.

The large number of strategies, counter-strategies and counter-counter-strategies means that it's incredibly important to have a long testing cycle and to include the gaming public in the testing process. No matter how much testing is done inside the company, any game that's complex enough to have many strategies and counter-strategies is also so complex that the developers cannot exhaustively test every possible scenario. By publishing via the Internet we expect to be able to provide immediate, incremental patches to handle play-balance issues both large and small rather than waiting several years for uber-patches as is more common with retail titles; our long development process (two years and counting!) is partially due to the effort we've spent on automating every part of the patching process.

Learning Curve: Another important factor is ease of use and the individual player's "learning curve." It's difficult to judge the quality of some games from an early view because the true level of game quality only becomes obvious after the investment of many hours of a player's time. Few players have the patience to master games with a difficult learning curve (myself included!), and the interest in those games quickly fades. Most good games, and most widely successful games, have a gentle learning curve so that new players aren't intimidated by the complexity of the overall game but can gain playing skills while having fun, while experienced players continue to add to their repertoire of tactics well after the game's launch.

Games that have "startup missions" or "walkthroughs" are essentially guiding players up the slope of the learning curve while entertaining them, but of course there has to be a careful balance between "easy enough" and "completely unchallenging". We've found that adding fresh testers all the way through the game development process dramatically improves the quality of the early missions. One of the difficulties we regularly faced in developing both the 'Craft and Diablo series was that our internal testers rapidly became the best players in the world, because playing 8 to 12 hours a day is like going to gaming college. We had to be careful about setting the difficulty level to meet their rapidly advancing skill level, and we frequently added new testers to the mix to ensure that the game difficulty increased only moderately for the early game missions.

Public beta tests growing from tens to thousands of gamers during the last few months of game development are an excellent means to squish bugs, validate play balance and check for hardware compatibility, but there is another form of trial that can be very useful as well, which is called "usability testing." This type of test requires watching individuals playing the game without providing any feedback to see how they navigate the interface, to learn how intuitively the interface works as they progress through the game. Standing back and observing a player move through the game for the first time is incredibly helpful to developers in discovering game areas that they need to improve in order to make things more user friendly or more logical to the average person.

The final developer's challenge related to the learning curve is to create a playing environment that establishes viable player match-ups so that novices can fight against others with a similar skill level, grandmasters can find a challenging game, and players of every other skill level in between can match up with others of like ability. Ladder rankings help establish viable player match-ups, and difficulty settings are an assist as well. Allowing players to choose individual difficulty settings is a big win, because there is no right or wrong answer; the players can agree on the game balance among themselves.

Fairness and Chance: Game balance should work in such a way that the better player usually wins, but the underdog always has a more than fair chance. Many games have a problem where the first few minutes of play determine the outcome, although it still takes a long time for the inevitable win. We called this the "Z-factor" at Blizzard, after the RTS game called "Z", by the Bitmap Brothers. While it was well-done in many regards, the game was generally a race to victory: as soon as one player captured one base more than his opponent, he would inevitably win no matter the strategy of the other player. With Warcraft and StarCraft, we tried to create a game where a good player will usually win over one who is average in skills, but where even the less-experienced player has a chance of winning from time to time with the use of novel strategies or even just a bit of game-generated luck.

It is difficult to work in elements of fairness and chance, because even small advantages tend to compound over time. Having one more Peon unit is like compound interest: if you don't start "investing" early, you won't be able to retire! An example of the developers helping out the underdog can be found in many racing games, where the racer who is behind is given a slight speed boost. Sometimes, that'll be enough to zoom to the front just at the finish line. Of course, some players learned to capitalize on this "underdog advantage," but games are generally better for its inclusion.

Another challenge is to create diversity rather than completely analogous units. A good game doesn't have units that are completely balanced one-for-one across the board, but it should have pairings that offer pluses and minuses that make them somewhat equal. Considering Warcraft's Elven Archers versus the Orcish Spearthrowers: we had tried to make them somewhat different, and so gave the Spearthrowers more damage points, but the Archers a bit more range. The problem is, of course, that the Spearthrowers can't do any damage at all if they don't get close enough to attack, and in a match of Elven Archers versus Orc Spearthrowers, the Spearthrowers are dead before they get a good chance to try to take out the Archers. In Warcraft II, we wanted the Paladin's Healing to be an excellent counter, pretty close to a "tit for tat," to the OgreMage's Blood Lust. However, due to the game mechanics and the spellingcasting constraints of Healing (such as having to individually select each healable unit), Blood Lust will always take the day. It wasn't until we changed from Warcraft's "unit equivalence" to StarCraft's "race equivalence" that we were able to correct the most egregious play imbalance issues.

Social Opportunities: On-line gaming is a very social experience, and gamers want the chance to play in games with their friends, guild members, or acquaintances. Players sometimes desire to have places to meet outside the gameworld itself, which is why forums, chat rooms, guildhalls, and message boards are so popular. Multiplayer games should cater to these social desires, and make it easy to meet friends (new and old), and to keep connections with peers. The fewer barriers there are to this social interaction, such as divided game servers that separate people by geography, the better.

The reason that most games split people into separate "geographic" groups is because it's (relatively) easy to write a standalone server, but much harder to have the servers communicate, as they have to be designed to share workload, minimize communication overhead and resource contention, and minimize the latency between users. While banking software can take 5 or 10 seconds to approve a transaction, having a conversation with another person with 5 second lag is an exercise in frustration. The reason that most companies don't make the effort to correct this is because it is difficult and time consuming. And certainly, if you don't design a system that will handle these challenges up front, it's very difficult to correct the problem later.

Moderation: We feel - as gamers as well as developers - that expectations of a certain level of behavior are a good thing. Some level of anarchism in a game world is appropriate, but it's important that there's a fair set of rules so that antisocial players don't end up ruining other players' experiences. People can actually think their choices are "justified" by the amusing illogic of concepts like "I had to hit him because he was going to hit me first." While some games provide game moderators to attempt to correct "social injustices", it is a daunting task to successfully perform game moderation. There are probably a few thousand game developers and moderators in the world, while there are millions of game players, so the logistics of being able to create a well-moderated environment for a successful game are difficult indeed.

So, how do developers meet the desire for free choice that the gamers want, while not allowing any one group to run amuck and spoil it for everyone? They offer choices. One choice could be a selection of environments or game types. For example, if you have a game that allows player-killing, decide whether to establish a separate world for those players, or whether to allow them to play in their choice of game worlds but with limits, such as mutual acceptance of PK Mode, or whether there are no limits, but there is a warning system in place. Do you want to allow back-stabbing, like alliance switching or hidden partnerships? Then give people enough choices that they know what they're getting into. Offer melee-type games which allow these changing alliance options, but also offer free-for-all games that disallow alliances from the outset so that those who want to play "every man for himself" and want to avoid betrayal by allies can choose that mode. For those who like to play in a group environment, offer team games as well as melee games. Team games will offer fixed alliances and remove the option for "teammate betrayal," where melee games will always contain that "edginess of the unknown" that some gamers enjoy.

Giving gamers tools to choose play partners, including rankings and buddy-lists is a good start. A goal of ArenaNet is to give players the tools to perform a great deal of self-moderation by creating their own guilds. Each player will be able to select a guild (or several) appropriate to his or her own playing style, and have the expectation of playing against others with similar "social instincts."

Player Choices: Any game requires a player to make choices. Whether it's what character class to be, what strategy to pursue, what technology to specialize in using, or what area of the gameworld to explore, there are many questions to answer. A player needs to feel satisfied with his game choices; this long-term satisfaction is essential to a successful on-line game experience. Developers need to create an environment where each person has the chance to feel like "the hero" and where choices made early on are not constraining later in the game. After all, does anyone really want to be an armor maker, or is it that the game world is not creating enough rewards for a more heroic play-style?

This challenge can be addressed by considering what sorts of psychological rewards might inadvertently be given for unexpected behaviors. Do developers really want to encourage "spawn camping," where people spend hours a day in one spot, killing the same monsters over and over again, in order to level up or get great items? By rewarding players for trading items, do they want to encourage players whose sole occupation is making applesauce? These phenomena are preventable if the development team will forecast a consequence for each choice.

Replayability: After you've handled the factors listed above, then replayability and longevity are definite considerations. No one wants to spend his hard-earned cash for a computer game that ends quickly and doesn't encourage replay. The best games are those that invite you back again and again, to learn a new character type, explore a fresh area, investigate a new group of skills or abilities, or to try a totally different strategy. Some of the simplest concepts are the most long-lived. Examining the rules of chess is easy, but the game itself draws lifelong dedication to learning its intricacies and mastering its art.

The technical hurdles of multi-player games

A Lag-free Environment: Few things raise the hackles of on-line gamers more than "lag" in a game. Lag occurs when the multiple computers involved in the game become unsynchronized and fail to reveal the same worldview or begin to respond to player commands at varying speeds. The game must utilize as little bandwidth as possible in order to run on the widest variety of systems and to avoid the evil Lag Monster. In addition to real-world difficulties introduced by the light speed delay of packets sent over long-distances, re-transmission of packets caused by network "hiccups", and the narrowness of the pipeline through which packets are sent, developers have to avoid a whole slew of programming complexities that actually induce lag.

First and foremost, a game must minimize the number and size of packets that need to be sent. Since each packet has a substantial amount of overhead in the form of a "packet header", sending two packets with one command each has twice the overhead of sending one packet containing two commands. Furthermore, every "large" packet that has to be sent delays all the packets "behind" it in the transmission queue because of the time required for the hardware to transmit the data.

In addition, we need to search for induced latency, which is caused when the programmer has done something in the code that causes packets to be delayed, which is quite a bit more common than you might imagine. The bottom line is if we achieve greater efficiency in the code, and a more direct order of operations, lag will be reduced.

Finally, the most important thing is to test the game the way it will be played! A lot of multiplayer games are tested, not over the Internet, but exclusively on the internal high-speed local area network. This is a recipe for disaster, as you can only find out how things will work in "real life" by doing "real life testing." At ArenaNet we've made a point of developing and playing our own game over the Internet to ensure that we suffer exactly as much as our users have to.

Responsiveness: Once everything possible has been done to minimize latency, the next step is to minimize the appearance of latency (also known as "latency masking"). For example, the 'Craft engine "acknowledges" orders immediately with sound effects ("Yes, my liege"), and by changing the cursor graphic, for example showing a target reticule when you are in targeting mode rather than the normal selection cursor. But in both instances, it may be several game cycles before the order is actually carried out. Even when a serious network hiccup occurs, the game will still accept user input, and although the result of the input may be delayed, the game still "feels" responsive. Only as a last resort does the game halt temporarily to display a network timeout indicator.

Conclusion

While working on our first ArenaNet title, we're keeping our past experiences in mind, while working to build something even better in the future. The challenges of creating a successful multiplayer game pose interesting obstacles, but they also present fascinating opportunities. We consider ourselves fortunate to have the chance to build something we're really excited about, and we look forward to sharing it with you.