Engineering Everquest – IEEE Spectrum

En Garde: Adam Joffe, CTO of Sony Online Entertainment, battles to keep the company’s online games up and running.
Photo/Illustration: (Left) Gregg Segal (Right) Bryan Christie

Up in the snow-covered hills, a guild of wizards and a group of blue-skinned dark elves close in on caves of ice. Inside lurk ominous frost spirits and who knows what other monsters in the shadows. The wizards summon their magic. The dark elves prepare for battle. It’s showdown time in Norrath.

So let the data flow. After all, Norrath exists solely in the coded world of EverQuest, a best-selling computer game played over the Internet. As its often-fanatical devotees test their mettle in a rich fantasy world of fierce creatures, powerful servers and routers strive to keep up with them in humming computer rooms from California to South Korea.

Published by Sony Online Entertainment in San Diego, the EverQuest franchise is more than a game: it’s the game competitors aim to beat in an industry that’s becoming a dominant force in the entertainment business. According to the market research firm IDC, in Framingham, Mass., annual worldwide sales of video-game hardware and software will swell to almost US $30 billion by 2008.

In the United States alone, people are spending almost $11 billion per year on video and computer games—more than they spend on movie tickets. That $11 billion bought 248 million computer and video games, according to the Entertainment Software Association in Washington, D.C.—two games for every household. Included in the figure are online games, such as EverQuest, played on personal computers, along with other games played on PCs; console games, such as those played on Microsoft’s Xbox; and the various software, graphics accelerator boards, and joysticks that give the games their zing. And as you might expect in a market so vibrant, there’s serious technology ferment.

Computer games, not scientific visualization, have been driving the design of graphics chips, some of which have more logic circuitry than state-of-the-art microprocessors. Games are also among the biggest drivers of new and creative software.

EverQuest and its sequel, EverQuest II, are at the hard-core end of a $5 billion online gaming spectrum that includes games like Blizzard Corp.’s World of Warcraft and Warner Bros. Entertainment Inc.’s The Matrix Online. At the opposite extreme of this spectrum are casual games such as poker and bingo. The people most devoted to EverQuest and its kind—who spend nearly as much time gaming as they do sleeping—are a minority of online gamers, making up about 35 percent of the total in 2003, according to game industry market research firm DFC Intelligence, in San Diego. But it is a lucrative 35 percent, accounting for half of the industry’s revenues. By 2009, DFC says, the hard-core gaming segment will not have grown as quickly as the other gamer categories, slipping to 27 percent, but will still account for 38 percent of online gaming’s projected $10 billion in annual revenue.

For Sony, the business of immersing people in imaginary medieval worlds has been a bonanza. Six years after EverQuest’s release, nearly 600 000 PC gamers are now shelling out $13 to $15 per month in subscription fees, in addition to the onetime fee of around $50 for the CD-ROMs they need to start playing. That’s quite a payback for a game that cost less than $3 million to bring to market.

The goal of Everquest is to build the wealth and power of a mythological alter ego. You use keyboard commands to roam around vibrant virtual worlds that have a medieval look [see screen shots, “Like Magic”]. You see the world unfold in front of you: if you’re climbing lush green hills, you see turf, trees, and rugged mountain ranges in the distance. If you’re swilling mead in a musty tavern, you see something like the Cantina scene from Star Wars. If you’re burying an axe in an ogre’s back—well, you get the idea.

The World of EverQuest: Like Magic Sony Online Entertainment’s EverQuest depicts scenes from the player’s point of view.
Image: Sony Online Entertainment

You start the game by creating your alter ego. You might be a stout, bearded dwarf or a towering, bad-tempered troll. You might be a zombie-banishing priest or an axe-wielding warrior. You can choose everything from the hue of your character’s skin to the hairstyle.

Once your character is complete, it’s time to venture off into one of EverQuest’s dozens of elaborately detailed virtual worlds, which contain whole continents to explore. Within the worlds, there are “burgs” to suit every kind of player—from hack-and-slash arenas where players take on other players to communal hubs for swashbuckling guilds. Some worlds have slightly different rules of play, but the main reason there are so many is simply to avoid overcrowding. At any moment, there are thousands of players online, so some locales in the game would become too crowded with creatures if not for the division into separate worlds

The game is played around the clock, so there are always creatures inhabiting the worlds. That feature is critical to the game’s appeal—because the action continues even when you are not playing. The next time you sign on, you may find your world is different from when you played before. While you are logged off, your character is safe from harm. But a bathroom break can lead to an untimely death if the player doesn’t log off before stepping away from the computer.

The basic idea is to develop your character by maneuvering it so that it accumulates experience and gets rich. Slaying a dragon, for example, is one way to riches. But that can be a complex affair. To do so, you probably would need to get together with some other players, because such a task usually demands the skills and brawn of more than one character. To collaborate, players use the game’s messaging features. Convening in a virtual town, the characters head off together to find the monster in whatever dungeon it may be lurking, their corresponding players chatting about strategies along the way. Once they confront the creature, they combine their skills—swordplay, magic spells, and brute strength, say—to slay the beast and steal its loot.

The most devoted fans call the game “EverCrack” and might play eight hours a day. Singles even meet inside and get married, not only in the game but sometimes in real life as well. The obsession with EverQuest is so strong that an underground economy Sony unofficially estimates to be worth as much as $150 million annually has sprung up to support it. Players barter virtual items collected in the game on eBay. Once a coveted sword went for $1500. Really. Someone paid $1500 in real money so that his or her EverQuest character would be able to wield a desirable weapon.

EverQuest is not just a social phenomenon or a source of cash for Sony; it’s also a phenomenal example of how a company manages rapid growth and the art—and science—of scaling up computing technology. More than 1500 servers around the world run EverQuest; collectively the machines have the bit-crunching capabilities of one of the world’s top 100 supercomputers. In just one of the data centers, more than 30 kilometers of wire and cable connect all the boxes. More than 9 million gigabytes of EverQuest data have been downloaded from Sony’s servers in the past six years.

It all adds up to a monster IT challenge. Although there are 13 data centers in Japan, South Korea, the Netherlands, and on both coasts of the United States, the heart of the monster resides in San Diego, at Sony’s mission control. Its 45 occupants, the real-life heroes behind EverQuest, aren’t wizards. They’re engineers.

Online role-playing games were nothing new when EverQuest debuted in 1999. Some of the first computer games in the 1970s—such as Colossal Cave Adventure—were single-player, text-only riffs on the fantasy role-playing game Dungeons and Dragons. As the early denizens of the Internet took roost that decade, they created multiuser Dungeons and Dragons-style text games for communities of players. In the 1980s and 1990s, the Ultima computer games took the role-playing experience into graphical form, culminating in 1997 with one of the first so-called massively multiplayer online role-playing games, Ultima Online.

EverQuest, which unlike Ultima is rendered from the player’s point of view, aimed to one-up its predecessor and bring the nascent genre into the mainstream. It did that and more. On the day of EverQuest’s release in March 1999, gamers flooded the 45-megabit-per-second cables connecting to Sony’s server farm. Along Interstate 5, in the heart of San Diego, you can still see the scars of that deluge; they’re the remains of long trenches in the pavement dug as part of a frantic push to add more lines to the Sony facility.

“Internet service providers were in the habit of oversupplying bandwidth,” recalls Michael Bowen, executive director of technical operations for Sony Online Entertainment. “They didn’t believe if you buy a pipeline this size, you fill it. But we filled it so much that other people in San Diego lost Internet access because pipes were so full. They scrambled to increase capacity because of us.” Sony’s ISP—the company declines to name it—raced to boost the bandwidth, ripping up Interstate 5 in the process.

Sony runs two other online games, PlanetSide and Star Wars Galaxies, out of its worldwide network of server facilities. At any given time, Sony is hosting more than 150 000 sword-wielding, laser-shooting, dragon-slaying gamers from all over the world.

“We’re looking at transaction rates that rival the Visas and Wall Street brokerage houses,” says Adam Joffe, chief technology officer for Sony Online Entertainment. “Thousands and thousands of transactions per second.” And it takes epic levels of processing power to keep it all going

“You’re About To Go into the Death Star,” Joffe says. A balding, goateed guy with narrow, rectangular glasses, Joffe places his palm in a biometric scanner in the first-floor lobby of a building near Sony Online Entertainment’s main offices, and a gray door unlocks noiselessly.

At any given time, Sony is hosting 150,000 gamers

The Death Star is a huge, warm, windowless room containing the rows and rows of servers that run Sony’s online games. The whooshing of a massive air-conditioning system is so loud that conversation is almost impossible. A large steel cage surrounds more than 500 servers stacked 32 high in towering racks—and this is just one battalion, albeit the largest, in Sony’s 1500-machine army of servers.

Other than the graphics and sound, which are loaded directly onto the players’ machines from CD-ROMs when EverQuest is installed, everything else needed to run the game—including the players’ characters—is stored on these machines. Sony calls them “world servers.”

It’s more than just a catchy name. Remember that to prevent overcrowding, EverQuest is divided into dozens of parallel worlds. The worlds are reflected in the server farm as interconnected combinations, or clusters, of servers; the cluster size is based on how many users Sony expects to support simultaneously. In other words, each world is a cluster consisting of between 20 and 30 dual-processor computers. And within the clusters, individual processors are devoted to producing different pieces of geography—a town, a forest, a labyrinthine castle—in those worlds.

When a player logs on to the game, the program, or client, being used connects to servers in the cluster the character was last playing in. Those servers then download data describing everything—the alter egos, locations, weapons, and other characteristics of other players who are logged on, plus all the relevant monsters and weapons nearby. A full EverQuest install, which requires six CD-ROMs, weighs in at about 3 gigabytes. To log on, players must have the latest version of the software, which is updated through downloadable patches every two to four weeks.

When the patch is large—say, 25 or 30 megabytes—play stops and Sony takes the servers down for as long as several hours (but usually less than 30 minutes). During that time, Sony updates the code on the servers and sends the new software to the players. The patches are not merely fixes to problems. They can be brand-new content—a new city, dungeon, or continent to explore.

Each world can support about 2500 players at a time, although it can store data on up to 10 000. When EverQuest launched in 1999, there were just 12 worlds supporting 100 000 players. Last year there were 52 supporting half a million.

As the game’s audience expands and evolves, so does the architecture behind the scenes. The process begins every time Sony adds big new chunks of content to an existing world. New dual-processor servers are first added to the rack deep inside the Death Star. The more content required, the more machines are pressed into service to provide it.

Once Joffe’s team adds a server to the rack, he loads a Unix operating system onto it. Custom-built software then automatically loads all the game software and the programs the new machine uses to communicate with other servers, configures its storage systems, and starts it up. The server is then ready to be added to the cluster that supplies data about a particular world in EverQuest. The entire process of joining the server farm occurs within minutes, Joffe says. As characters populate the virtual locales that the new machine produces, their data are stored there.

All those expansions make for a crowded server room. To be able to keep expanding its computing power without outgrowing the Death Star, Sony has begun using blade servers. Stacked like books on a shelf, the slim computers share certain hardware—such as power supplies, cooling fans, and network interfaces—among themselves, allowing more servers to be housed in a single rack [see “Blades Have the Edge,” IEEE Spectrum, April 2005]. But even blades have their limitations. “Blades are so dense that they produce too much heat,” Joffe says. “You can stuff them into small spaces, but you need space to cool them properly.”

A few years ago, every time a player in a particular world entered a specific virtual dungeon, chances were the same central processing unit, dedicated to that place alone, served up the data seen on the display. Each CPU of each server in a world cluster was assigned a geographical task. But that way of dividing up computing resources turned out to be too inefficient to handle the new content the game’s designers were pumping out. So a few years ago, Sony shifted to a new way of managing the system: just-in-time computing.

As Joffe explains it, EverQuest’s just-in-time system allocates computer resources based on user demand. “As they go into parts of the world that are dynamic, we launch a process that meets users’ needs,” Joffe says. For example, a player, Porslap the Dark Elf, is running through a corridor when he comes to a door. By opening that door, Porslap triggers actions on several machines. If, say, a dungeon lies behind the door, Sony’s system looks up the data and software that describe that dungeon on one computer, finds some idle processing resources within the cluster, probably on a different machine, and runs the software on it. Sony’s servers download the dungeon’s data to Porslap’s computer “just in time” to meet the inquiring elf.

When problems hit Everquest, as problems inevitably do, the first indication comes inside the Network Operations Center. The NOC, a crowded room in Sony Online Entertainment’s main office building, smells faintly of elementary-school glue. There are rows of flat monitors and lists of EverQuest world names projected on the wall: Morell Thule, Veeshan, The Nameless. There is a cushy couch against another wall, a couple of bottles of soda, and a few crumpled napkins.

The room has a college dorm all-nighter feel, and for good reason. Twenty-four hours per day, 365 days a year, the NOC is ground zero for Sony Online Entertainment’s technical support. Gamers game around the clock, so Sony has to be ready to handle problems whenever they happen, and developing and maintaining a system to overcome the challenges is essential to the game’s success. For now, all is well, but at any moment the crew might have to save a dying CPU in the Death Star or resurrect software that has been killed by bugs

Three people per shift work in the NOC, and there are three shifts per day. During each shift, NOC staff monitor game activity, responding to players in remote locations and working with a custom suite of software tools to fix problems along the way.

The center has shut down for just three days of work—all in 2003, when wildfires were closing in on the neighborhood. (Because all the tools run remotely, the staff members kept EverQuest going from their homes. A backup generator kept pumping power to the servers even as the fire threatened to black out parts of the city.)

The biggest problem facing the NOC is bugs, which can have unusual consequences in EverQuest. The bugs that have to do with the EverQuest virtual economy are especially bothersome. Players receive virtual money, called platinum, through accomplishments such as defeating dragons. The platinum is then used to purchase weapons and other objects within the game. Therefore, bugs relating to items and platinum have the potential to dismantle the game’s economy. For example, a shopkeeper in the game might hand over an expensive sword when a player had actually just paid for a less valuable dagger.

Recall that EverQuest’s economy extends into the real world, where the truly obsessed spend an estimated $150 million per year—20 percent of the total online gaming underground secondary market—purchasing everything from fully developed characters to rare weapons. Because of the overlap between the virtual and real economies, an incentive exists for cheating players to try to hack the system to duplicate platinum on the fly.

Call them virtual counterfeiters. To hunt them down, Sony has a team, separate from the NOC, that combs game logs for suspicious activity. “We have ways of observing our world and what goes on,” says Chris Kramer, Sony Online Entertainment’s public relations director. “We keep checks on the economy, so when we find that something happened to create a spike in the economy, we go back and track it immediately.”

Hackers who are caught are expelled. However, because it’s a virtual world, they can create new characters and return.

Auctioning and sale of virtual goodies from its games has long been against the rules. But in a version of “if you can’t beat ’em, join ’em,” Sony planned a late June launch of a Web site devoted to the auction and exchange of EverQuest booty, including platinum. It hopes to draw people away from the unofficial exchange sites by offering security. No more need to trust that a seller will actually meet you in a virtual dark alley and hand over the item you paid for.

Instead, items put up for auction will immediately be removed from the game and transferred to the winning bidder. Sony gets a fee from the seller and a percentage of the winning bid.

It’s the end of another day shift behind the scenes of EverQuest, and its many worlds keep turning. Online, the frost spirits in the ice caves have been vanquished. Meanwhile, in the real world, the troops in and around the NOC drive off into the late afternoon sun.

The work is far from done, though. As the EverQuest brand grows, the prospects for online gaming continue to evolve. In recent months, the gaming industry has experienced a boom in sophisticated handhelds, including Sony’s PSP, Nintendo’s DS, and Tiger Telematics’ Gizmondo. The PSP, Sony’s first such gadget, includes a three-dimensional graphics engine, an advanced sound engine, and two 32-bit microprocessors that each boast the same amount of power as the PlayStation 2’s single CPU. The device sold out within days of its release in Japan in December and in the United States in April.

With their Wi-Fi and other wireless capabilities, the PSP and Gizmondo seem on the brink of making a bid to unite the huge but separate worlds of online and portable gaming. Nevertheless, Sony has not disclosed any plans to roll out EverQuest for a mobile platform. In April, Sony Online Entertainment released its first PSP game, Untold Legends, which supports four-player wireless gaming. But unlike EverQuest, the game is run entirely on the console, without input from Sony’s servers.

There is also the question of how big a market online games like EverQuest can become. Some analysts question whether it can grow much more. EverQuest represents the extreme end of online gaming, requiring a big commitment from its players. The learning curve for a new player in EverQuest is tough, and players tend to spend 20 hours each week on the game. “There’s a group of people willing to do that, but it’s a small group of people,” says Schelley Olhava, an analyst with IDC. “I’d be surprised if the subscribers to games like this ever reached above 2 million in the United States.”

On the other hand, there are many ways for game makers to increase revenues. For example, Sony recently tested a service that let players order pizza from a national restaurant chain from within the game. And every now and then, a video game is turned into a motion picture, or vice versa. In a surprising turn in February, word hit Hollywood that Microsoft Corp. would be writing its own script for a movie version of its hit game Halo. In a world where Microsoft makes movies and virtual dark elves order real pizza, the battle for the entertainment industry has begun to get, well, entertaining.

About the Author

Contributing editor David Kushner wrote about the creators of the computer games Doom and Quake in his book, Masters of Doom, published in May 2003 by Random House.

To Probe Further

For all things EverQuest, go to http://eqlive.station.sony.com.

For alternative computing structures for games like EverQuest, see “FreeMMG: A Scalable and Cheat-Resistant Distribution Model for Internet Games,” by Fabio R. Cecin, et al., presented at the Eighth IEEE International Symposium on Distributed Simulation and Real-Time Applications, 21-23 October 2004.